Disaster Recovery Tasks

From Alteeve Wiki
Jump to navigation Jump to search

 AN!Wiki :: How To :: Disaster Recovery Tasks

Note: At this time, DR management is via the command line. DR management in the UI is planned.

Disaster Recovery on M3 clusters provides a way to create a copy of a server on a remote "DR Host". These are machines that generally match the capabilities of the Anvil! node the server runs on, but that's not a requirement.

Servers don't live migrate to the DR host, and the DR host itself is not a member of the subcluster. Simply speaking, the DR host has a copy of the server's virtual hard drive(s), and the hypervisor has the configuration needed to boot the server. If the production node is destroyed, you can boot the server on the DR host.

Linking DR Hosts

A DR host can be used by multiple Anvil! nodes, and multiple DR hosts can be used by a given Anvil! node. To support this, a "linkage" need to be created to connect a DR host to a subnode.

You can see the current list of Anvil! nodes, DR hosts and servers using the anvil-manage-dr --list command

anvil-manage-dr --list
Anvil! Nodes
- Node Name: [an-anvil-01], Description: [Demonstration anvil node]
 - No linked DR hosts yet.
- Node Name: [an-anvil-02], Description: [RX200 S8 Test Node]
 - No linked DR hosts yet.
- Node Name: [an-anvil-04], Description: [Test Mixed-Hardware Anvil! Node]
 - No linked DR hosts yet.

-=] DR Hosts
- Name: [an-a01dr01.alteeve.com]

-=] Servers
- Server name: [srv01-primary-dc] on Anvil! Node: [an-anvil-01]

This is a simple three-node Anvil! cluster with a single server and a single DR host. This is somewhat artificial, but it should keep things simple or these examples.

A Note of DR Host Naming

The host name for the DR host name an-a01dr01 indicates that it "belongs" to the an-anvil-01 node, but this is by convention, not a requirement.

Typically, to ensure that performance is the same if DR ever needs to be pressed into service, there will be one DR host matching the hardware of a given Anvil! node (pair). As such, the servers running on a given node will be protected by a common DR host. This is why the naming convention tends to be:

  • prefix-aXXdrYY, where XX is the corresponding Anvil! node sequence, and YY is the DR sequence, typically 01.

In any case, the aXX prefix is not important, except to the human reading it, as it has no programmatic importance.

It is not uncommon to use smaller capacity machines as the DR host(s), requiring multiple DR hosts for a given node.

Alternatively, it's not uncommon to buy one very large capacity DR host to handle protection of multiple Anvil! nodes. In such a case, typically prefix-a01dr01 is used. Again, you are free to use whatever host name makes more sense to you.

Linking a DR host to an Anvil! node

The first step is to link the DR host an-a01dr01 to the Anvil! node an-anvil-01.

anvil-manage-dr --link --anvil an-anvil-01 --dr-host an-a01dr01
The DR host: [an-a01dr01] as been linked to the Anvil! node: [an-anvil-01].

We can verify this with --list.

anvil-manage-dr --list
Anvil! Nodes
- Node Name: [an-anvil-01], Description: [Demonstration anvil node]
 - Linked: [an-a01dr01], link UUID: [c764b512-ebb4-487f-a631-d0ea6c5d262b]
- Node Name: [an-anvil-02], Description: [RX200 S8 Test Node]
 - No linked DR hosts yet.
- Node Name: [an-anvil-04], Description: [Test Mixed-Hardware Anvil! Node]
 - No linked DR hosts yet.

-=] DR Hosts
- Name: [an-a01dr01.alteeve.com]

-=] Servers
- Server name: [srv01-primary-dc] on Anvil! Node: [an-anvil-01]

Perfect, we can see that the DR host is linked (uuid: c764b512-ebb4-487f-a631-d0ea6c5d262b).

Protecting a Server

Before we can protect a server, you need to decide on a protocol.

DR Replication Protocols

When replication is setup, the storage is mirrored, as it already is between the subnodes in an Anvil! node. The key difference though is that, where replication is synchronous between subnodes, it can be synchronous, or asynchronous to the DR host.

Synchronous Replication

More specifically, when a write comes down from the server to the replicated storage, the data must be written to persistent storage on both subnodes before the guest server is told that the write is complete. In this way, the unexpected destruction of the active subnode will be recovered from on the peer with no loss of data (no more so than if a traditional machine lost power).

This means that the storage replication link (over the Storage Network) sets the limit on storage bandwidth and IOPS. Between subnodes, this is fairly easy to account for and minimize by using back-to-back, high speed and low latency interfaces. This is easy because the subnodes are physically very close to each other.

Asynchronous Replication

With DR hosts, of course, the point is that they are physically distant from the Anvil! node. As such, the replication traffic will almost certainly run over a higher latency and lower bandwdith connection. To prevent adverse performance impact, servers protected using asynchronous replication are allowed to fall behind production by a small amount.

Specifically, when a write is sent down from a server, the data is still synchronously replicated between the subnodes, as always, but the data destined for the DR host is considered "written" as soon as it reaches the host subnode's network transmit buffer. In this way, if the replication link has a higher latency, the guest server will not feel it.

Of course, this does mean that if the Anvil! node is unexpectedly destroyed, the data in the transmit buffer is lost.

However;

The Anvil! uses multiple volumes within a single replicated storage resource. What this means is that write ordering is maintained, even for servers with multiple (virtual) hard drives. In this way, write ordering is maintained. As such, data on the DR host, though it may have been allowed to fall back in time, will be consistent. The server on the DR host, on boot, will be able to replay file system journals, process database write-ahead logs, and take other actions to ensure data consistency, just as if a normal machine lost power and had to reboot.

In asynchronous replication, there are two options;

Short-Throw This is the default, and it uses the standard network transmit buffer
Long-Throw This is a commercial option that uses system RAM as an extension to the built-in kernel transmit buffer, used for particularly high latency, low bandwidth networks.

The reason there are two options is this;

If the rate disk writes bursts long enough to fill the transmit buffer, storage writes will slow until there is space on the buffer. In typical installs where DR is in a distant part of the same building, or in another building in the same campus, short-throw is generally fine. However, if the local storage is subject to long periods of burst writes, this could still overwhelm the local network transmit buffer.

There are other options to deal with filled transmit buffer while still using short-throw, which will be discussed later in this article.

For now, the key benefit of long-throw DR is that is uses a commercial product from our partner LINBIT called DRBD Proxy. This add-on, in short, uses up to 1 GiB of system RAM as an extension of the kernel's TCP transmit buffer. This greatly increases the amount of buffered data before contention becomes a concern, albeit at the expense that the data on the DR host is allowed to fall further behind during periods of burst writes or low bandwidth.

Protecting A Specific Server

Now that the discussion on protocols is behind us, lets protect our example server, srv01-primary-dc. For now, we'll use the default async short-throw protocol.


 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche! Enterprise Support:
Alteeve Support
Community Support
© Alteeve's Niche! Inc. 1997-2024   Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.