Disaster Recovery Tasks

From Alteeve Wiki
Jump to navigation Jump to search

 AN!Wiki :: How To :: Disaster Recovery Tasks

Note: At this time, DR management is via the command line. DR management in the UI is planned.

Disaster Recovery on M3 clusters provides a way to create a copy of a server on a remote "DR Host". These are machines that generally match the capabilities of the Anvil! node the server runs on, but that's not a requirement.

Servers don't live migrate to the DR host, and the DR host itself is not a member of the subcluster. Simply speaking, the DR host has a copy of the server's virtual hard drive(s), and the hypervisor has the configuration needed to boot the server. If the production node is destroyed, you can boot the server on the DR host.

Linking DR Hosts

A DR host can be used by multiple Anvil! nodes, and multiple DR hosts can be used by a given Anvil! node. To support this, a "linkage" need to be created to connect a DR host to a subnode.

You can see the current list of Anvil! nodes, DR hosts and servers using the anvil-manage-dr --show command

anvil-manage-dr --show
Anvil! Nodes
- Node Name: [an-anvil-01], Description: [Demonstration anvil node]
 - No linked DR hosts yet.
- Node Name: [an-anvil-02], Description: [RX200 S8 Test Node]
 - No linked DR hosts yet.
- Node Name: [an-anvil-04], Description: [Test Mixed-Hardware Anvil! Node]
 - No linked DR hosts yet.

-=] DR Hosts
- Name: [an-a01dr01.alteeve.com]

-=] Servers
- Server name: [srv01-primary-dc] on Anvil! Node: [an-anvil-01]

This is a simple three-node Anvil! cluster with a single server and a single DR host. This is somewhat artificial, but it should keep things simple or these examples.

A Note of DR Host Naming

The host name for the DR host name an-a01dr01 indicates that it "belongs" to the an-anvil-01 node, but this is by convention, not a requirement.

Typically, to ensure that performance is the same if DR ever needs to be pressed into service, there will be one DR host matching the hardware of a given Anvil! node (pair). As such, the servers running on a given node will be protected by a common DR host. This is why the naming convention tends to be:

  • prefix-aXXdrYY, where XX is the corresponding Anvil! node sequence, and YY is the DR sequence, typically 01.

In any case, the aXX prefix is not important, except to the human reading it, as it has no programmatic importance.

It is not uncommon to use smaller capacity machines as the DR host(s), requiring multiple DR hosts for a given node.

Alternatively, it's not uncommon to buy one very large capacity DR host to handle protection of multiple Anvil! nodes. In such a case, typically prefix-a01dr01 is used. Again, you are free to use whatever host name makes more sense to you.

Linking a DR host to an Anvil! node

The first step is to link the DR host an-a01dr01 to the Anvil! node an-anvil-01.

anvil-manage-dr --link --anvil an-anvil-01 --dr-host an-a01dr01
The DR host: [an-a01dr01] as been linked to the Anvil! node: [an-anvil-01].

We can verify this with --show.

anvil-manage-dr --show
Anvil! Nodes
- Node Name: [an-anvil-01], Description: [Demonstration anvil node]
 - Linked: [an-a01dr01], link UUID: [c764b512-ebb4-487f-a631-d0ea6c5d262b]
- Node Name: [an-anvil-02], Description: [RX200 S8 Test Node]
 - No linked DR hosts yet.
- Node Name: [an-anvil-04], Description: [Test Mixed-Hardware Anvil! Node]
 - No linked DR hosts yet.

-=] DR Hosts
- Name: [an-a01dr01.alteeve.com]

-=] Servers
- Server name: [srv01-primary-dc] on Anvil! Node: [an-anvil-01]

Perfect, we can see that the DR host is linked (uuid: c764b512-ebb4-487f-a631-d0ea6c5d262b).

Adding Volume Groups to Storage Groups

If the DR Host's storage differs from the node it's been linked to, it's unlikely that the volume group on the DR host will be connected automatically to a storage group on the node. If this is the case for you, please see:

Protecting a Server

Before we can protect a server, you need to decide on a protocol.

DR Replication Protocols

When replication is setup, the storage is mirrored, as it already is between the subnodes in an Anvil! node. The key difference though is that, where replication is synchronous between subnodes, it can be synchronous, or asynchronous to the DR host.

Synchronous Replication

More specifically, when a write comes down from the server to the replicated storage, the data must be written to persistent storage on both subnodes before the guest server is told that the write is complete. In this way, the unexpected destruction of the active subnode will be recovered from on the peer with no loss of data (no more so than if a traditional machine lost power).

This means that the storage replication link (over the Storage Network) sets the limit on storage bandwidth and IOPS. Between subnodes, this is fairly easy to account for and minimize by using back-to-back, high speed and low latency interfaces. This is easy because the subnodes are physically very close to each other.

Asynchronous Replication

With DR hosts, of course, the point is that they are physically distant from the Anvil! node. As such, the replication traffic will almost certainly run over a higher latency and lower bandwdith connection. To prevent adverse performance impact, servers protected using asynchronous replication are allowed to fall behind production by a small amount.

Specifically, when a write is sent down from a server, the data is still synchronously replicated between the subnodes, as always, but the data destined for the DR host is considered "written" as soon as it reaches the host subnode's network transmit buffer. In this way, if the replication link has a higher latency, the guest server will not feel it.

Of course, this does mean that if the Anvil! node is unexpectedly destroyed, the data in the transmit buffer is lost.


The Anvil! uses multiple volumes within a single replicated storage resource. What this means is that write ordering is maintained, even for servers with multiple (virtual) hard drives. In this way, write ordering is maintained. As such, data on the DR host, though it may have been allowed to fall back in time, will be consistent. The server on the DR host, on boot, will be able to replay file system journals, process database write-ahead logs, and take other actions to ensure data consistency, just as if a normal machine lost power and had to reboot.

In asynchronous replication, there are two options;

Short-Throw This is the default, and it uses the standard network transmit buffer
Long-Throw This is a commercial option that uses system RAM as an extension to the built-in kernel transmit buffer, used for particularly high latency, low bandwidth networks.

The reason there are two options is this;

If the rate disk writes bursts long enough to fill the transmit buffer, storage writes will slow until there is space on the buffer. In typical installs where DR is in a distant part of the same building, or in another building in the same campus, short-throw is generally fine. However, if the local storage is subject to long periods of burst writes, this could still overwhelm the local network transmit buffer.

There are other options to deal with filled transmit buffer while still using short-throw, which will be discussed later in this article.

For now, the key benefit of long-throw DR is that is uses a commercial product from our partner LINBIT called DRBD Proxy. This add-on, in short, uses up to 1 GiB of system RAM as an extension of the kernel's TCP transmit buffer. This greatly increases the amount of buffered data before contention becomes a concern, albeit at the expense that the data on the DR host is allowed to fall further behind during periods of burst writes or low bandwidth.

Protecting A Specific Server

Now that the discussion on protocols is behind us, lets protect our example server, srv01-primary-dc. For now, we'll use the default async short-throw protocol.

Before we start, lets look at the replicated storage status. In this example, we're on the subnode an-a01n01 hosting the server.

Sync progress:
  Res                 To           Vol
- srv01-primary-dc -> an-a01n02 ->   0: [0;0]  //  ds:UpToDate/UpToDate, ro:Primary/Secondary
* Total transfer speed is about: [0 MiB/sec]

We can see above that the srv01-primary-dc is replicating to an-a01n02.

Now, lets protect the server!

Note: If you get the error message "The DR host: [an-a01dr01.alteeve.com] doesn't appear to be in the storage group: [Storage group 1]. Unable to proceed.", please see Managing Storage Groups.
anvil-manage-dr --protect --server srv01-primary-dc --dr-host an-a01dr01
Sanity checks complete!
Beginning to protect the server: [srv01-primary-dc]!
Verified that there is enough space on DR to proceed.
* The connection protocol will be: ..... [short-throw]
* We will update the DRBD resource file: [/etc/drbd.d/srv01-primary-dc.res]
The following LV(s) will be created:

- Resource: [srv01-primary-dc], Volume: [0]
 - The LV: [/dev/rhel_an-a01dr01/srv01-primary-dc_0] with the size: [100.00 GiB (107,374,182,400 Bytes)] will be created.
Proceed? [y/N]


Thank you, proceeding.
The job has been recorded with the UUID: [2e7385e8-9f11-400d-a20d-d8bb54c6002e], it will start in just a moment if anvil-daemon is running.
Note: It might take a minute or two for the server to start being protected, please be patient.

After a minute, we can see that the server is now being sync'ed over to the DR host!

Sync progress:
  Res                 To            Vol
- srv01-primary-dc -> an-a01dr01 ->   0: [0;0]  //  ds:UpToDate/Inconsistent, ro:Primary/Secondary
- srv01-primary-dc ->  an-a01n02 ->   0: [0;0]  //  ds:UpToDate/UpToDate, ro:Primary/Secondary
* Total transfer speed is about: [0 MiB/sec]

In this example, the initial synchronization is running from an-a01n02 to an-a01dr01. So if we switch over to the DR host, we can watch the actual transfer progress.

Sync progress:
  Res                 To           Vol
- srv01-primary-dc -> an-a01n01 ->   0: [0;0]  //  ds:Inconsistent/UpToDate, ro:Secondary/Primary
- srv01-primary-dc -> an-a01n02 ->   0: [==>.................] 18.0% 118,888 KiB/Sec (ETA 0:12:03) //  ds:Inconsistent/UpToDate, ro:Secondary/Secondary
* Total transfer speed is about: [116.101 MiB/sec]

How long this initial synchronization takes will depend on the replication link speed and the amount of data to be copied. After a while though, the sync will be complete, and from then on, the server will be streaming to the DR host.


Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche! Enterprise Support:
Alteeve Support
Community Support
© Alteeve's Niche! Inc. 1997-2024   Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.