Red Hat Cluster Service 2 Tutorial - Archive

AN!Wiki :: How To :: Red Hat Cluster Service 2 Tutorial - Archive

Note: This is the second edition of the original Red Hat Cluster Service 2 Tutorial. This version is updated to use the Red Hat Cluster Suite, Stable version 3. It replaces Xen in favour of KVM to stay in-line with Red Hat's supported configuration. It also uses corosync, replacing openais, as the core cluster communication stack.

This paper has one goal;

• Creating a 2-node, high-availability cluster hosting KVM virtual machines using RHCS "stable 3" with DRBD and clustered LVM for synchronizing storage data. This is an updated version of the earlier Red Hat Cluster Service 2 Tutorial Tutorial. You will find much in common with that tutorial if you've previously followed that document. Please don't skip large sections though. There are some differences that are subtle but important.

Grab a coffee, put on some nice music and settle in for some geekly fun.

The Task Ahead

Before we start, let's take a few minutes to discuss clustering and it's complexities.

Technologies We Will Use

• Red Hat Enterprise Linux 6 (EL6); You can use a derivative like CentOS v6.
• Red Hat Cluster Services "Stable" version 3. This describes the following core components:
  • Corosync; Provides cluster communications using the totem protocol.
  • Cluster Manager (cman); Manages the starting, stopping and managing of the cluster.
  • Resource Manager (rgmanager); Manages cluster resources and services. Handles service recovery during failures.
  • Clustered Logical Volume Manager (clvm); Cluster-aware (disk) volume manager. Backs GFS2 filesystems and KVM virtual machines.
  • Global File Systems version 2 (gfs2); Cluster-aware, concurrently mountable file system.
• Distributed Redundant Block Device (DRBD); Keeps shared data synchronized across cluster nodes.
• KVM; Hypervisor that controls and supports virtual machines.

A Note on Patience

There is nothing inherently hard about clustering. However, there are many components that you need to understand before you can begin. The result is that clustering has an inherently steep learning curve.

You must have patience. Lots of it.

Many technologies can be learned by creating a very simple base and then building on it. The classic "Hello, World!" script created when first learning a programming language is an example of this. Unfortunately, there is no real analogue to this in clustering. Even the most basic cluster requires several pieces be in place and working together. If you try to rush by ignoring pieces you think are not important, you will almost certainly waste time. A good example is setting aside fencing, thinking that your test cluster's data isn't important. The cluster software has no concept of "test". It treats everything as critical all the time and will shut down if anything goes wrong.

Take your time, work through these steps, and you will have the foundation cluster sooner than you realize. Clustering is fun because it is a challenge.

Prerequisites

It is assumed that you are familiar with Linux systems administration, specifically Red Hat Enterprise Linux and its derivatives. You will need to have somewhat advanced networking experience as well. You should be comfortable working in a terminal (directly or over ssh). Familiarity with XML will help, but is not terribly required as it's use here is pretty self-evident.

If you feel a little out of depth at times, don't hesitate to set this tutorial aside. Branch over to the components you feel the need to study more, then return and continue on. Finally, and perhaps most importantly, you must have patience! If you have a manager asking you to "go live" with a cluster in a month, tell him or her that it simply won't happen. If you rush, you will skip important points and you will fail.

Patience is vastly more important than any pre-existing skill.

Focus and Goal

There is a different cluster for every problem. Generally speaking though, there are two main problems that clusters try to resolve; Performance and High Availability. Performance clusters are generally tailored to the application requiring the performance increase. There are some general tools for performance clustering, like Red Hat's LVS (Linux Virtual Server) for load-balancing common applications like the Apache web-server.

This tutorial will focus on High Availability clustering, often shortened to simply HA and not to be confused with the Linux-HA "heartbeat" cluster suite, which we will not be using here. The cluster will provide a shared file systems and will provide for the high availability on KVM-based virtual servers. The goal will be to have the virtual servers live-migrate during planned node outages and automatically restart on a surviving node when the original host node fails.

Below is a very brief overview;

High Availability clusters like ours have two main parts; Cluster management and resource management.

The cluster itself is responsible for maintaining the cluster nodes in a group. This group is part of a "Closed Process Group", or CPG. When a node fails, the cluster manager must detect the failure, reliably eject the node from the cluster using fencing and then reform the CPG. Each time the cluster changes, or "re-forms", the resource manager is called. The resource manager checks to see how the cluster changed, consults it's configuration and determines what to do, if anything.

The details of all this will be discussed in detail a little later on. For now, it's sufficient to have in mind these two major roles and understand that they are somewhat independent entities.

Platform

This tutorial was written using RHEL version 6.1 and CentOS version 6.0 x86_64. No attempt was made to test on i686 or other EL6 derivatives. That said, there is no reason to believe that this tutorial will not apply to any variant. As much as possible, the language will be distro-agnostic. It is advised that you use an x86_64 (64-bit) platform if at all possible.

A Word On Complexity

Introducing the Fabimer Principle:

Clustering is not inherently hard, but it is inherently complex. Consider;

• Any given program has N bugs.
  • RHCS uses; cman, corosync, dlm, fenced, rgmanager, and many more smaller apps.
  • We will be adding DRBD, GFS2, clvmd, libvirtd and KVM.
  • Right there, we have N^10 possible bugs. We'll call this A.
• A cluster has Y nodes.
  • In our case, 2 nodes, each with 3 networks across 6 interfaces bonded into pairs.
  • The network infrastructure (Switches, routers, etc). We will be using two managed switches, adding another layer of complexity.
  • This gives us another Y^(2*(3*2))+2, the +2 for managed switches. We'll call this B.
• Let's add the human factor. Let's say that a person needs roughly 5 years of cluster experience to be considered an proficient. For each year less than this, add a Z "oops" factor, (5-Z)^2. We'll call this C.
• So, finally, add up the complexity, using this tutorial's layout, 0-years of experience and managed switches.
  • (N^10) * (Y^(2*(3*2))+2) * ((5-0)^2) == (A * B * C) == an-unknown-but-big-number.

This isn't meant to scare you away, but it is meant to be a sobering statement. Obviously, those numbers are somewhat artificial, but the point remains.

Any one piece is easy to understand, thus, clustering is inherently easy. However, given the large number of variables, you must really understand all the pieces and how they work together. DO NOT think that you will have this mastered and working in a month. Certainly don't try to sell clusters as a service without a lot of internal testing.

Clustering is kind of like chess. The rules are pretty straight forward, but the complexity can take some time to master.

Overview of Components

When looking at a cluster, there is a tendency to want to dive right into the configuration file. That is not very useful in clustering.

• When you look at the configuration file, it is quite short.

It isn't like most applications or technologies though. Most of us learn by taking something, like a configuration file, and tweaking it this way and that to see what happens. I tried that with clustering and learned only what it was like to bang my head against the wall.

• Understanding the parts and how they work together is critical.

You will find that the discussion on the components of clustering, and how those components and concepts interact, will be much longer than the initial configuration. It is true that we could talk very briefly about the actual syntax, but it would be a disservice. Please, don't rush through the next section or, worse, skip it and go right to the configuration. You will waste far more time than you will save.

• Clustering is easy, but it has a complex web of inter-connectivity. You must grasp this network if you want to be an effective cluster administrator!

Component; cman

This was, traditionally, the cluster manager. In the 3.0 series, which is what all versions of EL6 will use, cman acts mainly as a quorum provider, tallying votes and deciding on a critical property of the cluster: quorum. As of the 3.1 series, which future EL releases will use, cman will be removed entirely.

The cman service is used to start and stop the cluster communication, membership, locking, fencing and other cluster foundation applications.

Component; corosync

Corosync is the heart of the cluster. Almost all other cluster compnents operate though this.

In Red Hat clusters, corosync is configured via the central cluster.conf file. It can be configured directly in corosync.conf, but given that we will be building an RHCS cluster, we will only use cluster.conf. That said, almost all corosync.conf options are available in cluster.conf. This is important to note as you will see references to both configuration files when searching the Internet.

Corosync sends messages using multicast messaging by default. Recently, unicast support has been added, but due to network latency, it is only recommended for use with small clusters of two to four nodes. We will be using multicast in this tutorial.

A Little History

There were significant changes between RHCS version 2, which we are using, and version 3 available on EL6 and recent Fedoras.

In the RHCS version 2, there was a component called openais which provided totem. The OpenAIS project was designed to be the heart of the cluster and was based around the Service Availability Forum's Application Interface Specification. AIS is an open API designed to provide inter-operable high availability services.

In 2008, it was decided that the AIS specification was overkill for most clustered applications being developed in the open source community. At that point, OpenAIS was split in to two projects: Corosync and OpenAIS. The former, Corosync, provides totem, cluster membership, messaging, and basic APIs for use by clustered applications, while the OpenAIS project became an optional add-on to corosync for users who want the full AIS API.

You will see a lot of references to OpenAIS while searching the web for information on clustering. Understanding it's evolution will hopefully help you avoid confusion.

Concept; quorum

Quorum is defined as the minimum set of hosts required in order to provide clustered services and is used to prevent split-brain situations.

The quorum algorithm used by the RHCS cluster is called "simple majority quorum", which means that more than half of the hosts must be online and communicating in order to provide service. While simple majority quorum a very common quorum algorithm, other quorum algorithms exist (grid quorum, YKD Dyanamic Linear Voting, etc.).

The idea behind quorum is that, when a cluster splits into two or more partitions, which ever group of machines has quorum can safely start clustered services knowing that no other lost nodes will try to do the same.

Take this scenario;

• You have a cluster of four nodes, each with one vote.
  • The cluster's expected_votes is 4. A clear majority, in this case, is 3 because (4/2)+1, rounded down, is 3.
  • Now imagine that there is a failure in the network equipment and one of the nodes disconnects from the rest of the cluster.
  • You now have two partitions; One partition contains three machines and the other partition has one.
  • The three machines will have quorum, and the other machine will lose quorum.
  • The partition with quorum will reconfigure and continue to provide cluster services.
  • The partition without quorum will withdraw from the cluster and shut down all cluster services.

When the cluster reconfigures and the partition wins quorum, it will fence the node(s) in the partition without quorum. Once the fencing has been confirmed successful, the partition with quorum will begin accessing clustered resources, like shared filesystems.

This also helps explain why an even 50% is not enough to have quorum, a common question for people new to clustering. Using the above scenario, imagine if the split were 2 and 2 nodes. Because either can't be sure what the other would do, neither can safely proceed. If we allowed an even 50% to have quorum, both partition might try to take over the clustered services and disaster would soon follow.

There is one, and only one except to this rule.

In the case of a two node cluster, as we will be building here, any failure results in a 50/50 split. If we enforced quorum in a two-node cluster, there would never be high availability because and failure would cause both nodes to withdraw. The risk with this exception is that we now place the entire safety of the cluster on fencing, a concept we will cover in a second. Fencing is a second line of defense and something we are loath to rely on alone.

Even in a two-node cluster though, proper quorum can be maintained by using a quorum disk, called a qdisk. Unfortunately, qdisk on a DRBD resource comes with it's own problems, so we will not be able to use it here.

Concept; Virtual Synchrony

Many cluster operations, like fencing, distributed locking and so on, have to occur in the same order across all nodes. This concept is called "virtual synchrony".

This is provided by corosync using "closed process groups", CPG. A closed process group is simply a private group of processes in a cluster. Within this closed group, all messages between members are ordered. Delivery, however, is not guaranteed. If a member misses messages, it is up to the member's application to decide what action to take.

Let's look at two scenarios showing how locks are handled using CPG;

• The cluster starts up cleanly with two members.
• Both members are able to start service:foo.
• Both want to start it, but need a lock from DLM to do so.
  • The an-node01 member has it's totem token, and sends it's request for the lock.
  • DLM issues a lock for that service to an-node01.
  • The an-node02 member requests a lock for the same service.
  • DLM rejects the lock request.
• The an-node01 member successfully starts service:foo and announces this to the CPG members.
• The an-node02 sees that service:foo is now running on an-node01 and no longer tries to start the service.

• The two members want to write to a common area of the /shared GFS2 partition.
  • The an-node02 sends a request for a DLM lock against the FS, gets it.
  • The an-node01 sends a request for the same lock, but DLM sees that a lock is pending and rejects the request.
  • The an-node02 member finishes altering the file system, announces the changed over CPG and releases the lock.
  • The an-node01 member updates it's view of the filesystem, requests a lock, receives it and proceeds to update the filesystems.
  • It completes the changes, annouces the changes over CPG and releases the lock.

Messages can only be sent to the members of the CPG while the node has a totem tokem from corosync.

Concept; Fencing

Fencing is a absolutely critical part of clustering. Without fully working fence devices, your cluster will fail.

Was that strong enough, or should I say that again? Let's be safe:

DO NOT BUILD A CLUSTER WITHOUT PROPER, WORKING AND TESTED FENCING.

Sorry, I promise that this will be the only time that I speak so strongly. Fencing really is critical, and explaining the need for fencing is nearly a weekly event.

So then, let's discuss fencing.

When a node stops responding, an internal timeout and counter start ticking away. During this time, no DLM locks are allowed to be issued. Anything using DLM, including rgmanager, clvmd and gfs2, are effectively hung. The hung node is detected using a totem token timeout. That is, if a token is not received from a node within a period of time, it is considered lost and a new token is sent. After a certain number of lost tokens, the cluster declares the node dead. The remaining nodes reconfigure into a new cluster and, if they have quorum (or if quorum is ignored), a fence call against the silent node is made.

The fence daemon will look at the cluster configuration and get the fence devices configured for the dead node. Then, one at a time and in the order that they appear in the configuration, the fence daemon will call those fence devices, via their fence agents, passing to the fence agent any configured arguments like username, password, port number and so on. If the first fence agent returns a failure, the next fence agent will be called. If the second fails, the third will be called, then the forth and so on. Once the last (or perhaps only) fence device fails, the fence daemon will retry again, starting back at the start of the list. It will do this indefinitely until one of the fence devices success.

Here's the flow, in point form:

• The totem token moves around the cluster members. As each member gets the token, it sends sequenced messages to the CPG members.
• The token is passed from one node to the next, in order and continuously during normal operation.
• Suddenly, one node stops responding.
  • A timeout starts (~238ms by default), and each time the timeout is hit, and error counter increments and a replacement token is created.
  • The silent node responds before the failure counter reaches the limit.
    • The failure counter is reset to 0
    • The cluster operates normally again.
• Again, one node stops responding.
  • Again, the timeout begins. As each totem token times out, a new packet is sent and the error count increments.
  • The error counts exceed the limit (4 errors is the default); Roughly one second has passed (238ms * 4 plus some overhead).
  • The node is declared dead.
  • The cluster checks which members it still has, and if that provides enough votes for quorum.
    • If there are too few votes for quorum, the cluster software freezes and the node(s) withdraw from the cluster.
    • If there are enough votes for quorum, the silent node is declared dead.
      • corosync calls fenced, telling it to fence the node.
      • The fenced daemon notifies DLM and locks are blocked.
      • Which fence device(s) to use, that is, what fence_agent to call and what arguments to pass, is gathered.
      • For each configured fence device:
        • The agent is called and fenced waits for the fence_agent to exit.
        • The fence_agent's exit code is examined. If it's a success, recovery starts. If it failed, the next configured fence agent is called.
      • If all (or the only) configured fence fails, fenced will start over.
      • fenced will wait and loop forever until a fence agent succeeds. During this time, the cluster is effectively hung.
    • Once a fence_agent succeeds, fenced notifies DLM and lost locks are recovered.
      • GFS2 partitions recover using their journal.
      • Lost cluster resources are recovered as per rgmanager's configuration (including file system recovery as needed).
• Normal cluster operation is restored, minus the lost node.

This skipped a few key things, but the general flow of logic should be there.

This is why fencing is so important. Without a properly configured and tested fence device or devices, the cluster will never successfully fence and the cluster will remain hung until a human can intervene.

Component; totem

The totem protocol defines message passing within the cluster and it is used by corosync. A token is passed around all the nodes in the cluster, and nodes can only send messages while they have the token. A node will keep it's messages in memory until it gets the token back with no "not ack" messages. This way, if a node missed a message, it can request it be resent when it gets it's token. If a node isn't up, it will simply miss the messages.

The totem protocol supports something called 'rrp', Redundant Ring Protocol. Through rrp, you can add a second backup ring on a separate network to take over in the event of a failure in the first ring. In RHCS, these rings are known as "ring 0" and "ring 1". The RRP is being re-introduced in RHCS version 3. It's use is experimental and should only be used with plenty of testing.

Component; rgmanager

When the cluster membership changes, corosync tells the rgmanager that it needs to recheck it's services. It will examine what changed and then will start, stop, migrate or recover cluster resources as needed.

Within rgmanager, one or more resources are brought together as a service. This service is then optionally assigned to a failover domain, an subset of nodes that can have preferential ordering.

The rgmanager daemon runs separately from the cluster manager, cman. This means that, to fully start the cluster, we need to start both cman and then rgmanager.

Component; qdisk

Note: qdisk does not work reliably on a DRBD resource, so we will not be using it in this tutorial.

A Quorum disk, known as a qdisk is small partition on SAN storage used to enhance quorum. It generally carries enough votes to allow even a single node to take quorum during a cluster partition. It does this by using configured heuristics, that is custom tests, to decided which which node or partition is best suited for providing clustered services during a cluster reconfiguration. These heuristics can be simple, like testing which partition has access to a given router, or they can be as complex as the administrator wishes using custom scripts.

Though we won't be using it here, it is well worth knowing about when you move to a cluster with SAN storage.

Component; DRBD

DRBD; Distributed Replicating Block Device, is a technology that takes raw storage from two or more nodes and keeps their data synchronized in real time. It is sometimes described as "RAID 1 over Cluster Nodes", and that is conceptually accurate. In this tutorial's cluster, DRBD will be used to provide that back-end storage as a cost-effective alternative to a traditional SAN device.

To help visualize DRBD's use and role, Take a look at how we will implement our cluster's storage.

This shows;

• Each node having four physical disks tied together in a RAID Level 5 array and presented to the Node's OS as a single drive which is found at /dev/sda.
• Each node's OS uses three primary partitions for /boot, <swap> and /.
• Three extended partitions are created;
  • /dev/sda5 backs a small partition used as a GFS2-formatted shared mount point.
  • /dev/sda6 backs the VMs designed to run primarily on an-node01.
  • /dev/sda7 backs the VMs designed to run primarily on an-node02.
• All three extended partitions are combined using DRBD to create three DRBD resources;
  • /dev/drbd0 is backed by /dev/sda5.
  • /dev/drbd1 is backed by /dev/sda6.
  • /dev/drbd2 is backed by /dev/sda7.
• All three DRBD resources are managed by clustered LVM.
• The GFS2-formatted LV is mounted on /shared on both nodes.
• Each VM gets it's own LV.
• All three DRBD resources sync over the Storage Network, which uses the bonded bond1 (backed be eth1 and eth4).

Don't worry if this seems illogical at this stage. The main thing to look at are the drbdX devices and how they each tie back to a corresponding sdaY device on either node.

 _________________________________________________                 _________________________________________________ 
| [ an-node01 ]                                   |               |                                   [ an-node02 ] |
|  ________       __________                      |               |                      __________       ________  |
| [_disk_1_]--+--[_/dev/sda_]                     |               |                     [_/dev/sda_]--+--[_disk_1_] |
|  ________   |    |   ___________    _______     |               |     _______    ___________   |    |   ________  |
| [_disk_2_]--+    +--[_/dev/sda1_]--[_/boot_]    |               |    [_/boot_]--[_/dev/sda1_]--+    +--[_disk_2_] |
|  ________   |    |   ___________    ________    |               |    ________    ___________   |    |   ________  |
| [_disk_3_]--+    +--[_/dev/sda2_]--[_<swap>_]   |               |   [_<swap>_]--[_/dev/sda2_]--+    +--[_disk_3_] |
|  ________   |    |   ___________    ___         |               |         ___    ___________   |    |   ________  |
| [_disk_4_]--/    +--[_/dev/sda3_]--[_/_]        |               |        [_/_]--[_/dev/sda3_]--+    \--[_disk_4_] |
|                  |   ___________                |               |                ___________   |                  |
|                  +--[_/dev/sda5_]------------\  |               |  /------------[_/dev/sda5_]--+                  |
|                  |   ___________             |  |               |  |             ___________   |                  |
|                  +--[_/dev/sda6_]----------\ |  |               |  | /----------[_/dev/sda6_]--+                  |
|                  |   ___________           | |  |               |  | |           ___________   |                  |
|                  \--[_/dev/sda7_]--------\ | |  |               |  | | /--------[_/dev/sda7_]--/                  |
|        _______________    ____________   | | |  |               |  | | |   ____________    _______________        |
|    /--[_Clustered_LVM_]--[_/dev/drbd2_]--/ | |  |               |  | | \--[_/dev/drbd2_]--[_Clustered_LVM_]--\    |
|   _|__                     |   _______     | |  |               |  | |      |   _______                    __|_   |
|  [_PV_]                    \--{_bond1_}    | |  |               |  | |      \--{_bond1_}                  [_PV_]  |
|   _|_______                                | |  |               |  | |                                _______|_   |
|  [_an2-vg0_]                               | |  |               |  | |                               [_an2-vg0_]  |
|    |   _______________________    .......  | |  |               |  | |   _____     _______________________   |    |
|    +--[_/dev/an2-vg0/vm0003_1_]---:.vm3.:  | |  |               |  | |  [_vm3_]---[_/dev/an2-vg0/vm0003_1_]--+    |
|    |   _______________________    .......  | |  |               |  | |   _____     _______________________   |    |
|    \--[_/dev/an2-vg0/vm0004_1_]---:.vm4.:  | |  |               |  | |  [_vm4_]---[_/dev/an2-vg0/vm0004_1_]--/    |
|          _______________    ____________   | |  |               |  | |   ____________    _______________          |
|      /--[_Clustered_LVM_]--[_/dev/drbd1_]--/ |  |               |  | \--[_/dev/drbd1_]--[_Clustered_LVM_]--\      |
|     _|__                     |   _______     |  |               |  |      |   _______                    __|_     |
|    [_PV_]                    \--{_bond1_}    |  |               |  |      \--{_bond1_}                  [_PV_]    |
|     _|_______                                |  |               |  |                                ___ ___|_     |
|    [_an1-vg0_]                               |  |               |  |                               [_an1-vg0_]    |
|      |   _______________________     _____   |  |               |  |      .......    ___________________   |      |
|      +--[_/dev/an1-vg0/vm0001_1_]---[_vm1_]  |  |               |  |      :.vm1.:---[_/dev/vg0/vm0001_1_]--+      |
|      |   _______________________     _____   |  |               |  |      .......    ___________________   |      |
|      \--[_/dev/an1-vg0/vm0002_1_]---[_vm2_]  |  |               |  |      :.vm2.:---[_/dev/vg0/vm0002_1_]--/      |
|            _______________    ____________   |  |               |  |   ____________    _______________            |
|        /--[_Clustered_LVM_]--[_/dev/drbd0_]--/  |               |  \--[_/dev/drbd0_]--[_Clustered_LVM_]--\        |
|       _|__                     |   _______      |               |       |   _______                    __|_       |
|      [_PV_]                    \--{_bond1_}     |               |       \--{_bond1_}                  [_PV_]      |
|       _|__________                              |               |                              __________|_       |
|      [_shared-vg0_]                             |               |                             [_shared-vg0_]      |
|       _|_________________________               |               |               _________________________|_       |
|      [_/dev/shared-vg0/lv_shared_]              |               |              [_/dev/shared-vg0/lv_shared_]      |
|        |   ______    _________                  |               |                  _________    ______   |        |
|        \--[_GFS2_]--[_/shared_]                 |               |                 [_/shared_]--[_GFS2_]--/        |
|                                          _______|   _________   |_______                                          |
|                                         | bond1 =--| Storage |--= bond1 |                                         |
|                                         |______||  | Network |  ||______|                                         |
|_________________________________________________|  |_________|  |_________________________________________________|
.

Component; Clustered LVM

With DRBD providing the raw storage for the cluster, we must next consider partitions. This is where Clustered LVM, known as CLVM, comes into play.

CLVM is ideal in that by using DLM, the distributed lock manager. It won't allow access to cluster members outside of corosync's closed process group, which, in turn, requires quorum.

It is ideal because it can take one or more raw devices, known as "physical volumes", or simple as PVs, and combine their raw space into one or more "volume groups", known as VGs. These volume groups then act just like a typical hard drive and can be "partitioned" into one or more "logical volumes", known as LVs. These LVs are where KVM's virtual machine guests will exist and where we will create our GFS2 clustered file system.

LVM is particularly attractive because of how flexible it is. We can easily add new physical volumes later, and then grow an existing volume group to use the new space. This new space can then be given to existing logical volumes, or entirely new logical volumes can be created. This can all be done while the cluster is online offering an upgrade path with no down time.

Component; GFS2

With DRBD providing the clusters raw storage space, and Clustered LVM providing the logical partitions, we can now look at the clustered file system. This is the role of the Global File System version 2, known simply as GFS2.

It works much like standard filesystem, with user-land tools like mkfs.gfs2, fsck.gfs2 and so on. The major difference is that it and clvmd use the cluster's distributed locking mechanism provided by the dlm_controld daemon. Once formatted, the GFS2-formatted partition can be mounted and used by any node in the cluster's closed process group. All nodes can then safely read from and write to the data on the partition simultaneously.

Component; DLM

One of the major roles of a cluster is to provide distributed locking for clustered storage and resource management.

Whenever a resource, GFS2 filesystem or clustered LVM LV needs a lock, it sends a request to dlm_controld which runs in userspace. This communicates with DLM in kernel. If the lock group does not yet exist, DLM will create it and then give the lock to the requester. Should a subsequant lock request come in for the same lock group, it will be rejected. Once the application using the lock is finished with it, it will release the lock. After this, another node may request and receive a lock for the lock group.

If a node fails, fenced will alert dlm_controld that a fence is pending and new lock requests will block. After a successful fence, fenced will alert DLM that the node is gone and any locks the victim node held are released. At this time, other nodes may request a lock on the lock groups the lost node held and can perform recovery, like replaying a GFS2 filesystem journal, prior to resuming normal operation.

Note that DLM locks are not used for actually locking the file system. That job is still handled by plock() calls (POSIX locks).

Component; KVM

Two of the most popular open-source virtualization platforms available in the Linux world today and Xen and KVM. The former is maintained by Citrix and the other by Redhat. It would be difficult to say which is "better", as they're both very good. Xen can be argued to be more mature where KVM is the "official" solution supported by Red Hat in EL6.

We will be using the KVM hypervisor within which our highly-available virtual machine guests will reside. It is a type-2 hypervisor, which means that the host operating system runs directly on the bare hardware. Contrasted against Xen, which is a type-1 hypervisor where even the installed OS is itself just another virtual machine.

Network

The cluster will use three Class B networks, broken down as:

Purpose	Subnet	Notes
Internet-Facing Network (IFN)	10.255.0.0/16	Each node will use 10.255.0.x where x matches the node ID. Virtual Machines in the cluster that need to be connected to the Internet will use 10.255.y.z where y corresponds to the cluster and z is a simple sequence number matching the VM ID.
Storage Network (SN)	10.10.0.0/16	Each node will use 10.10.0.x where x matches the node ID.
Back-Channel Network (BCN)	10.20.0.0/16	Each node will use 10.20.0.x where x matches the node ID. Node-specific IPMI or other out-of-band management devices will use 10.20.1.x where x matches the node ID. Multi-port fence devices will use 10.20.2.z where z is a simple sequence. Miscellaneous equipment in the cluster, like managed switches, will use 10.20.3.z where z is a simple sequence.

We will be using six interfaces, bonded into three pairs of two NICs in Active/Passive (mode 1). Each link of each bond will be on alternate, unstacked switches. This configuration is the only configuration supported by Red Hat in clusters.

Note: Only the bonded interface used by corosync must be in Active/Passive configuration (bond0 in this tutorial). If you want to experiment with other bonding modes for bond1 or bond2, please feel free to do so. That is outside the scope of this tutorial, however.

If you can not install six interfaces in your server, then four interfaces will do with the SN and BCN networks merged.

In this tutorial, we will use two D-Link DGS-3100-24, unstacked, using three VLANs to isolate the three networks.

You could just as easily use four or six unmanaged 5 port or 8 port switches. What matters is that the three subnets are isolated and that each link of each bond is on a separate switch. Lastly, only connect the IFN switches or VLANs to the Internet for security reasons.

The actual mapping of interfaces to bonds to networks will be:

Subnet	Link 1	Link 2	Bond	IP
BCN	eth0	eth3	bond0	10.20.0.x
SN	eth1	eth4	bond1	10.10.0.x
IFN	eth2	eth5	bond2	10.255.0.x

Setting Up the Network

Warning: The following steps can easily get confusing, given how many files we need to edit. Losing access to your server's network is a very real possibility! Do not continue without direct access to your servers! If you have out-of-band access via iKVM, console redirection or similar, be sure to test that it is working before proceeding.

Managed and Stacking Switch Notes

Note: If you have two stacked switches, do not stack them!

There are two things you need to be wary of with managed switches.

• Don't stack them. It may seem like it makes sense to stack them and create Link Aggregation Groups, but this is not supported. Leave the two switches as independent units.
• Disable Spanning Tree Protocol on all ports used by the cluster. Otherwise, when a lost switch is recovered, STP negotiation will cause traffic to stop on the ports for upwards of thirty seconds. This is more than enough time to partition a cluster.

Enable STP on the ports you use for uplinking the two switches and disable it on all other ports.

Making Sure We Know Our Interfaces

When you installed the operating system, the network interfaces names are somewhat randomly assigned to the physical network interfaces. It more than likely that you will want to re-order.

Before you start moving interface names around, you will want to consider which physical interfaces you will want to use on which networks. At the end of the day, the names themselves have no meaning. At the very least though, make them consistent across nodes.

Some things to consider, in order of importance:

• If you have a shared interface for your out-of-band management interface, like IPMI or iLO, you will want that interface to be on the Back-Channel Network.
• For redundancy, you want to spread out which interfaces are paired up. In my case, I have three interfaces on my mainboard and three additional add-in cards. I will pair each onboard interface with an add-in interface. In my case, my IPMI interface physically piggy-backs on one of the onboard interfaces so this interface will need to be part of the BCN bond.
• Your interfaces with the lowest latency should be used for the back-channel network.
• Your two fastest interfaces should be used for your storage network.
• The remaining two slowest interfaces should be used for the Internet-Facing Network bond.

In my case, all six interfaces are identical, so there is little to consider. The left-most interface on my system has IPMI, so it's paired network interface will be eth0. I simply work my way left, incrementing as I go. What you do will be whatever makes most sense to you.

There is a separate, short tutorial on re-ordering network interface;

• Changing the ethX to Ethernet Device Mapping in EL6 and Fedora 12+

Once you have the physical interfaces named the way you like, proceed to the next step.

Planning Our Network

To setup our network, we will need to edit the ifcfg-ethX, ifcfg-bondX and ifcfg-vbrX scripts. The last one will create bridges which will be used to route network connections to the virtual machines. We won't be creating an vbr1 bridge though, and bond1 will be dedicated to the storage and never used by a VM. The bridges will have the IP addresses, not the bonded interfaces. They will instead be slaved to their respective bridges.

We're going to be editing a lot of files. It's best to lay out what we'll be doing in a chart. So our setup will be:

Node	BCN IP and Device	SN IP and Device	IFN IP and Device
an-node01	10.20.0.1 on vbr0 (bond0 slaved)	10.10.0.1 on bond1	10.255.0.1 on vbr2 (bond2 slaved)
an-node02	10.20.0.2 on vbr0 (bond0 slaved)	10.10.0.2 on bond1	10.255.0.2 on vbr2 (bond2 slaved)

Creating Some Network Configuration Files

Bridge configuration files must have a file name that sorts after the interfaces and bridges. The actual device name can be whatever you want though. If the system tries to start the bridge before it's interface is up, it will fail. I personally like to use the name vbrX for "virtual machine bridge". You can use whatever makes sense to you, with the above concern in mind.

Start by touching the configuration files we will need.

touch /etc/sysconfig/network-scripts/ifcfg-bond{0,1,2}
touch /etc/sysconfig/network-scripts/ifcfg-vbr{0,2}

Now make a backup of your configuration files, in case something goes wrong and you want to start over.

mkdir /root/backups/
rsync -av /etc/sysconfig/network-scripts/ifcfg-eth* /root/backups/

sending incremental file list
ifcfg-eth0
ifcfg-eth1
ifcfg-eth2
ifcfg-eth3
ifcfg-eth4
ifcfg-eth5

sent 1467 bytes  received 126 bytes  3186.00 bytes/sec
total size is 1119  speedup is 0.70

Configuring Our Bridges

Now lets start in reverse order. We'll write the bridge configuration, then the bond interfaces and finally alter the interface configuration files. The reason for doing this in reverse is to minimize the amount of time where a sudden restart would leave us without network access.

an-node01 BCN Bridge:

vim /etc/sysconfig/network-scripts/ifcfg-vbr0

# Back-Channel Network - Bridge
DEVICE="vbr0"
TYPE="Bridge"
BOOTPROTO="static"
IPADDR="10.20.0.1"
NETMASK="255.255.0.0"

an-node01 IFN Bridge:

vim /etc/sysconfig/network-scripts/ifcfg-vbr2

# Internet-Facing Network - Bridge
DEVICE="vbr2"
TYPE="Bridge"
BOOTPROTO="static"
IPADDR="10.255.0.1"
NETMASK="255.255.0.0"
GATEWAY="10.255.255.254"
DNS1="192.139.81.117"
DNS2="192.139.81.1"
DEFROUTE="yes"

Creating the Bonded Interfaces

Now we can create the actual bond configuration files.

To explain the BONDING_OPTS options;

• mode=1 sets the bonding mode to active-backup.
• The miimon=100 tells the bonding driver to check if the network cable has been unplugged or plugged in every 100 milliseconds.
• The use_carrier=1 tells the driver to use the driver to maintain the link state. Some drivers don't support that. If you run into trouble, try changing this to 0.
• The updelay=0 and downdelay=0 tells the driver not to wait before changing the state of an interface when the link goes up or down. There are some cases where the link will report that it is ready before the switch can push data. Other times, you might have the switch drop the link for a very brief time. Altering these values gives you a chance to either delay the use of an interface until the switch is ready or prevent dropping a link when the link is just bouncing.

an-node01 BCN Bond:

vim /etc/sysconfig/network-scripts/ifcfg-bond0

# Back-Channel Network - Bond
DEVICE="bond0"
BRIDGE="vbr0"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=0 downdelay=0"

an-node01 SN Bond:

vim /etc/sysconfig/network-scripts/ifcfg-bond1

# Storage Network - Bond
DEVICE="bond1"
BOOTPROTO="static"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=0 downdelay=0"
IPADDR="10.10.0.1"
NETMASK="255.255.0.0"

an-node01 IFN Bond:

vim /etc/sysconfig/network-scripts/ifcfg-bond2

# Internet-Facing Network - Bond
DEVICE="bond2"
BRIDGE="vbr2"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=0 downdelay=0"

Alter The Interface Configurations

Now, finally, alter the interfaces themselves to join their respective bonds.

an-node01's eth0, the BCN bond0, Link 1:

vim /etc/sysconfig/network-scripts/ifcfg-eth0

# Back-Channel Network - Link 1
HWADDR="00:E0:81:C7:EC:49"
DEVICE="eth0"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond0"
SLAVE="yes"

an-node01's eth1, the SN bond1, Link 1:

vim /etc/sysconfig/network-scripts/ifcfg-eth0

# Storage Network - Link 1
HWADDR="00:E0:81:C7:EC:48"
DEVICE="eth1"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond1"
SLAVE="yes"

an-node01's eth2, the IFN bond2, Link 1:

vim /etc/sysconfig/network-scripts/ifcfg-eth0

# Internet-Facing Network - Link 1
HWADDR="00:E0:81:C7:EC:47"
DEVICE="eth2"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond2"
SLAVE="yes"

an-node01's eth3, the BCN bond0, Link 2:

vim /etc/sysconfig/network-scripts/ifcfg-eth3

# Back-Channel Network - Link 2
DEVICE="eth3"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond0"
SLAVE="yes"

an-node01's eth4, the SN bond1, Link 2:

vim /etc/sysconfig/network-scripts/ifcfg-eth0

# Storage Network - Link 2
HWADDR="00:1B:21:BF:6F:FE"
DEVICE="eth4"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond1"
SLAVE="yes"

an-node01's eth5, the IFN bond2, Link 2:

vim /etc/sysconfig/network-scripts/ifcfg-eth0

# Internet-Facing Network - Link 2
HWADDR="00:1B:21:BF:70:02"
DEVICE="eth5"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond2"
SLAVE="yes"

Loading The New Network Configuration

Simple restart the network service.

/etc/init.d/network restart

Updating /etc/hosts

On both nodes, update the /etc/hosts file to reflect your network configuration. Remember to add entries for your IPMI, switched PDUs and other devices.

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

# an-node01
10.20.0.1	an-node01 an-node01.bcn an-node01.alteeve.com
10.20.1.1	an-node01.ipmi
10.10.0.1	an-node01.sn
10.255.0.1	an-node01.ifn

# an-node01
10.20.0.2	an-node02 an-node02.bcn an-node02.alteeve.com
10.20.1.2	an-node02.ipmi
10.10.0.2	an-node02.sn
10.255.0.2	an-node02.ifn

# Fence devices
10.20.2.1       pdu1 pdu1.alteeve.com
10.20.2.2       pdu2 pdu2.alteeve.com

Warning: Which ever switch you have the IPMI interfaces connected to, be sure to connect the PDU into the opposite switch! If both fence types are on one switch, then that switch becomes a single point of failure again.

Configuring The Cluster Foundation

We need to configure the cluster in two stages. This is because we have something of a chicken-and-egg problem.

• We need clustered storage for our virtual machines.
• Our clustered storage needs the cluster for fencing.

Conveniently, clustering has two logical parts;

• Cluster communication and membership.
• Cluster resource management.

The first, communication and membership, covers which nodes are part of the cluster and ejecting faulty nodes from the cluster, among other tasks. The second part, resource management, is provided by a second tool called rgmanager. It's this second part that we will set aside for later.

Installing Required Programs

Installing the cluster software is pretty simple;

yum install cman corosync rgmanager ricci gfs2-utils

Configuration Methods

In Red Hat Cluster Services, the heart of the cluster is found in the /etc/cluster/cluster.conf XML configuration file.

There are three main ways of editing this file. Two are already well documented, so I won't bother discussing them, beyond introducing them. The third way is by directly hand-crafting the cluster.conf file. This method is not very well documented, and directly manipulating configuration files is my preferred method. As my boss loves to say; "The more computers do for you, the more they do to you".

The first two, well documented, graphical tools are:

• system-config-cluster, older GUI tool run directly from one of the cluster nodes.
• Conga, comprised of the ricci node-side client and the luci web-based server (can be run on machines outside the cluster).

I do like the tools above, but I often find issues that send me back to the command line. I'd recommend setting them aside for now as well. Once you feel comfortable with cluster.conf syntax, then by all means, go back and use them. I'd recommend not relying on them though, which might be the case if you try to use them too early in your studies.

The First cluster.conf Foundation Configuration

The very first stage of building the cluster is to create a configuration file that is as minimal as possible. To do that, we need to define a few thing;

• The name of the cluster and the cluster file version.
  • Define cman options
  • The nodes in the cluster
    • The fence method for each node
  • Define fence devices
  • Define fenced options

That's it. Once we've defined this minimal amount, we will be able to start the cluster for the first time! So lets get to it, finally.

Name the Cluster and Set The Configuration Version

The cluster tag is the parent tag for the entire cluster configuration file.

vim /etc/cluster/cluster.conf

<?xml version="1.0"?>
<cluster name="an-clusterA" config_version="1">
</cluster>

This has two attributes that we need to set are name="" and config_version="".

The name="" attribute defines the name of the cluster. It must be unique amongst the clusters on your network. It should be descriptive, but you will not want to make it too long, either. You will see this name in the various cluster tools and you will enter in, for example, when creating a GFS2 partition later on. This tutorial uses the cluster name an_clusterA. The reason for the A is to help differentiate it from the nodes which use sequence numbers.

The config_version="" attribute is an integer marking the version of the configuration file. Whenever you make a change to the cluster.conf file, you will need to increment this version number by 1. If you don't increment this number, then the cluster tools will not know that the file needs to be reloaded. As this is the first version of this configuration file, it will start with 1. Note that this tutorial will increment the version after every change, regardless of whether it is explicitly pushed out to the other nodes and reloaded. The reason is to help get into the habit of always increasing this value.

Configuring cman Options

We are going to setup a special case for our cluster; A 2-Node cluster.

This is a special case because traditional quorum will not be useful. With only two nodes, each having a vote of 1, the total votes is 2. Quorum needs 50% + 1, which means that a single node failure would shut down the cluster, as the remaining node's vote is 50% exactly. That kind of defeats the purpose to having a cluster at all.

So to account for this special case, there is a special attribute called two_node="1". This tells the cluster manager to continue operating with only one vote. This option requires that the expected_votes="" attribute be set to 1. Normally, expected_votes is set automatically to the total sum of the defined cluster nodes' votes (which itself is a default of 1). This is the other half of the "trick", as a single node's vote of 1 now always provides quorum (that is, 1 meets the 50% + 1 requirement).

<?xml version="1.0"?>
<cluster name="an-clusterA" config_version="2">
	<cman expected_votes="1" two_node="1"/>
</cluster>

Take note of the self-closing <... /> tag. This is an XML syntax that tells the parser not to look for any child or a closing tags.

Defining Cluster Nodes

This example is a little artificial, please don't load it into your cluster as we will need to add a few child tags, but one thing at a time.

This actually introduces two tags.

The first is parent clusternodes tag, which takes no variables of it's own. It's sole purpose is to contain the clusternode child tags.

<?xml version="1.0"?>
<cluster name="an-clusterA" config_version="3">
	<cman expected_votes="1" two_node="1"/>
	<clusternodes>
		<clusternode name="an-node01.alteeve.com" nodeid="1" />
		<clusternode name="an-node02.alteeve.com" nodeid="2" />
	</clusternodes>
</cluster>

The clusternode tag defines each cluster node. There are many attributes available, but we will look at just the two required ones.

The first is the name="" attribute. This should match the name given by uname -n ($HOSTNAME) when run on each node. The IP address that the name resolves to also sets the interface and subnet that the totem ring will run on. That is, the main cluster communications, which we are calling the Back-Channel Network. This is why it is so important to setup our /etc/hosts file correctly. Please see the clusternode's name attribute document for details on how name to interface mapping is resolved.

The second attribute is nodeid="". This must be a unique integer amongst the <clusternode ...> tags. It is used by the cluster to identify the node.

Defining Fence Devices

Fencing devices are designed to forcible eject a node from a cluster. This is generally done by forcing it to power off or reboot. Some SAN switches can logically disconnect a node from the shared storage device, which has the same effect of guaranteeing that the defective node can not alter the shared storage. A common, third type of fence device is one that cuts the mains power to the server.

In this tutorial, our nodes support IPMI, which we will use as the primary fence device. We also have an APC brand switched PDU which will act as a backup in case a fault in the node disables the IPMI BMC.

Note: Not all brands of switched PDUs are supported as fence devices. Before you purchase a fence device, confirm that it is supported.

All fence devices are contained within the parent fencedevices tag. This parent tag has no attributes. Within this parent tag are one or more fencedevice child tags.

<?xml version="1.0"?>
<cluster name="an-clusterA" config_version="4">
	<cman expected_votes="1" two_node="1"/>
	<clusternodes>
		<clusternode name="an-node01.alteeve.com" nodeid="1" />
		<clusternode name="an-node02.alteeve.com" nodeid="2" />
	</clusternodes>
	<fencedevices>
		<fencedevice name="ipmi_an01" agent="fence_ipmilan" ipaddr="an-node01.ipmi" login="root" passwd="secret" />
		<fencedevice name="ipmi_an02" agent="fence_ipmilan" ipaddr="an-node02.ipmi" login="root" passwd="secret" />
		<fencedevice name="pdu2" agent="fence_apc" ipaddr="pdu2.alteeve.com" login="apc" passwd="secret" />
	</fencedevices>
</cluster>

Every fence device used in your cluster will have it's own fencedevice tag. If you are using IPMI, this means you will have a fencedevice entry for each node, as each physical IPMI BMC is a unique fence device. On the other hand, fence devices that support multiple nodes, like switched PDUs, will have just one entry. In our case, we're using both types, so we have three fences devices; The two IPMI BMCs plus the switched PDU.

All fencedevice tags share two basic attributes; name="" and agent="".

• The name attribute must be unique among all the fence devices in your cluster. As we will see in the next step, this name will be used within the <clusternode...> tag.
• The agent tag tells the cluster which fence agent to use when the fenced daemon needs to communicate with the physical fence device. A fence agent is simple a shell script that acts as a glue layer between the fenced daemon and the fence hardware. This agent takes the arguments from the daemon, like what port to act on and what action to take, and executes the node. The agent is responsible for ensuring that the execution succeeded and returning an appropriate success or failure exit code, depending. For those curious, the full details are described in the FenceAgentAPI. If you have two or more of the same fence device, like IPMI, then you will use the same fence agent value a corresponding number of times.

Beyond these two attributes, each fence agent will have it's own subset of attributes. The scope of which is outside this tutorial, though we will see examples for IPMI and a switched PDU. Most, if not all, fence agents have a corresponding man page that will show you what attributes it accepts and how they are used. The two fence agents we will see here have their attributes defines in the following man pages.

• man fence_ipmilan - IPMI fence agent.
• man fence_apc - APC-brand switched PDU.

The example above is what this tutorial will use.

Example <fencedevice...> Tag For IPMI

Here we will show what IPMI <fencedevice...> tags look like. We won't be using it ourselves, but it is quite popular as a fence device so I wanted to show an example of it's use.

                ...
                        <fence>
                                <method name="ipmi">
                                        <device name="ipmi_an01" action="reboot"/>
                                </method>
                        </fence>
                ...
	<fencedevices>
		<fencedevice name="ipmi_an01" agent="fence_ipmilan" ipaddr="an-node01.ipmi" login="root" passwd="secret" />
		<fencedevice name="ipmi_an02" agent="fence_ipmilan" ipaddr="an-node02.ipmi" login="root" passwd="secret" />
	</fencedevices>

• ipaddr; This is the resolvable name or IP address of the device. If you use a resolvable name, it is strongly advised that you put the name in /etc/hosts as DNS is another layer of abstraction which could fail.
• login; This is the login name to use when the fenced daemon connects to the device.
• passwd; This is the login password to use when the fenced daemon connects to the device.
• name; This is the name of this particular fence device within the cluster which, as we will see shortly, is matched in the <clusternode...> element where appropriate.

Note: We will see shortly that, unlike switched PDUs or other network fence devices, IPMI does not have ports. This is because each IPMI BMC supports just it's host system. More on that later.

Example <fencedevice...> Tag For APC Switched PDUs

Here we will show how to configure APC switched PDU <fencedevice...> tags. We won't be using it in this tutorial, but in the real world, it is highly recommended as a backup fence device for IPMI and similar primary fence devices.

		...
			<fence>
				<method name="pdu">
					<device name="pdu2" port="1" action="reboot"/>
				</method>
			</fence>
		...
	<fencedevices>
		<fencedevice name="pdu2" agent="fence_apc" ipaddr="pdu2.alteeve.com" login="apc" passwd="secret" />
	</fencedevices>

• ipaddr; This is the resolvable name or IP address of the device. If you use a resolvable name, it is strongly advised that you put the name in /etc/hosts as DNS is another layer of abstraction which could fail.
• login; This is the login name to use when the fenced daemon connects to the device.
• passwd; This is the login password to use when the fenced daemon connects to the device.
• name; This is the name of this particular fence device within the cluster which, as we will see shortly, is matched in the <clusternode...> element where appropriate.

Using the Fence Devices

Now we have nodes and fence devices defined, we will go back and tie them together. This is done by:

• Defining a fence tag containing all fence methods and devices.
  • Defining one or more method tag(s) containing the device call(s) needed for each fence attempt.
    • Defining one or more device tag(s) containing attributes describing how to call the fence device to kill this node.

Here is how we implement IPMI as the primary fence device with the switched PDU as the backup method.

<?xml version="1.0"?>
<cluster name="an-clusterA" config_version="5">
        <cman expected_votes="1" two_node="1"/>
        <clusternodes>
                <clusternode name="an-node01.alteeve.com" nodeid="1">
                        <fence>
                                <method name="ipmi">
                                        <device name="ipmi_an01" action="reboot"/>
                                </method>
                                <method name="pdu">
                                        <device name="pdu2" port="1" action="reboot"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="an-node02.alteeve.com" nodeid="2">
                        <fence>
                                <method name="ipmi">
                                        <device name="ipmi_an02" action="reboot"/>
                                </method>
                                <method name="pdu">
                                        <device name="pdu2" port="2" action="reboot"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <fencedevices>
                <fencedevice name="ipmi_an01" agent="fence_ipmilan" ipaddr="an-node01.ipmi" login="root" passwd="secret" />
                <fencedevice name="ipmi_an02" agent="fence_ipmilan" ipaddr="an-node02.ipmi" login="root" passwd="secret" />
                <fencedevice name="pdu2" agent="fence_apc" ipaddr="pdu2.alteeve.com" login="apc" passwd="secret" />
        </fencedevices>
</cluster>

First, notice that the fence tag has no attributes. It's merely a container for the method(s).

The next level is the method named ipmi. This name is merely a description and can be whatever you feel is most appropriate. It's purpose is simply to help you distinguish this method from other methods. The reason for method tags is that some fence device calls will have two or more steps. A classic example would be a node with a redundant power supply on a switch PDU acting as the fence device. In such a case, you will need to define multiple device tags, one for each power cable feeding the node. In such a case, the cluster will not consider the fence a success unless and until all contained device calls execute successfully.

The same pair of method and device tags are supplied a second time. The first pair defined the IPMI interfaces, and the second pair defined the switched PDU. Note that the PDU definition needs a port="" attribute where the IPMI fence device does not. When a fence call is needed, the fence devices will be called in the order they are found here. If both devices fail, the cluster will go back to the start and try again, looping indefinitely until one device succeeds.

Note: It's important to understand why we use IPMI as the primary fence device. It is suggested, but not required, that the fence device confirm that the node is off. IPMI can do this, the switched PDU does not. Thus, IPMI won't return a success unless the node is truly off. The PDU though will return a success once the power is cut to the requested port. However, a misconfigured node with redundant PDU may in fact still be running, leading to disastrous consequences.

The actual fence device configuration is the final piece of the puzzle. It is here that you specify per-node configuration options and link these attributes to a given fencedevice. Here, we see the link to the fencedevice via the name, fence_na01 in this example.

Let's step through an example fence call to help show how the per-cluster and fence device attributes are combined during a fence call.

• The cluster manager decides that a node needs to be fenced. Let's say that the victim is an-node02.
• The first method in the fence section under an-node02 is consulted. Within it there are two method entries, named ipmi and pdu. The IPMI method's device has one attribute while the PDU's device has two attributes;
  • port; only found in the PDU method, this tells the cluster that an-node02 is connected to switched PDU's port number 2.
  • action; Found on both devices, this tells the cluster that the fence action to take is reboot. How this action is actually interpreted depends on the fence device in use, though the name certainly implies that the node will be forced off and then restarted.
• The cluster searches in fencedevices for a fencedevice matching the name ipmi_an02. This fence device has four attributes;
  • agent; This tells the cluster to call the fence_ipmilan fence agent script, as we discussed earlier.
  • ipaddr; This tells the fence agent where on the network to find this particular IPMI BMC. This is how multiple fence devices of the same type can be used in the cluster.
  • login; This is the login user name to use when authenticating against the fence device.
  • passwd; This is the password to supply along with the login name when authenticating against the fence device.
• Should the IPMI fence call fail for some reason, the cluster will move on to the second pdu method, repeating the steps above but using the PDU values.

When the cluster calls the fence agent, it does so by initially calling the fence agent script with no arguments.

/usr/sbin/fence_ipmilan

Then it will pass to that agent the following arguments:

ipaddr=an-node02.ipmi
login=root
passwd=secret
action=reboot

As you can see then, the first three arguments are from the fencedevice attributes and the last one is from the device attributes under an-node02's clusternode's fence tag.

If this method fails, then the PDU will be called in a very similar way, but with an extra argument from the device attributes.

/usr/sbin/fence_apc

Then it will pass to that agent the following arguments:

ipaddr=pdu2.alteeve.com
login=root
passwd=secret
port=2
action=reboot

Should this fail, the cluster will go back and try the IPMI interface again. It will loop through the fence device methods forever until one of the methods succeeds.

Give Nodes More Time To Start

Clusters with more than three nodes will have to gain quorum before they can fence other nodes. As we saw earlier though, this is not really the case when using the two_node="1" attribute in the cman tag. What this means in practice is that if you start the cluster on one node and then wait too long to start the cluster on the second node, the first will fence the second.

The logic behind this is; When the cluster starts, it will try to talk to it's fellow node and then fail. With the special two_node="1" attribute set, the cluster knows that it is allowed to start clustered services, but it has no way to say for sure what state the other node is in. It could well be online and hosting services for all it knows. So it has to proceed on the assumption that the other node is alive and using shared resources. Given that, and given that it can not talk to the other node, it's only safe option is to fence the other node. Only then can it be confident that it is safe to start providing clustered services.

<?xml version="1.0"?>
<cluster name="an-clusterA" config_version="7">
        <cman expected_votes="1" two_node="1"/>
        <clusternodes>
                <clusternode name="an-node01.alteeve.com" nodeid="1">
                        <fence>
                                <method name="ipmi">
                                        <device name="ipmi_an01" action="reboot"/>
                                </method>
                                <method name="pdu">
                                        <device name="pdu2" port="1" action="reboot"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="an-node02.alteeve.com" nodeid="2">
                        <fence>
                                <method name="ipmi">
                                        <device name="ipmi_an02" action="reboot"/>
                                </method>
                                <method name="pdu">
                                        <device name="pdu2" port="2" action="reboot"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <fencedevices>
                <fencedevice name="ipmi_an01" agent="fence_ipmilan" ipaddr="an-node01.ipmi" login="root" passwd="secret" />
                <fencedevice name="ipmi_an02" agent="fence_ipmilan" ipaddr="an-node02.ipmi" login="root" passwd="secret" />
                <fencedevice name="pdu2" agent="fence_apc" ipaddr="pdu2.alteeve.com" login="apc" passwd="secret" />
        </fencedevices>
        <fence_daemon post_join_delay="30"/>
</cluster>

The new tag is fence_daemon, seen near the bottom if the file above. The change is made using the post_join_delay="60" attribute. By default, the cluster will declare the other node dead after just 6 seconds. The reason is that the larger this value, the slower the start-up of the cluster services will be. During testing and development though, I find this value to be far too short and frequently led to unnecessary fencing. Once your cluster is setup and working, it's not a bad idea to reduce this value to the lowest value that you are comfortable with.

Configuring Totem

This is almost a misnomer, as we're more or less not configuring the totem protocol in this cluster.

<?xml version="1.0"?>
<cluster name="an-clusterA" config_version="8">
        <cman expected_votes="1" two_node="1"/>
        <clusternodes>
                <clusternode name="an-node01.alteeve.com" nodeid="1">
                        <fence>
                                <method name="ipmi">
                                        <device name="ipmi_an01" action="reboot"/>
                                </method>
                                <method name="pdu">
                                        <device name="pdu2" port="1" action="reboot"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="an-node02.alteeve.com" nodeid="2">
                        <fence>
                                <method name="ipmi">
                                        <device name="ipmi_an02" action="reboot"/>
                                </method>
                                <method name="pdu">
                                        <device name="pdu2" port="2" action="reboot"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <fencedevices>
                <fencedevice name="ipmi_an01" agent="fence_ipmilan" ipaddr="an-node01.ipmi" login="root" passwd="secret" />
                <fencedevice name="ipmi_an02" agent="fence_ipmilan" ipaddr="an-node02.ipmi" login="root" passwd="secret" />
                <fencedevice name="pdu2" agent="fence_apc" ipaddr="pdu2.alteeve.com" login="apc" passwd="secret" />
        </fencedevices>
        <fence_daemon post_join_delay="30"/>
        <totem rrp_mode="none" secauth="off"/>
</cluster>

Note: At this time, redundant ring protocol is not supported (RHEL6.1 and lower) and will be in technology preview mode (RHEL6.2 and above). For this reason, we will not be using it. However, we are using bonding, so we still have removed a single point of failure.

RRP is an optional second ring that can be used for cluster communication in the case of a break down in the first ring. However, if you wish to explore it further, please take a look at the clusternode element tag called <altname...>. When altname is used though, then the rrp_mode attribute will need to be changed to either active or passive (the details of which are outside the scope of this tutorial).

The second option we're looking at here is the secauth="off" attribute. This controls whether the cluster communications are encrypted or not. We can safely disable this because we're working on a known-private network, which yields two benefits; It's simpler to setup and it's a lot faster. If you must encrypt the cluster communications, then you can do so here. The details of which are also outside the scope of this tutorial though.

Validating and Pushing the /etc/cluster/cluster.conf File

One of the most noticeable changes in RHCS cluster stable 3 is that we no longer have to make a long, cryptic xmllint call to validate our cluster configuration. Now we can simply call ccs_config_validate.

ccs_config_validate

Configuration validates

If there was a problem, you need to go back and fix it. DO NOT proceed until your configuration validates. Once it does, we're ready to move on!

With it validated, we need to push it to the other node. As the cluster is not running yet, we will push it out using rsync.

rsync -av /etc/cluster/cluster.conf root@an-node02:/etc/cluster/

sending incremental file list
cluster.conf

sent 1228 bytes  received 31 bytes  2518.00 bytes/sec
total size is 1148  speedup is 0.91

Setting Up ricci

Another change from RHCS stable 2 is how configuration changes are propagated. Before, after a change, we'd push out the updated cluster configuration by calling ccs_tool update /etc/cluster/cluster.conf. Now this is done with cman_tool version -r. More fundamentally though, the cluster needs to authenticate against each node and does this using the local ricci system user. The user has no password initially, so we need to set one.

On both nodes:

passwd ricci

Changing password for user ricci.
New password: 
Retype new password: 
passwd: all authentication tokens updated successfully.

You will need to enter this password once from each node against the other node. We will see this later.

Now make sure that the ricci daemon is set to start on boot and is running now.

chkconfig ricci on
/etc/init.d/ricci start

Starting ricci:                                            [  OK  ]

Note: If you don't see [ OK ], don't worry, it is probably because it was already running.

Starting the Cluster for the First Time

It's a good idea to open a second terminal on either node and tail the /var/log/messages syslog file. All cluster messages will be recorded here and it will help to debug problems if you can watch the logs. To do this, in the new terminal windows run;

clear; tail -f -n 0 /var/log/messages

This will clear the screen and start watching for new lines to be written to syslog. When you are done watching syslog, press the <ctrl> + c key combination.

How you lay out your terminal windows is, obviously, up to your own preferences. Below is a configuration I have found very useful.

Terminal window layout for watching 2 nodes. Left windows are used for entering commands and the left windows are used for tailing syslog.

With the terminals setup, lets start the cluster!

Warning: If you don't start cman on both nodes within 30 seconds, the slower node will be fenced.

On both nodes, run:

/etc/init.d/cman start

Starting cluster: 
   Checking Network Manager...                             [  OK  ]
   Global setup...                                         [  OK  ]
   Loading kernel modules...                               [  OK  ]
   Mounting configfs...                                    [  OK  ]
   Starting cman...                                        [  OK  ]
   Waiting for quorum...                                   [  OK  ]
   Starting fenced...                                      [  OK  ]
   Starting dlm_controld...                                [  OK  ]
   Starting gfs_controld...                                [  OK  ]
   Unfencing self...                                       [  OK  ]
   Joining fence domain...                                 [  OK  ]

Here is what you should see in syslog:

Sep 14 13:33:58 an-node01 kernel: DLM (built Jun 27 2011 19:51:46) installed
Sep 14 13:33:58 an-node01 corosync[18897]:   [MAIN  ] Corosync Cluster Engine ('1.2.3'): started and ready to provide service.
Sep 14 13:33:58 an-node01 corosync[18897]:   [MAIN  ] Corosync built-in features: nss rdma
Sep 14 13:33:58 an-node01 corosync[18897]:   [MAIN  ] Successfully read config from /etc/cluster/cluster.conf
Sep 14 13:33:58 an-node01 corosync[18897]:   [MAIN  ] Successfully parsed cman config
Sep 14 13:33:58 an-node01 corosync[18897]:   [TOTEM ] Initializing transport (UDP/IP).
Sep 14 13:33:58 an-node01 corosync[18897]:   [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Sep 14 13:33:58 an-node01 corosync[18897]:   [TOTEM ] The network interface [10.20.0.1] is now up.
Sep 14 13:33:58 an-node01 corosync[18897]:   [QUORUM] Using quorum provider quorum_cman
Sep 14 13:33:58 an-node01 corosync[18897]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Sep 14 13:33:58 an-node01 corosync[18897]:   [CMAN  ] CMAN 3.0.12 (built Jul  4 2011 22:35:06) started
Sep 14 13:33:58 an-node01 corosync[18897]:   [SERV  ] Service engine loaded: corosync CMAN membership service 2.90
Sep 14 13:33:58 an-node01 corosync[18897]:   [SERV  ] Service engine loaded: openais checkpoint service B.01.01
Sep 14 13:33:58 an-node01 corosync[18897]:   [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Sep 14 13:33:58 an-node01 corosync[18897]:   [SERV  ] Service engine loaded: corosync configuration service
Sep 14 13:33:58 an-node01 corosync[18897]:   [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Sep 14 13:33:58 an-node01 corosync[18897]:   [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Sep 14 13:33:58 an-node01 corosync[18897]:   [SERV  ] Service engine loaded: corosync profile loading service
Sep 14 13:33:58 an-node01 corosync[18897]:   [QUORUM] Using quorum provider quorum_cman
Sep 14 13:33:58 an-node01 corosync[18897]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Sep 14 13:33:58 an-node01 corosync[18897]:   [MAIN  ] Compatibility mode set to whitetank.  Using V1 and V2 of the synchronization engine.
Sep 14 13:33:58 an-node01 corosync[18897]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 14 13:33:58 an-node01 corosync[18897]:   [CMAN  ] quorum regained, resuming activity
Sep 14 13:33:58 an-node01 corosync[18897]:   [QUORUM] This node is within the primary component and will provide service.
Sep 14 13:33:58 an-node01 corosync[18897]:   [QUORUM] Members[1]: 1
Sep 14 13:33:58 an-node01 corosync[18897]:   [QUORUM] Members[1]: 1
Sep 14 13:33:58 an-node01 corosync[18897]:   [CPG   ] downlist received left_list: 0
Sep 14 13:33:58 an-node01 corosync[18897]:   [CPG   ] chosen downlist from node r(0) ip(10.20.0.1) 
Sep 14 13:33:58 an-node01 corosync[18897]:   [MAIN  ] Completed service synchronization, ready to provide service.
Sep 14 13:34:02 an-node01 corosync[18897]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 14 13:34:02 an-node01 corosync[18897]:   [QUORUM] Members[2]: 1 2
Sep 14 13:34:02 an-node01 corosync[18897]:   [QUORUM] Members[2]: 1 2
Sep 14 13:34:02 an-node01 corosync[18897]:   [CPG   ] downlist received left_list: 0
Sep 14 13:34:02 an-node01 corosync[18897]:   [CPG   ] downlist received left_list: 0
Sep 14 13:34:02 an-node01 corosync[18897]:   [CPG   ] chosen downlist from node r(0) ip(10.20.0.1) 
Sep 14 13:34:02 an-node01 corosync[18897]:   [MAIN  ] Completed service synchronization, ready to provide service.
Sep 14 13:34:02 an-node01 fenced[18954]: fenced 3.0.12 started
Sep 14 13:34:02 an-node01 dlm_controld[18978]: dlm_controld 3.0.12 started
Sep 14 13:34:02 an-node01 gfs_controld[19000]: gfs_controld 3.0.12 started

Now to confirm that the cluster is operating properly, run cman_tool status;

cman_tool status

Version: 6.2.0
Config Version: 8
Cluster Name: an-clusterA
Cluster Id: 29382
Cluster Member: Yes
Cluster Generation: 8
Membership state: Cluster-Member
Nodes: 2
Expected votes: 1
Total votes: 2
Node votes: 1
Quorum: 1  
Active subsystems: 7
Flags: 2node 
Ports Bound: 0  
Node name: an-node01.alteeve.com
Node ID: 1
Multicast addresses: 239.192.114.57 
Node addresses: 10.20.0.1

We can see that the both nodes are talking because of the Nodes: 2 entry.

If you ever want to see the nitty-gritty configuration, you can run corosync-objctl.

corosync-objctl

cluster.name=an-clusterA
cluster.config_version=8
cluster.cman.expected_votes=1
cluster.cman.two_node=1
cluster.cman.nodename=an-node01.alteeve.com
cluster.cman.cluster_id=29382
cluster.clusternodes.clusternode.name=an-node01.alteeve.com
cluster.clusternodes.clusternode.nodeid=1
cluster.clusternodes.clusternode.fence.method.name=ipmi
cluster.clusternodes.clusternode.fence.method.device.name=ipmi_an01
cluster.clusternodes.clusternode.fence.method.device.action=reboot
cluster.clusternodes.clusternode.fence.method.name=pdu
cluster.clusternodes.clusternode.fence.method.device.name=pdu2
cluster.clusternodes.clusternode.fence.method.device.port=1
cluster.clusternodes.clusternode.fence.method.device.action=reboot
cluster.clusternodes.clusternode.name=an-node02.alteeve.com
cluster.clusternodes.clusternode.nodeid=2
cluster.clusternodes.clusternode.fence.method.name=ipmi
cluster.clusternodes.clusternode.fence.method.device.name=ipmi_an02
cluster.clusternodes.clusternode.fence.method.device.action=reboot
cluster.clusternodes.clusternode.fence.method.name=pdu
cluster.clusternodes.clusternode.fence.method.device.name=pdu2
cluster.clusternodes.clusternode.fence.method.device.port=2
cluster.clusternodes.clusternode.fence.method.device.action=reboot
cluster.fencedevices.fencedevice.name=ipmi_an01
cluster.fencedevices.fencedevice.agent=fence_ipmilan
cluster.fencedevices.fencedevice.ipaddr=an-node01.ipmi
cluster.fencedevices.fencedevice.login=root
cluster.fencedevices.fencedevice.passwd=secret
cluster.fencedevices.fencedevice.name=ipmi_an02
cluster.fencedevices.fencedevice.agent=fence_ipmilan
cluster.fencedevices.fencedevice.ipaddr=an-node02.ipmi
cluster.fencedevices.fencedevice.login=root
cluster.fencedevices.fencedevice.passwd=secret
cluster.fencedevices.fencedevice.name=pdu2
cluster.fencedevices.fencedevice.agent=fence_apc
cluster.fencedevices.fencedevice.ipaddr=pdu2.alteeve.com
cluster.fencedevices.fencedevice.login=apc
cluster.fencedevices.fencedevice.passwd=secret
cluster.fence_daemon.post_join_delay=30
cluster.totem.rrp_mode=none
cluster.totem.secauth=off
totem.rrp_mode=none
totem.secauth=off
totem.version=2
totem.nodeid=1
totem.vsftype=none
totem.token=10000
totem.join=60
totem.fail_recv_const=2500
totem.consensus=2000
totem.key=an-clusterA
totem.interface.ringnumber=0
totem.interface.bindnetaddr=10.20.0.1
totem.interface.mcastaddr=239.192.114.57
totem.interface.mcastport=5405
libccs.next_handle=7
libccs.connection.ccs_handle=3
libccs.connection.config_version=8
libccs.connection.fullxpath=0
libccs.connection.ccs_handle=4
libccs.connection.config_version=8
libccs.connection.fullxpath=0
libccs.connection.ccs_handle=5
libccs.connection.config_version=8
libccs.connection.fullxpath=0
logging.timestamp=on
logging.to_logfile=yes
logging.logfile=/var/log/cluster/corosync.log
logging.logfile_priority=info
logging.to_syslog=yes
logging.syslog_facility=local4
logging.syslog_priority=info
aisexec.user=ais
aisexec.group=ais
service.name=corosync_quorum
service.ver=0
service.name=corosync_cman
service.ver=0
quorum.provider=quorum_cman
service.name=openais_ckpt
service.ver=0
runtime.services.quorum.service_id=12
runtime.services.cman.service_id=9
runtime.services.ckpt.service_id=3
runtime.services.ckpt.0.tx=0
runtime.services.ckpt.0.rx=0
runtime.services.ckpt.1.tx=0
runtime.services.ckpt.1.rx=0
runtime.services.ckpt.2.tx=0
runtime.services.ckpt.2.rx=0
runtime.services.ckpt.3.tx=0
runtime.services.ckpt.3.rx=0
runtime.services.ckpt.4.tx=0
runtime.services.ckpt.4.rx=0
runtime.services.ckpt.5.tx=0
runtime.services.ckpt.5.rx=0
runtime.services.ckpt.6.tx=0
runtime.services.ckpt.6.rx=0
runtime.services.ckpt.7.tx=0
runtime.services.ckpt.7.rx=0
runtime.services.ckpt.8.tx=0
runtime.services.ckpt.8.rx=0
runtime.services.ckpt.9.tx=0
runtime.services.ckpt.9.rx=0
runtime.services.ckpt.10.tx=0
runtime.services.ckpt.10.rx=0
runtime.services.ckpt.11.tx=2
runtime.services.ckpt.11.rx=3
runtime.services.ckpt.12.tx=0
runtime.services.ckpt.12.rx=0
runtime.services.ckpt.13.tx=0
runtime.services.ckpt.13.rx=0
runtime.services.evs.service_id=0
runtime.services.evs.0.tx=0
runtime.services.evs.0.rx=0
runtime.services.cfg.service_id=7
runtime.services.cfg.0.tx=0
runtime.services.cfg.0.rx=0
runtime.services.cfg.1.tx=0
runtime.services.cfg.1.rx=0
runtime.services.cfg.2.tx=0
runtime.services.cfg.2.rx=0
runtime.services.cfg.3.tx=0
runtime.services.cfg.3.rx=0
runtime.services.cpg.service_id=8
runtime.services.cpg.0.tx=4
runtime.services.cpg.0.rx=8
runtime.services.cpg.1.tx=0
runtime.services.cpg.1.rx=0
runtime.services.cpg.2.tx=0
runtime.services.cpg.2.rx=0
runtime.services.cpg.3.tx=16
runtime.services.cpg.3.rx=23
runtime.services.cpg.4.tx=0
runtime.services.cpg.4.rx=0
runtime.services.cpg.5.tx=2
runtime.services.cpg.5.rx=3
runtime.services.confdb.service_id=11
runtime.services.pload.service_id=13
runtime.services.pload.0.tx=0
runtime.services.pload.0.rx=0
runtime.services.pload.1.tx=0
runtime.services.pload.1.rx=0
runtime.services.quorum.service_id=12
runtime.connections.active=6
runtime.connections.closed=110
runtime.connections.fenced:18954:16.service_id=8
runtime.connections.fenced:18954:16.client_pid=18954
runtime.connections.fenced:18954:16.responses=5
runtime.connections.fenced:18954:16.dispatched=9
runtime.connections.fenced:18954:16.requests=5
runtime.connections.fenced:18954:16.sem_retry_count=0
runtime.connections.fenced:18954:16.send_retry_count=0
runtime.connections.fenced:18954:16.recv_retry_count=0
runtime.connections.fenced:18954:16.flow_control=0
runtime.connections.fenced:18954:16.flow_control_count=0
runtime.connections.fenced:18954:16.queue_size=0
runtime.connections.dlm_controld:18978:24.service_id=8
runtime.connections.dlm_controld:18978:24.client_pid=18978
runtime.connections.dlm_controld:18978:24.responses=5
runtime.connections.dlm_controld:18978:24.dispatched=8
runtime.connections.dlm_controld:18978:24.requests=5
runtime.connections.dlm_controld:18978:24.sem_retry_count=0
runtime.connections.dlm_controld:18978:24.send_retry_count=0
runtime.connections.dlm_controld:18978:24.recv_retry_count=0
runtime.connections.dlm_controld:18978:24.flow_control=0
runtime.connections.dlm_controld:18978:24.flow_control_count=0
runtime.connections.dlm_controld:18978:24.queue_size=0
runtime.connections.dlm_controld:18978:19.service_id=3
runtime.connections.dlm_controld:18978:19.client_pid=18978
runtime.connections.dlm_controld:18978:19.responses=0
runtime.connections.dlm_controld:18978:19.dispatched=0
runtime.connections.dlm_controld:18978:19.requests=0
runtime.connections.dlm_controld:18978:19.sem_retry_count=0
runtime.connections.dlm_controld:18978:19.send_retry_count=0
runtime.connections.dlm_controld:18978:19.recv_retry_count=0
runtime.connections.dlm_controld:18978:19.flow_control=0
runtime.connections.dlm_controld:18978:19.flow_control_count=0
runtime.connections.dlm_controld:18978:19.queue_size=0
runtime.connections.gfs_controld:19000:22.service_id=8
runtime.connections.gfs_controld:19000:22.client_pid=19000
runtime.connections.gfs_controld:19000:22.responses=5
runtime.connections.gfs_controld:19000:22.dispatched=8
runtime.connections.gfs_controld:19000:22.requests=5
runtime.connections.gfs_controld:19000:22.sem_retry_count=0
runtime.connections.gfs_controld:19000:22.send_retry_count=0
runtime.connections.gfs_controld:19000:22.recv_retry_count=0
runtime.connections.gfs_controld:19000:22.flow_control=0
runtime.connections.gfs_controld:19000:22.flow_control_count=0
runtime.connections.gfs_controld:19000:22.queue_size=0
runtime.connections.fenced:18954:25.service_id=8
runtime.connections.fenced:18954:25.client_pid=18954
runtime.connections.fenced:18954:25.responses=5
runtime.connections.fenced:18954:25.dispatched=8
runtime.connections.fenced:18954:25.requests=5
runtime.connections.fenced:18954:25.sem_retry_count=0
runtime.connections.fenced:18954:25.send_retry_count=0
runtime.connections.fenced:18954:25.recv_retry_count=0
runtime.connections.fenced:18954:25.flow_control=0
runtime.connections.fenced:18954:25.flow_control_count=0
runtime.connections.fenced:18954:25.queue_size=0
runtime.connections.corosync-objctl:19188:23.service_id=11
runtime.connections.corosync-objctl:19188:23.client_pid=19188
runtime.connections.corosync-objctl:19188:23.responses=435
runtime.connections.corosync-objctl:19188:23.dispatched=0
runtime.connections.corosync-objctl:19188:23.requests=438
runtime.connections.corosync-objctl:19188:23.sem_retry_count=0
runtime.connections.corosync-objctl:19188:23.send_retry_count=0
runtime.connections.corosync-objctl:19188:23.recv_retry_count=0
runtime.connections.corosync-objctl:19188:23.flow_control=0
runtime.connections.corosync-objctl:19188:23.flow_control_count=0
runtime.connections.corosync-objctl:19188:23.queue_size=0
runtime.totem.pg.mrp.srp.orf_token_tx=2
runtime.totem.pg.mrp.srp.orf_token_rx=744
runtime.totem.pg.mrp.srp.memb_merge_detect_tx=365
runtime.totem.pg.mrp.srp.memb_merge_detect_rx=365
runtime.totem.pg.mrp.srp.memb_join_tx=3
runtime.totem.pg.mrp.srp.memb_join_rx=5
runtime.totem.pg.mrp.srp.mcast_tx=46
runtime.totem.pg.mrp.srp.mcast_retx=0
runtime.totem.pg.mrp.srp.mcast_rx=57
runtime.totem.pg.mrp.srp.memb_commit_token_tx=4
runtime.totem.pg.mrp.srp.memb_commit_token_rx=4
runtime.totem.pg.mrp.srp.token_hold_cancel_tx=4
runtime.totem.pg.mrp.srp.token_hold_cancel_rx=7
runtime.totem.pg.mrp.srp.operational_entered=2
runtime.totem.pg.mrp.srp.operational_token_lost=0
runtime.totem.pg.mrp.srp.gather_entered=2
runtime.totem.pg.mrp.srp.gather_token_lost=0
runtime.totem.pg.mrp.srp.commit_entered=2
runtime.totem.pg.mrp.srp.commit_token_lost=0
runtime.totem.pg.mrp.srp.recovery_entered=2
runtime.totem.pg.mrp.srp.recovery_token_lost=0
runtime.totem.pg.mrp.srp.consensus_timeouts=0
runtime.totem.pg.mrp.srp.mtt_rx_token=1903
runtime.totem.pg.mrp.srp.avg_token_workload=0
runtime.totem.pg.mrp.srp.avg_backlog_calc=0
runtime.totem.pg.mrp.srp.rx_msg_dropped=0
runtime.blackbox.dump_flight_data=no
runtime.blackbox.dump_state=no
cman_private.COROSYNC_DEFAULT_CONFIG_IFACE=xmlconfig:cmanpreconfig

Testing Fencing

We need to thoroughly test our fence configuration and devices before we proceed. Should the cluster call a fence, and if the fence call fails, the cluster will hang until the fence finally succeeds. There is no way to abort a fence, so this could effectively hang the cluster. If we have problems, we need to find them now.

We need to run two tests from each node against the other node for a total of four tests.

• The first test will use fence_ipmilan. To do this, we will hang the victim node by running echo c > /proc/sysrq-trigger on it. This will immediately and completely hang the kernel. The other node should detect the failure and reboot the victim. You can confirm that IPMI was used by watching the fence PDU and not seeing it power-cycle the port.
• Secondly, we will pull the power on the victim node. This is done to ensure that the IPMI BMC is also dead and will simulate a failure in the power supply. You should see the other node try to fence the victim, fail initially, then try again using the second, switched PDU. If you want the PDU, you should see the power indicator LED go off and then come back on.

Note: To "pull the power", we can actually just log into the PDU and turn off the victim's power. In this case, we'll see the power restored when the PDU is used to fence the node. We can actually use the fence_apc fence agent to pull the power, as we'll see.

Test	Victim	Pass?
echo c > /proc/sysrq-trigger	an-node01
fence_apc -a pdu2.alteeve.com -l apc -p secret -n 1 -o off	an-node01
echo c > /proc/sysrq-trigger	an-node02
fence_apc -a pdu2.alteeve.com -l apc -p secret -n 2 -o off	an-node02

After the lost node is recovered, remember to restart cman before starting the next test.

Hanging an-node01

Be sure to be tailing the /var/log/messages on an-node02. Go to an-node01's first terminal and run the following command.

Warning: This command will not return and you will lose all ability to talk to this node until it is rebooted.

On an-node01 run:

echo c > /proc/sysrq-trigger

On an-node02's syslog terminal, you should see the following entries in the log.

Sep 15 16:08:17 an-node02 corosync[12347]:   [TOTEM ] A processor failed, forming new configuration.
Sep 15 16:08:19 an-node02 corosync[12347]:   [QUORUM] Members[1]: 2
Sep 15 16:08:19 an-node02 corosync[12347]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 15 16:08:19 an-node02 corosync[12347]:   [CPG   ] downlist received left_list: 1
Sep 15 16:08:19 an-node02 corosync[12347]:   [CPG   ] chosen downlist from node r(0) ip(10.20.0.2) 
Sep 15 16:08:19 an-node02 corosync[12347]:   [MAIN  ] Completed service synchronization, ready to provide service.
Sep 15 16:08:19 an-node02 kernel: dlm: closing connection to node 1
Sep 15 16:08:19 an-node02 fenced[12403]: fencing node an-node01.alteeve.com
Sep 15 16:08:33 an-node02 fenced[12403]: fence an-node01.alteeve.com success

Perfect!

If you are watching an-node01's display, you should now see it starting back up. Once it finished booting, log into it and restart cman before moving on to the next test.

Cutting the Power to an-node01

As was discussed earlier, IPMI has a fatal flaw as a fence device. IPMI's BMC draws it's power from the same power supply as the node itself. Thus, when the power supply itself fails (or the mains connection is pulled/tripped over), fencing via IPMI will fail. This makes the power supply a single point of failure, which is what the PDU protects us against.

So to simulate a failed power supply, we're going to use an-node02's fence_apc fence agent to turn off the power to an-node01.

Alternatively, you could also just unplug the power and the fence would still succeed. The fence call only needs to confirm that the node is off to succeed. Whether the node restarts after or not is not important so far as the cluster is concerned.

From an-node02, pull the power on an-node01 with the following call;

fence_apc -a pdu2.alteeve.com -l apc -p secret -n 1 -o off

Success: Powered OFF

Back on an-node02's syslog, we should see the following entries;

Sep 15 16:18:06 an-node02 corosync[12347]:   [TOTEM ] A processor failed, forming new configuration.
Sep 15 16:18:08 an-node02 corosync[12347]:   [QUORUM] Members[1]: 2
Sep 15 16:18:08 an-node02 corosync[12347]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 15 16:18:08 an-node02 kernel: dlm: closing connection to node 1
Sep 15 16:18:08 an-node02 corosync[12347]:   [CPG   ] downlist received left_list: 1
Sep 15 16:18:08 an-node02 corosync[12347]:   [CPG   ] chosen downlist from node r(0) ip(10.20.0.2) 
Sep 15 16:18:08 an-node02 corosync[12347]:   [MAIN  ] Completed service synchronization, ready to provide service.
Sep 15 16:18:08 an-node02 fenced[12403]: fencing node an-node01.alteeve.com
Sep 15 16:18:31 an-node02 fenced[12403]: fence an-node01.alteeve.com dev 0.0 agent fence_ipmilan result: error from agent
Sep 15 16:18:31 an-node02 fenced[12403]: fence an-node01.alteeve.com success

Hoozah!

Notice that there is an error from the fence_ipmilan. This is exactly what we expected because of the IPMI BMC losing power.

So now we know that an-node01 can be fenced successfully from both fence devices. Now we need to run the same tests against an-node02.

Hanging an-node02

Warning: DO NOT ASSUME THAT an-node02 WILL FENCE PROPERLY JUST BECAUSE an-node01 PASSED!. There are many ways that a fence could fail; Bad password, misconfigured device, plugged into the wrong port on the PDU and so on. Always test all nodes using all methods!

Be sure to be tailing the /var/log/messages on an-node02. Go to an-node01's first terminal and run the following command.

Note: This command will not return and you will lose all ability to talk to this node until it is rebooted.

On an-node02 run:

echo c > /proc/sysrq-trigger

On an-node01's syslog terminal, you should see the following entries in the log.

Sep 15 15:26:19 an-node01 corosync[2223]:   [TOTEM ] A processor failed, forming new configuration.
Sep 15 15:26:21 an-node01 corosync[2223]:   [QUORUM] Members[1]: 1
Sep 15 15:26:21 an-node01 corosync[2223]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 15 15:26:21 an-node01 corosync[2223]:   [CPG   ] downlist received left_list: 1
Sep 15 15:26:21 an-node01 corosync[2223]:   [CPG   ] chosen downlist from node r(0) ip(10.20.0.1) 
Sep 15 15:26:21 an-node01 corosync[2223]:   [MAIN  ] Completed service synchronization, ready to provide service.
Sep 15 15:26:21 an-node01 fenced[2280]: fencing node an-node02.alteeve.com
Sep 15 15:26:21 an-node01 kernel: dlm: closing connection to node 2
Sep 15 15:26:36 an-node01 fenced[2280]: fence an-node02.alteeve.com success

Again, perfect!

Cutting the Power to an-node02

From an-node01, pull the power on an-node02 with the following call;

fence_apc -a pdu2.alteeve.com -l apc -p secret -n 2 -o off

Success: Powered OFF

Back on an-node01's syslog, we should see the following entries;

Sep 15 15:36:30 an-node01 corosync[2223]:   [TOTEM ] A processor failed, forming new configuration.
Sep 15 15:36:32 an-node01 corosync[2223]:   [QUORUM] Members[1]: 1
Sep 15 15:36:32 an-node01 corosync[2223]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 15 15:36:32 an-node01 kernel: dlm: closing connection to node 2
Sep 15 15:36:32 an-node01 corosync[2223]:   [CPG   ] downlist received left_list: 1
Sep 15 15:36:32 an-node01 corosync[2223]:   [CPG   ] chosen downlist from node r(0) ip(10.20.0.1) 
Sep 15 15:36:32 an-node01 corosync[2223]:   [MAIN  ] Completed service synchronization, ready to provide service.
Sep 15 15:36:32 an-node01 fenced[2280]: fencing node an-node02.alteeve.com
Sep 15 15:36:55 an-node01 fenced[2280]: fence an-node02.alteeve.com dev 0.0 agent fence_ipmilan result: error from agent
Sep 15 15:36:55 an-node01 fenced[2280]: fence an-node02.alteeve.com success

Woot!

We can now safely say that our fencing is setup and working properly.

Testing Network Redundancy

Next up of the testing block is our network configuration. Seeing as we've build our bonds, we need to now test that they are working properly.

To run this test, we're going to do the following;

• Make sure that cman has started on both nodes.
• On both nodes, start a ping flood against the opposing node in the first window and start tailing syslog in the second window.
• Look at the current state of the bonds to see which interfaces are active.
• Pull the power on the switch those interfaces are using. If the interfaces are spread across both switches, don't worry. Pick one and we will kill it again later.
• Check the state of the bonds again and see that they've switched to their backup links. If a node gets fenced, you know something went wrong.
• Wait about a minute, then restore power to the lost switch. Wait a good five minutes to ensure that it is in fact back up and that the network was not interrupted.
• Repeat the power off/on cycle for the second switch.
• If the initial state had the bonds spread across both switches, repeat the power off/on for the first switch.

If all of these steps pass and the cluster doesn't partition, then you can be confident that your network is configured properly for full redundancy.

How to Know if the Tests Passed

Well, the most obvious answer to this question is if the cluster is still working after a switch is powered off.

We can be a little more subtle than that though.

The state of each bond is viewable by looking in the special /proc/net/bonding/bondX files, where X is the bond number. Lets take a look at bond0 on an-node01.

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:e0:81:c7:ec:49

Slave Interface: eth3
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1b:21:9d:59:fc

We can see that the currently active interface is eth0. This is the key bit we're going to be watching for these tests. I know that eth0 on an-node01 is connected to by first switch. So when I pull the power to that switch, I should see eth3 take over.

We'll also be watching syslog. If things work right, we should not see any messages from the cluster during failure and recovery.

If you have the screen space for it, I'd recommend opening six more terminal windows, one for each bond. Run watch cat /proc/net/bonding/bondX so that you can quickly see any change in the bond states. Below's an example of the layout I use for this test.

Terminal layout used for monitoring the bonded link status during HA network testing. The right window shows two columns of terminals, an-node01 on the left and an-node02 on the right, stacked into three rows, bond0 on the top, bond1 in the middle and bond2 at the bottom. The left window shows the standard tail on syslog.

Failing The First Switch

In my case, all of the bonds on both nodes are using their first links as the current active links. This means that all network traffic is going through the first switch. So I will first power down that switch. You need to sort out which switch you should shut down first. If you've got the network traffic going over both switches, then just pick on to start with.

Note: Make sure that cman is running before beginning the test!

After killing the switch, I can see in syslog the following messages:

Sep 16 13:12:46 an-node01 kernel: e1000e: eth2 NIC Link is Down
Sep 16 13:12:46 an-node01 kernel: e1000e: eth0 NIC Link is Down
Sep 16 13:12:46 an-node01 kernel: e1000e: eth1 NIC Link is Down
Sep 16 13:12:46 an-node01 kernel: bonding: bond1: link status definitely down for interface eth1, disabling it
Sep 16 13:12:46 an-node01 kernel: bonding: bond1: making interface eth4 the new active one.
Sep 16 13:12:46 an-node01 kernel: bonding: bond0: link status definitely down for interface eth0, disabling it
Sep 16 13:12:46 an-node01 kernel: bonding: bond0: making interface eth3 the new active one.
Sep 16 13:12:46 an-node01 kernel: device eth0 left promiscuous mode
Sep 16 13:12:46 an-node01 kernel: device eth3 entered promiscuous mode
Sep 16 13:12:46 an-node01 kernel: bonding: bond2: link status definitely down for interface eth2, disabling it
Sep 16 13:12:46 an-node01 kernel: bonding: bond2: making interface eth5 the new active one.
Sep 16 13:12:46 an-node01 kernel: device eth2 left promiscuous mode
Sep 16 13:12:46 an-node01 kernel: device eth5 entered promiscuous mode

I can look at an-node01's /proc/net/bonding/bond0 file and see:

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth3
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: down
Link Failure Count: 1
Permanent HW addr: 00:e0:81:c7:ec:49

Slave Interface: eth3
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1b:21:9d:59:fc

Notice Currently Active Slave is now eth3? You can also see now that eth0's link is down (MII Status: down).

It's the same story for all the other bonds on both switches.

If we check the status of the cluster, we'll see that all is good.

cman_tool status

Version: 6.2.0
Config Version: 8
Cluster Name: an-clusterA
Cluster Id: 29382
Cluster Member: Yes
Cluster Generation: 72
Membership state: Cluster-Member
Nodes: 2
Expected votes: 1
Total votes: 2
Node votes: 1
Quorum: 1  
Active subsystems: 7
Flags: 2node 
Ports Bound: 0  
Node name: an-node01.alteeve.com
Node ID: 1
Multicast addresses: 239.192.114.57 
Node addresses: 10.20.0.1

How cool is that?!

Restoring The First Switch

Now that we've confirmed all of the bonds are working on the backup switch, lets restore power to the first switch.

Warning: Be sure to wait a solid five minutes after restoring power before declaring the recovery a success!

It is very important to wait for a while after restoring power to the switch. Some of the common problems that can break your cluster will not show up immediately. A good example is a misconfiguration of STP. In this case, the switch will come up, a short time will pass and then the switch will trigger an STP reconfiguration. Once this happens, both switches will block traffic for many seconds. This will partition you cluster.

So then, lets power it back up.

Within a few moments, you should see this in your syslog;

Sep 16 13:23:57 an-node01 kernel: e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:23:57 an-node01 kernel: e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:23:57 an-node01 kernel: bonding: bond0: link status definitely up for interface eth0.
Sep 16 13:23:57 an-node01 kernel: bonding: bond1: link status definitely up for interface eth1.
Sep 16 13:23:58 an-node01 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:23:58 an-node01 kernel: bonding: bond2: link status definitely up for interface eth2.

It looks up, but lets keep waiting for another minute (note the time stamps).

Sep 16 13:24:52 an-node01 kernel: e1000e: eth0 NIC Link is Down
Sep 16 13:24:52 an-node01 kernel: bonding: bond0: link status definitely down for interface eth0, disabling it
Sep 16 13:24:53 an-node01 kernel: e1000e: eth1 NIC Link is Down
Sep 16 13:24:53 an-node01 kernel: bonding: bond1: link status definitely down for interface eth1, disabling it
Sep 16 13:24:54 an-node01 kernel: e1000e: eth2 NIC Link is Down
Sep 16 13:24:54 an-node01 kernel: bonding: bond2: link status definitely down for interface eth2, disabling it
Sep 16 13:24:55 an-node01 kernel: e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:24:55 an-node01 kernel: bonding: bond0: link status definitely up for interface eth0.
Sep 16 13:24:56 an-node01 kernel: e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:24:56 an-node01 kernel: bonding: bond1: link status definitely up for interface eth1.
Sep 16 13:24:57 an-node01 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:24:57 an-node01 kernel: bonding: bond2: link status definitely up for interface eth2.
Sep 16 13:24:58 an-node01 kernel: e1000e: eth0 NIC Link is Down
Sep 16 13:24:58 an-node01 kernel: bonding: bond0: link status definitely down for interface eth0, disabling it
Sep 16 13:24:59 an-node01 kernel: e1000e: eth1 NIC Link is Down
Sep 16 13:24:59 an-node01 kernel: bonding: bond1: link status definitely down for interface eth1, disabling it
Sep 16 13:25:00 an-node01 kernel: e1000e: eth2 NIC Link is Down
Sep 16 13:25:00 an-node01 kernel: bonding: bond2: link status definitely down for interface eth2, disabling it
Sep 16 13:25:00 an-node01 kernel: e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:25:00 an-node01 kernel: bonding: bond0: link status definitely up for interface eth0.
Sep 16 13:25:02 an-node01 kernel: e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:25:02 an-node01 kernel: bonding: bond1: link status definitely up for interface eth1.
Sep 16 13:25:02 an-node01 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:25:02 an-node01 kernel: bonding: bond2: link status definitely up for interface eth2.

See all that bouncing? That is caused by many switches showing a link (that is the MII status) without actually being able to push traffic. As part of the switches boot sequence, the links will go down and come back up a couple of times.

This is partly why the updelay option exists in the BONDING_OPTS, but it is also why we don't set primary or enable _reselect. Every time the active slave interface changes, there is a small chance of a problem that could break the cluster.

You will notice after several minutes that the backup slave interfaces are still in use, despite the first switch being back online. This is just fine. We can check the cluster status again and we'll see that everything is still fine. The recovery test passed!

Failing The Second Switch

For the same reason that we need to test all fence devices from both nodes, we also need to test failure and recovery of both switches. So now lets pull the plug on the second switch!

As before, we'll see messages showing the interfaces dropping.

Sep 16 13:35:36 an-node01 kernel: e1000e: eth3 NIC Link is Down
Sep 16 13:35:36 an-node01 kernel: bonding: bond0: link status definitely down for interface eth3, disabling it
Sep 16 13:35:36 an-node01 kernel: bonding: bond0: making interface eth0 the new active one.
Sep 16 13:35:36 an-node01 kernel: device eth3 left promiscuous mode
Sep 16 13:35:36 an-node01 kernel: device eth0 entered promiscuous mode
Sep 16 13:35:38 an-node01 kernel: e1000e: eth5 NIC Link is Down
Sep 16 13:35:38 an-node01 kernel: bonding: bond2: link status definitely down for interface eth5, disabling it
Sep 16 13:35:38 an-node01 kernel: bonding: bond2: making interface eth2 the new active one.
Sep 16 13:35:38 an-node01 kernel: device eth5 left promiscuous mode
Sep 16 13:35:38 an-node01 kernel: device eth2 entered promiscuous mode
Sep 16 13:35:39 an-node01 kernel: e1000e: eth4 NIC Link is Down
Sep 16 13:35:39 an-node01 kernel: bonding: bond1: link status definitely down for interface eth4, disabling it
Sep 16 13:35:39 an-node01 kernel: bonding: bond1: making interface eth1 the new active one.

Let's take a look at an-node01's bond0 again.

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 3
Permanent HW addr: 00:e0:81:c7:ec:49

Slave Interface: eth3
MII Status: down
Link Failure Count: 1
Permanent HW addr: 00:1b:21:9d:59:fc

We can see that eth0 has returned to the active slave and that eth3 is now down. Again, it's the same story across the other bonds and cman_tool status shows that all is right in the world.

Restoring The Second Switch

Again, we're going to wait a good five minutes after restoring power before calling this test a success.

Checking out an-node01's syslog, we see the links came back and didn't bounce. These are two identical switches and should behave the same but didn't. This is a good example of why you need to test everything, even when you have identical hardware. You just can't guess how things will behave until you test and see for yourself.

Sep 16 13:53:54 an-node01 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:53:54 an-node01 kernel: e1000e: eth5 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:53:54 an-node01 kernel: bonding: bond0: link status definitely up for interface eth3.
Sep 16 13:53:54 an-node01 kernel: bonding: bond2: link status definitely up for interface eth5.
Sep 16 13:53:55 an-node01 kernel: e1000e: eth4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 16 13:53:55 an-node01 kernel: bonding: bond1: link status definitely up for interface eth4.

Now we're done! We can truly say we've got a full high-availability network configuration that is tested and trusted!

Installing DRBD

DRBD is an open-source application for real-time, block-level disk replication created and maintained by Linbit. We will use this to keep the data on our cluster consistent between the two nodes.

To install it, we have two choices;

• Install from source files.
• Install from ELRepo.

Installing from source ensures that you have full control over the installed software. However, you become solely responsible for installing future patches and bugfixes.

Installing from ELRepo means seceding some control to the ELRepo maintainers, but it also means that future patches and bugfixes are applied as part of a standard update.

Which you choose is, ultimately, a decision you need to make.

Option A - Install From Source

On Both nodes run:

# Obliterate peer - fence via cman
wget -c https://alteeve.com/files/an-cluster/sbin/obliterate-peer.sh -O /sbin/obliterate-peer.sh
chmod a+x /sbin/obliterate-peer.sh
ls -lah /sbin/obliterate-peer.sh

# Download, compile and install DRBD
wget -c http://oss.linbit.com/drbd/8.3/drbd-8.3.11.tar.gz
tar -xvzf drbd-8.3.11.tar.gz
cd drbd-8.3.11
./configure \
   --prefix=/usr \
   --localstatedir=/var \
   --sysconfdir=/etc \
   --with-utils \
   --with-km \
   --with-udev \
   --with-pacemaker \
   --with-rgmanager \
   --with-bashcompletion
make
make install
chkconfig --add drbd
chkconfig drbd off

Option B - Install From ELRepo

On Both nodes run:

# Obliterate peer - fence via cman
wget -c https://alteeve.com/files/an-cluster/sbin/obliterate-peer.sh -O /sbin/obliterate-peer.sh
chmod a+x /sbin/obliterate-peer.sh
ls -lah /sbin/obliterate-peer.sh

# Install the ELRepo GPG key, add the repo and install DRBD.
rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm
yum install drbd83-utils kmod-drbd83

Creating The DRBD Partitions

It is possible to use LVM on the hosts, and simply create LVs to back our DRBD resources. However, this causes confusion as LVM will see the PV signatures on both the DRBD backing devices and the DRBD device itself. Getting around this requires editing LVM's filter option, which is somewhat complicated. Not overly so, mind you, but enough to be outside the scope of this document.

Also, by working with fdisk directly, it will give us a chance to make sure that the DRBD partitions start on an even 64 KiB boundry. This is important for decent performance on Windows VMs, as we will see later. This is true for both traditional platter and modern solid-state drives.

On our nodes, we created three primary disk partitions;

• /dev/sda1; The /boot partition.
• /dev/sda2; The root / partition.
• /dev/sda3; The swap partition.

We will create a new extended partition. Then within it we will create three new partitions;

• /dev/sda5; a small partition we will later use for our shared GFS2 partition.
• /dev/sda6; a partition big enough to host the VMs that will normally run on an-node01.
• /dev/sda7; a partition big enough to host the VMs that will normally run on an-node02.

As we create each partition, we will do a little math to ensure that the start sector is on a 64 KiB boundry.

Alignment Math

Before we can start the alignment math, we need to know how big each sector is on our hard drive. This is almost always 512 bytes, but it's still best to be sure. To check, run;

fdisk -l /dev/sda | grep Sector

Sector size (logical/physical): 512 bytes / 512 bytes

So now that we have confirmed our sector size, we can look at the math.

• Each 64 KiB block will use 128 sectors ((64 * 1024) / 512) == 128.
• As we create each each partition, we will be asked to enter the starting sector (using fdisk -u). Take the first free sector and divide it by 128. If it does not divide evenly, then;
  • Add 127 (one sector shy of another block to guarantee we've gone past the start sector we want).
  • Divide the new number by 128. This will give you a fractional number. Remove (do not round!) any number after the decimal place.
  • Multiply by 128 to get the sector number we want.

Lets look at a example using real numbers. Lets say we create a new partition and the first free sector is 92807568;

92807568 ÷ 128 = 725059.125

We have a remainder, so it's not on an even 64 KiB block boundry. Now we need to figure out what sector above 92807568 is evenly divisible by 128. To do that, lets add 127 (one sector shy of the next 64 KiB block), divide by 128 to get the number of 64 KiB blocks (with a remainder), remove the remainder to get an even number (do not round, you just want the bare integer), then finally multiply by 128 to get the sector number. This will give us the sector number we want our partition to start on.

92807568 + 127 = 92807695

92807695 ÷ 128 = 725060.1171875

int(725060.1171875) = 725060

725060 x 128 = 92807680

So now we know that sector number 92807680 is the first sector above 92807568 that falls on an even 64 KiB block. Now we need to alter our partition's starting sector. To do this, we will need to go into fdisk's extra functions.

Note: Pay attention to the last sector number of each partition you create. As you create partitions, fdisk will see free space, as tiny as it is, and it will default to that as the first sector for the next partition. This is annoying. My noting the last sector of each partition you create, you can add 1 sector and do the math to find the first sector above that which sits on a 64 KiB boundary.

Creating the Three Partitions

Here I will show you the values I entered to create the three partitions I needed on my nodes.

DO NOT COPY THIS!

The values you enter will almost certainly be different.

Start fdisk in sector mode on /dev/sda.

Note: If you are using software RAID, you will need to do the following steps on all disks, then you can proceed to create the RAID partitions normally and they will be aligned.

fdisk -u /dev/sda

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c').

Disable DOS compatibility because hey, it's not the 80s any more.

Command (m for help): c

DOS Compatibility flag is not set

Lets take a look at the current partition layout.

Command (m for help): p

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00056856

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      526335      262144   83  Linux
/dev/sda2          526336    84412415    41943040   83  Linux
/dev/sda3        84412416    92801023     4194304   82  Linux swap / Solaris

Perfect. Now let's create a new extended partition that will use the rest of the disk. We don't care if this is aligned so we'll just accept the default start and end sectors.

Command (m for help): n

Command action
   e   extended
   p   primary partition (1-4)

e

Selected partition 4
First sector (92801024-976773167, default 92801024):

Just press <enter>.

Using default value 92801024
Last sector, +sectors or +size{K,M,G} (92801024-976773167, default 976773167):

Just press <enter> again.

Using default value 976773167

Now we'll create the first partition. This will be a 20GB partition used by the shared GFS2 partition. As it will never host a VM, I don't care if it is aligned.

Command (m for help): n

First sector (92803072-976773167, default 92803072):

Just press <enter>.

Using default value 92803072

Last sector, +sectors or +size{K,M,G} (92803072-976773167, default 976773167): +20G

Now we will create the last two partitions that will host our VMs. I want to split the remaining space in half, so I need to do a little bit more math before I can proceed. I will need to see how many sectors are still free, divide by two to get the number of sectors in half the remaining free space, the add the number of already-used sectors so that I know where the first partition should end. We'll do this math in just a moment.

So let's print the current partition layout:

Command (m for help): p

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00056856

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      526335      262144   83  Linux
/dev/sda2          526336    84412415    41943040   83  Linux
/dev/sda3        84412416    92801023     4194304   82  Linux swap / Solaris
/dev/sda4        92801024   976773167   441986072    5  Extended
/dev/sda5        92803072   134746111    20971520   83  Linux

Start to create the new partition. Before be can sort out the last sector, we first need to find the first sector.

Command (m for help): n

First sector (134748160-976773167, default 134748160):

Now I see that it the first free sector is 134748160. I divide this by 128 and I get 1052720. It is an even number, so I don't need to do anything more as it is already on a 64 KiB boundry! So I can just press <enter> to accept it.

Using default value 134748160
Last sector, +sectors or +size{K,M,G} (134748160-976773167, default 976773167):

Now we need to do the math to find what sector marks half of the remaining free space. Let's gather some numbers;

• This partition started at sector 134748160
• The default end sector is 976773167
• That means that there are currently (976773167 - 134748160) == 842025007 sectors free.
• Half of that is (842025007 / 2) == int(421012503.5) == 421012503 sectors free (int() simply means to take the remainder off the number).
• So if we want a partition that is 421012503 long, we need to add the start sector to get our offset. That is, (421012503 + 134748160) == 555760663. This is what we will enter now.

Last sector, +sectors or +size{K,M,G} (134748160-976773167, default 976773167): 555760663

Now to create the last partition, we will repeat the steps above.

Command (m for help): n

First sector (555762712-976773167, default 555762712):

Let's make sure that 555762712 is on a 64 KiB boundry;

• (555762712 / 128) == 4341896.1875 is not an even number, so we need to find the next sector on an even boundary.
• Add 127 sectors and divide by 128 again;
  • (555762712 + 127) == 555762839
  • (555762839 / 128) == int(4341897.1796875) == 4341897
  • (4341897 * 128) == 555762816
• Now we know that we want our start sector to be 555762816.

First sector (555762712-976773167, default 555762712): 555762816

Last sector, +sectors or +size{K,M,G} (555762816-976773167, default 976773167):

This is the last partition, so we can just press <enter> to get the last sector on the disk.

Using default value 976773167

Lets take a final look at the new partition before committing the changes to disk.

Command (m for help): p

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00056856

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      526335      262144   83  Linux
/dev/sda2          526336    84412415    41943040   83  Linux
/dev/sda3        84412416    92801023     4194304   82  Linux swap / Solaris
/dev/sda4        92801024   976773167   441986072    5  Extended
/dev/sda5        92803072   134746111    20971520   83  Linux
/dev/sda6       134748160   555760663   210506252   83  Linux
/dev/sda7       555762816   976773167   210505176   83  Linux

Perfect. If you divide partition six or seven's start sector by 128, you will see that both have no remainder which means that they are, if fact, aligned. This is the last time we need to worry about alignment because LVM uses an even multiple of 64 KiB in it's extent sizes, so all normal extent sized will always produce LVs on even 64 KiB boudaries.

So now write out the changes, re-probe the disk (or reboot) and then repeat all these steps on the other node.

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

No reprobe using partprobe.

partprobe /dev/sda

Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda 
(Device or resource busy).  As a result, it may not reflect all of your changes 
until after reboot.

In my case, the probe failed so I will reboot. To do this most safely, stop the cluster before calling reboot.

/etc/init.d/cman stop

Stopping cluster: 
   Leaving fence domain...                                 [  OK  ]
   Stopping gfs_controld...                                [  OK  ]
   Stopping dlm_controld...                                [  OK  ]
   Stopping fenced...                                      [  OK  ]
   Stopping cman...                                        [  OK  ]
   Waiting for corosync to shutdown:                       [  OK  ]
   Unloading kernel modules...                             [  OK  ]
   Unmounting configfs...                                  [  OK  ]

Now reboot.

reboot

Configuring DRBD

DRBD is configured in two parts;

• Global and common configuration options
• Resource configurations

We will be creating three separate DRBD resources, so we will create three separate resource configuration files. More on that in a moment.

Configuring DRBD Global and Common Options

The first file to edit is /etc/drbd.d/global_common.conf. In this file, we will set global configuration options and set default resource configuration options. These default resource options can be overwritten in the actual resource files which we'll create once we're done here.

cp /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.conf.orig
vim /etc/drbd.d/global_common.conf 
diff -u  /etc/drbd.d/global_common.conf.orig /etc/drbd.d/global_common.conf

--- /etc/drbd.d/global_common.conf.orig	2011-09-14 14:03:56.364566109 -0400
+++ /etc/drbd.d/global_common.conf	2011-09-14 14:23:37.287566400 -0400
@@ -15,24 +15,81 @@
 		# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
 		# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
 		# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
+
+		# This script is a wrapper for RHCS's 'fence_node' command line
+		# tool. It will call a fence against the other node and return
+		# the appropriate exit code to DRBD.
+		fence-peer		"/sbin/obliterate-peer.sh";
 	}
 
 	startup {
 		# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
+
+		# This tells DRBD to promote both nodes to Primary on start.
+		become-primary-on	both;
+
+		# This tells DRBD to wait five minutes for the other node to
+		# connect. This should be longer than it takes for cman to
+		# timeout and fence the other node *plus* the amount of time it
+		# takes the other node to reboot. If you set this too short,
+		# you could corrupt your data. If you want to be extra safe, do
+		# not use this at all and DRBD will wait for the other node 
+		# forever.
+		wfc-timeout		300;
+		
+		# This tells DRBD to wait for the other node for three minutes
+		# if the other node was degraded the last time it was seen by
+		# this node. This is a way to speed up the boot process when
+		# the other node is out of commission for an extended duration.
+		degr-wfc-timeout	120;
 	}
 
 	disk {
 		# on-io-error fencing use-bmbv no-disk-barrier no-disk-flushes
 		# no-disk-drain no-md-flushes max-bio-bvecs
+
+		# This tells DRBD to block IO and fence the remote node (using
+		# the 'fence-peer' helper) when connection with the other node 
+		# is unexpectedly lost. This is what helps prevent split-brain
+		# condition and it is incredible important in dual-primary 
+		# setups!
+		fencing			resource-and-stonith;
 	}
 
 	net {
 		# sndbuf-size rcvbuf-size timeout connect-int ping-int ping-timeout max-buffers
 		# max-epoch-size ko-count allow-two-primaries cram-hmac-alg shared-secret
 		# after-sb-0pri after-sb-1pri after-sb-2pri data-integrity-alg no-tcp-cork
+
+		# This tells DRBD to allow two nodes to be Primary at the same
+		# time. It is needed when 'become-primary-on both' is set.
+		allow-two-primaries;
+
+		# The following three commands tell DRBD how to react should 
+		# our best efforts fail and a split brain occurs. You can learn
+		# more about these options by reading the drbd.conf man page.
+		# NOTE! It is not possible to safely recover from a split brain
+		# where both nodes were primary. This care requires human
+		# intervention, so 'disconnect' is the only safe policy.
+		after-sb-0pri		discard-zero-changes;
+		after-sb-1pri		discard-secondary;
+		after-sb-2pri		disconnect;
 	}
 
 	syncer {
 		# rate after al-extents use-rle cpu-mask verify-alg csums-alg
+
+		# This alters DRBD's default syncer rate. Note that is it 
+		# *very* important that you do *not* configure the syncer rate
+		# to be too fast. If it is too fast, it can significantly 
+		# impact applications using the DRBD resource. If it's set to a
+		# rate higher than the underlying network and storage can 
+		# handle, the sync can stall completely.
+		# This should be set to ~30% of the *tested* sustainable read 
+		# or write speed of the raw /dev/drbdX device (whichever is 
+		# slower). In this example, the underlying resource was tested
+		# as being able to sustain roughly 60 MB/sec, so this is set to
+		# one third of that rate, 20M.
+		rate			20M;
 	}
 }

Configuring the DRBD Resources

As mentioned earlier, we are going to create three DRBD resources.

• Resource r0, which will be device /dev/drbd0, will be the shared GFS2 partition.
• Resource r1, which will be device /dev/drbd1, will provide disk space for VMs that will normally run on an-node01.
• Resource r2, which will be device /dev/drbd2, will provide disk space for VMs that will normally run on an-node02.

Note: The reason for the two separate VM resources is to help protect against data loss in the off chance that a split-brain occurs, despite our counter-measures. As we will see later, recovering from a split brain requires discarding the changes on one side of the resource. If VMs are running on the same resource but on different nodes, this would lead to data loss. Using two resources helps prevent that scenario.

Each resource configuration will be in it's own file saved as /etc/drbd.d/rX.res. The three of them will be pretty much the same. So let's take a look at the first GFS2 resource r0.res, then we'll just look at the changes for r1.res and r2.res. These files won't exist initially.

vim /etc/drbd.d/r0.res

# This is the resource used for the shared GFS2 partition.
resource r0 {
	# This is the block device path.
        device          /dev/drbd0;

	# We'll use the normal internal metadisk (takes about 32MB/TB)
        meta-disk       internal;

	# This is the `uname -n` of the first node
        on an-node01.alteeve.com {
		# The 'address' has to be the IP, not a hostname. This is the
		# node's SN (bond1) IP. The port number must be unique amoung
		# resources.
                address         10.10.0.1:7789;

		# This is the block device backing this resource on this node.
                disk            /dev/sda5;
        }
	# Now the same information again for the second node.
        on an-node02.alteeve.com {
                address         10.10.0.2:7789;
                disk            /dev/sda5;
        }
}

Now copy this to r1.res and edit for the an-node01 VM resource. The main differences are the resource name, r1, the block device, /dev/drbd1, the port, 7790 and the backing block devices, /dev/sda6.

cp /etc/drbd.d/r0.res /etc/drbd.d/r1.res
vim /etc/drbd.d/r1.res

# This is the resource used for VMs that will normally run on an-node01.
resource r1 {
	# This is the block device path.
        device          /dev/drbd1;

	# We'll use the normal internal metadisk (takes about 32MB/TB)
        meta-disk       internal;

	# This is the `uname -n` of the first node
        on an-node01.alteeve.com {
		# The 'address' has to be the IP, not a hostname. This is the
		# node's SN (bond1) IP. The port number must be unique amoung
		# resources.
                address         10.10.0.1:7790;

		# This is the block device backing this resource on this node.
                disk            /dev/sda6;
        }
	# Now the same information again for the second node.
        on an-node02.alteeve.com {
                address         10.10.0.2:7790;
                disk            /dev/sda6;
        }
}

The last resource is again the same, with the same set of changes.

cp /etc/drbd.d/r1.res /etc/drbd.d/r2.res
vim /etc/drbd.d/r2.res

# This is the resource used for VMs that will normally run on an-node02.
resource r2 {
	# This is the block device path.
        device          /dev/drbd2;

	# We'll use the normal internal metadisk (takes about 32MB/TB)
        meta-disk       internal;

	# This is the `uname -n` of the first node
        on an-node01.alteeve.com {
		# The 'address' has to be the IP, not a hostname. This is the
		# node's SN (bond1) IP. The port number must be unique amoung
		# resources.
                address         10.10.0.1:7791;

		# This is the block device backing this resource on this node.
                disk            /dev/sda7;
        }
	# Now the same information again for the second node.
        on an-node02.alteeve.com {
                address         10.10.0.2:7791;
                disk            /dev/sda7;
        }
}

The final step is to validate the configuration. This is done by running the following command;

drbdadm dump

# /etc/drbd.conf
common {
    protocol               C;
    net {
        allow-two-primaries;
        after-sb-0pri    discard-zero-changes;
        after-sb-1pri    discard-secondary;
        after-sb-2pri    disconnect;
    }
    disk {
        fencing          resource-and-stonith;
    }
    syncer {
        rate             20M;
    }
    startup {
        wfc-timeout      300;
        degr-wfc-timeout 120;
        become-primary-on both;
    }
    handlers {
        pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
        pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
        local-io-error   "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
        fence-peer       /sbin/obliterate-peer.sh;
    }
}

# resource r0 on an-node01.alteeve.com: not ignored, not stacked
resource r0 {
    on an-node01.alteeve.com {
        device           /dev/drbd0 minor 0;
        disk             /dev/sda5;
        address          ipv4 10.10.0.1:7789;
        meta-disk        internal;
    }
    on an-node02.alteeve.com {
        device           /dev/drbd0 minor 0;
        disk             /dev/sda5;
        address          ipv4 10.10.0.2:7789;
        meta-disk        internal;
    }
}

# resource r1 on an-node01.alteeve.com: not ignored, not stacked
resource r1 {
    on an-node01.alteeve.com {
        device           /dev/drbd1 minor 1;
        disk             /dev/sda6;
        address          ipv4 10.10.0.1:7790;
        meta-disk        internal;
    }
    on an-node02.alteeve.com {
        device           /dev/drbd1 minor 1;
        disk             /dev/sda6;
        address          ipv4 10.10.0.2:7790;
        meta-disk        internal;
    }
}

# resource r2 on an-node01.alteeve.com: not ignored, not stacked
resource r2 {
    on an-node01.alteeve.com {
        device           /dev/drbd2 minor 2;
        disk             /dev/sda7;
        address          ipv4 10.10.0.1:7791;
        meta-disk        internal;
    }
    on an-node02.alteeve.com {
        device           /dev/drbd2 minor 2;
        disk             /dev/sda7;
        address          ipv4 10.10.0.2:7791;
        meta-disk        internal;
    }
}

You'll note that the output is formatted differently, but the values themselves are the same. If there had of been errors, you would have seen them printed. Fix any problems before proceeding. Once you get a clean dump, copy the configuration over to the other node.

rsync -av /etc/drbd.d root@an-node02:/etc/

sending incremental file list
drbd.d/
drbd.d/global_common.conf
drbd.d/global_common.conf.orig
drbd.d/r0.res
drbd.d/r1.res
drbd.d/r2.res

sent 7619 bytes  received 129 bytes  15496.00 bytes/sec
total size is 7946  speedup is 1.03

Initializing The DRBD Resources

Now that we have DRBD configured, we need to initialize the DRBD backing devices and then bring up the resources for the first time.

Note: To save a bit of time and typing, the following sections will use a little bash magic. When commands need to be run on all three resources, rather than running the same command three times with the different resource names, we will use the short-hand form r{0,1,2} or r{0..2}.

On both nodes, create the new metadata on the backing devices. You may need to type yes to confirm the action if any data is seen. If DRBD sees an actual file system, it will error and insist that you clear the partition. You can do this by running; dd if=/dev/zero of=/dev/sdaX bs=4M count=1000, where X is the partition you want to clear.

drbdadm create-md r{0..2}

md_offset 21474832384
al_offset 21474799616
bm_offset 21474144256

Found some data

 ==> This might destroy existing data! <==

Do you want to proceed?

[need to type 'yes' to confirm] yes

Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
md_offset 215558397952
al_offset 215558365184
bm_offset 215551782912

Found some data

 ==> This might destroy existing data! <==

Do you want to proceed?

[need to type 'yes' to confirm] yes

Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
md_offset 215557296128
al_offset 215557263360
bm_offset 215550681088

Found some data

 ==> This might destroy existing data! <==

Do you want to proceed?

[need to type 'yes' to confirm] yes

Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success

Before you go any further, we'll need to load the drbd kernel module. Note that you won't normally need to do this. Later, after we get everything running the first time, we'll be able to start and stop the DRBD resources using the /etc/init.d/drbd script, which loads and unloads the drbd kernel module as needed.

modprobe drbd

Now go back to the terminal windows we had used to watch the cluster start. We now want to watch the output of cat /proc/drbd so we can keep tabs on the current state of the DRBD resources. We'll do this by using the watch program, which will refresh the output of the cat call every couple of seconds.

watch cat /proc/drbd

version: 8.3.11 (api:88/proto:86-96)
GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by dag@Build64R6, 2011-08-08 08:54:05

Back in the first terminal, we need to attach the backing device, /dev/sda{5..7} to their respective DRBD resources, r{0..2}. After running the following command, you will see no output on the first terminal, but the second terminal's /proc/drbd should update.

drbdadm attach r{0..2}

version: 8.3.11 (api:88/proto:86-96)
GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by dag@Build64R6, 2011-08-08 08:54:05
 0: cs:StandAlone ro:Secondary/Unknown ds:Inconsistent/DUnknown   r----s
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:20970844
 1: cs:StandAlone ro:Secondary/Unknown ds:Inconsistent/DUnknown   r----s
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:210499788
 2: cs:StandAlone ro:Secondary/Unknown ds:Inconsistent/DUnknown   r----s
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:210498712

Take note of the connection state, cs:StandAlone, the current role, ro:Secondary/Unknown and the disk state, ds:Inconsistent/DUnknown. This tells us that our resources are not talking to one another, are not usable because they are in the Secondary state (you can't even read the /dev/drbdX device) and that the backing device does not have an up to date view of the data.

This all makes sense of course, as the resources are brand new.

So the next step is to connect the two nodes together. As before, we won't see any output from the first terminal, but the second terminal will change.

Note: After running the following command on the first node, it's connection state will become cs:WFConnection which means that it is waiting for a connection from the other node.

drbdadm connect r{0..2}

version: 8.3.11 (api:88/proto:86-96)
GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by dag@Build64R6, 2011-08-08 08:54:05
 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:20970844
 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:210499788
 2: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:210498712

We can now see that the two nodes are talking to one another properly as the connection state has changed to cs:Connected. They can see that their peer node is in the same state as they are; Secondary/Inconsistent.

Seeing as the resources are brand new, there is no data to synchronize or save. So we're going to issue a special command that will only ever be used this one time. It will tell DRBD to immediately consider the DRBD resources to be up to date.

On one node only, run;

drbdadm -- --clear-bitmap new-current-uuid r{0..2}

As before, look to the second terminal to see the new state of affairs.

version: 8.3.11 (api:88/proto:86-96)
GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by dag@Build64R6, 2011-08-08 08:54:05
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
 1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
 2: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

Voila!

We could promote both sides to Primary by running drbdadm primary r{0..2} on both nodes, but there is no purpose in doing that at this stage as we can safely say our DRBD is ready to go. So instead, let's just stop DRBD entirely. We'll also prevent it from starting on boot as drbd will be managed by the cluster in a later step.

On both nodes run;

chkconfig drbd off
/etc/init.d/drbd stop

Stopping all DRBD resources: .

The second terminal will start complaining that /proc/drbd no longer exists. This is because the drbd init script unloaded the drbd kernel module.

Configuring Clustered Storage

Before we can provision the first virtual machine, we must first create the storage that will back them. This will take a few steps;

• Configuring LVM's clustered locking and creating the PVs, VGs and LVs
• Formatting and configuring the shared GFS2 partition.
• Adding storage to the cluster's resource management.

Configuring Clustered LVM Locking

Before we create the clustered LVM, we need to first make a couple of changes to the LVM configuration.

• We need to filter out the DRBD backing devices so that LVM doesn't see the same signature twice.
• Switch and enforce the locking type from local locking to clustered locking.

The configuration option to filter out the DRBD backing device is, surprisingly, filter = [ ... ]. By default, it is set to allow everything via the "a/.*/" regular expression. We're only using DRBD in our LVM, so we're going to flip that to reject everything except DRBD by changing the regex to "a|/dev/drbd*|", "r/.*/".

For the locking, we're going to change the locking_type from 1 (local locking) to 3, clustered locking. We're also going to disallow fall-back to local locking. Normal

Creating The Shared GFS2 Partition

On both

/etc/init.d/drbd start
/etc/init.d/clvmd start

On an-node01

pvcreate /dev/drbd0 && \
vgcreate -c y vg0 /dev/drbd0 && \
lvcreate -L 10G -n shared /dev/vg0

# Change this to match your cluster name
mkfs.gfs2 -p lock_dlm -j 2 -t ClusterA:shared /dev/vg0/shared

On both

mkdir /shared
mount /dev/vg0/shared /shared/
echo `gfs2_edit -p sb /dev/vg0/shared | grep sb_uuid | sed -e "s/.*sb_uuid  *\(.*\)/UUID=\L\1\E \/shared\t\tgfs2\trw,suid,dev,exec,nouser,async\t0 0/"` >> /etc/fstab
/etc/init.d/gfs2 status

On an-node01

mkdir /shared/definitions
mkdir /shared/provision

# Change this to the proper 'vmXXXX-YY'
lvcreate -l 100%free -n vm0001-01 /dev/vg0

/etc/init.d/gfs2 stop && /etc/init.d/clvmd stop && /etc/init.d/drbd stop
/etc/init.d/rgmanager start && watch clustat

Thanks

 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche!	Enterprise Support: Alteeve Support	Community Support
© Alteeve's Niche! Inc. 1997-2024		Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.