Abandoned - 2-Node EL5 Cluster

From Alteeve Wiki
Jump to navigation Jump to search

 AN!Wiki :: How To :: Abandoned - 2-Node EL5 Cluster

This paper is VERY much a work in progress. It is NOT complete and should NOT be followed. It is largely a dumping ground for my thoughts and notes as I work through the process. Please do not reference or link to this talk until this warning is removed. Really, it would be very embarrassing for both of us. :)

Related articles:

Progress

Mar. 14, 2010: Happy π-day all! I've moved the body of this paper to here and will start moving pieces back. I'm switching over to more of "choose your own adventure" style, given the various possible configurations available in clustering. I won't pretend to cover them all, but I will at least be able to insert pointers to different layouts at points where you may want to branch out from this document's path. Also, I have changed my approach to providing iSCSI/SAN and thus will remove discussion of that component until the 3+ Node CentOS5 Cluster + SoftSAN paper where this paper will act as a prerequisite.

Overview

This paper has two goals;

  1. How to assemble the simplest cluster possible, a 2 Node Cluster, which you can then expand on for your own needs.
  2. How to create a "floating" virtual machine that can move between the two nodes in the event of a node failure, maximizing up time.

Prerequisites

It is expected that you are already comfortable with the Linux command line, specifically bash, and that you are familiar with general administrative tasks in Red Hat based distributions, specifically CentOS. You will also need to be comfortable using editos like vim, nano or similar. This paper uses vim in examples. Simply substitute your favourite editor in it's place.

You are also expected to be comfortable with networking concepts. You will be expected to understand TCP/IP, multicast, broadcast, subnets and netmasks, routing and other relatively basic networking concepts. Please take the time to become familiar with these concepts before proceeding.

This said, where feasible, as much detail as is possible will be provided. For example, all configuration file locations will be shown and functioning sample files will be provided.

Platform

This paper will implement the Red Hat Cluster Suite using the CentOS binary-compatible distribution. This paper uses the x86_64 repositories, however, if you are on an i386 (32 bit) system, you should be able to following along fine. Simply replace x86_64 with .i386 in package names.

You can either download the stock CentOS 5-series DVD ISO (currently at version 5.4 which is used in this document), or you can try out the alpha AN!Cluster Install DVD. (4.3GB iso). If you use the later, please test it out on a development or test cluster. If you have any problems with the AN!Cluster variant CentOS distro, please contact me and let me know what your trouble was.

Focus

Clusters can serve to solve three problems; Reliability, Performance and Scalability.

This focus of the cluster described in this paper is primarily reliability. Second to this, scalability will be the priority leaving performance to be addressed only when it does not impact the first two criteria. This is not to indicate that performance is not a valid priority, it simply isn't the priority of this paper.

Goal

At the end of this paper, you should have a fully functioning two-node array capable of hosting a "floating" virtual machine. That is, a virtual machine that exists on one node and can be easily moved to the other node with minimal effort and down time. This should conclude with a solid foundation for adding more virtual servers up to the limit of your cluster's resources.

This paper should also server to show how to build the foundation of any other cluster configuration. This paper has a core focus of introducing the main issues that come with clustering and hopes to serve as a foundation for any cluster configuration outside the scope of this paper.

Begin

Let's begin!

Hardware

We will need two physical servers each with the following hardware:

  • One or more multi-core CPUs with Virtualization support.
  • Three network cards; At least one should be gigabit or faster.
  • One or more hard drives.

This paper uses the following hardware:

  • ASUS M4A78L-M
  • AMD Athlon II x2 250
  • 2GB Kingston DDR2 KVR800D2N6K2/4G (split between the two nodes)
  • 1x Intel 82540 PCI NICs
  • 1x D-Link DGE-560T

This is not an endorsement of the above hardware. I bought what was within my budget that would server the purposes of creating this document. What you purchase shouldn't matter, so long at the minimum requirements are met.

OS Install

Start with a stock CentOS 5.x install. This How-To uses CentOS 5.4 x86_64, however it should be fairly easy to adapt to other CentOS 5*, RHEL5 or other RHEL5-based distributions.

These are sample kickstart script used by this paper. Be sure to set your how password string and network settings.

Warning! These kickstart scripts will erase your hard drive! Adapt them, don't blindly use them.

AN!SAN kickstart files. At this time, they have no SAN or iSCSI specific options and can be used as generic cluster node configuration scripts.

AN!Cluster Install

If you are feeling brave, below is a link to a custom install DVD that contains the kickstart scripts to setup nodes and an an_cluster directory with all the configuration files.

  • Download the custom AN!Cluster Install DVD. (4.3GB iso).

Post OS Install

Once the OS is installed, we need to do some ground work.

  1. Setup networking.
  2. Limit dom0's memory.
  3. Change the default run-level.
  4. Change when xend starts.

Post-Install Network Configuration

This cluster uses Xen, which fairly dramatically impacts networking. Terms you need to be familiar with are:

  • dom0
    • This is the "first" virtual machine with special access to the underlying hardware. This looks like the host operating system but is in fact just another virtual server running under Xen. This is also the virtual machine that can directly see the Xen networking infrastructure.
  • domu
    • These are the virtual servers setup in and managed by the dom0 virtual machine. These are what most people think of when talking about "virtual servers" under Xen.

Ethernet Devices and Subnets

The most important thing to do after the install is to identify which ethX device matches which network card. This is important in two cases;

  • The fastest network card should be allocated to the DRBD partition.
  • If you have IPMI piggy-backed on a physical network card, it should be allocated to the back-channel subnet.

This paper has the following configuration:

  • eth0; Internet-polluted subnet.
  • eth1; DRBD subnet.
  • eth2; Back-channel subnet.

To change which ethX device maps to which ethernet card, please see:

If you are unfamiliar with how networking works in Xen, please read this article:

Choosing your Subnets

There will be three subnets in our two node cluster;

  • Internet-polluted subnet; 192.168.1.0/24
    • This subnet will ultimately be directly accessible only by the firewall virtual server. All other virtual machines and the node's dom0s will access the internet via the firewall for security reasons. During setup though, the 'dom0' servers will directly access this subnet.
  • DRBD subnet; 10.0.0.0/24
    • Only the two 'dom0' servers will access to this subnet. It is used for DRBD communication and as a backup for the totem ring protocol.
  • Back-channel; 10.0.1.0/24
    • This is the private subnet used for communication between the dom0 and domU virtual servers. This subnet will have no direct access to the internet. This paper will use the 10.0.1.0/24 subnet for this

I like to assign the same last octal to a given node's subnets. This helps me keep track of which node I am working with at any given time. Here is how I setup my two nodes:

  • an_san01
    • eth0: 192.168.1.71
    • eth1: 10.0.0.71
    • eth2: 10.0.1.71
  • an_san02
    • eth0: 192.168.1.72
    • eth1: 10.0.0.72
    • eth2: 10.0.1.72

/etc/hosts

Some applications expect to be able to call nodes by their name. To accommodate this, and to ensure that inter-node communication takes place on the back-channel subnet, we add the following to the /etc/hosts file: 

vim /etc/hosts

# By back-channel IPs to name mapping.
10.0.1.71	an_san01 an_san01.alteeve.com
10.0.1.72	an_san02 an_san02.alteeve.com

Note: Delete any entries matching the output from uname -a. There is a good chance there will be an entry that resolves to 127.0.0.1!

Obviously, adapt the names and IPs to match your nodes and subnets. The only critical thing is to make sure that the name returned by uname -n is resolvable to the back-channel subnet. I like to add a short-form name for convenience.

iptables

Be sure to flush netfilter tables and disable iptables and ip6tables from starting on your nodes. This is because the 'dom0' servers will not be connected directly to the Internet and we want to minimize the chance of an errant iptables rule messing up our configuration. If, before launch, you wish to implement a firewall, feel free to do so but be sure to thoroughly test your cluster to ensure no problems were introduced. 

chkconfig --level 2345 iptables off
/etc/init.d/iptables stop
chkconfig --level 2345 ip6tables off
/etc/init.d/ip6tables stop

Limit dom0's Memory

Normally, 'dom0' will claim and use memory not allocated to virtual machines. This can cause trouble if, for example, you've moved a VM off of a node and then want to move it or another VM back. For a period of time, the node will claim that there is not enough free memory for the migration. By setting a hard limit of dom0's memory usage, this scenario won't happen and you will not need to delay migrations.

To do this, add dom0_mem=512M to the Xen kernel image's first module line in grub. For example, you should have a line like: 

vim /boot/grub/menu.lst

title CentOS (2.6.18-164.11.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-164.11.1.el5
        module /vmlinuz-2.6.18-164.11.1.el5xen ro root=/dev/san02/lv01 rhgb quiet dom0_mem=512M
        module /initrd-2.6.18-164.11.1.el5xen.img

You can change the '512M' with the amount of RAM you want to allocate to dom0. Note that if you used the AN!Cluster install DVD or the AN!Cluster kickstart files, this should already be set for you.

REMEMBER!

If you update your kernel, be sure to re-add this argument to the new kernel's argument list.

Change the Default Run-Level

If you don't plan to work on your nodes directly, it makes sense to switch the default run level from 5 to 3. This prevents Gnome from starting at boot, thus freeing up a lot of memory and system resources and reducing the possible attack vectors.

To do this, edit /etc/inittab, change the id:5:initdefault: line to id:3:initdefault: and then switch to run level 3: 

vim /etc/inittab

id:3:initdefault:

init 3

Change when xend starts

Normally, xend starts at priority 98 in /etc/rc.X/. This can cause problems with other packages that expect the network to be stable. This is because xend takes all the networks down when it starts. To prevent these problems, we will move the xend init script to start priority 11. We'll also adapt the stop priority to 89, though this is less critical

First, edit the actual initialization script and change the line '# chkconfig: 2345 98 01' to 'chkconfig: 2345 11 89'. 

vim /etc/init.d/xend

# chkconfig: 2345 11 89

Now, use chkconfig to change the apply the changes: 

chkconfig --del xend
chkconfig --add xend

You should now see the symlink /etc/rc3.d/S11xend and /etc/rc3.d/K89xend.

Initial Cluster Setup

Some things, like cluster-aware LVM, won't work until the cluster is setup. For this reason, we need to setup the cluster infrastructure before going any further.

If you didn't read up on Networking in Xen works, now would be a very good time to do so. A lot of the networking from here on in will seem cryptic otherwise when it's actually fairly straight forward.

Adding New NICs to Xen

By default, xend only manages eth0. We need to add eth2 and, if you wish, eth1. Personally, I like to put all my ethernet devices under Xen's control for future flexibility, but this opens a possible security vector as a bridge is created for the DRBD subnet. Whether you add it or not I will leave to your preferences.

You can see which devices are under Xen's control by running ifconfig and checking to see if there is a pethX corresponding to each ethX device. For example, here is what you would see if only eth0 was under Xen's control: 

ifconfig

eth0      Link encap:Ethernet  HWaddr 90:E6:BA:71:82:D8  
          inet addr:192.168.1.71  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::92e6:baff:fe71:82d8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:121 errors:0 dropped:0 overruns:0 frame:0
          TX packets:97 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:20605 (20.1 KiB)  TX bytes:16270 (15.8 KiB)

eth1      Link encap:Ethernet  HWaddr 00:21:91:19:96:5A  
          inet addr:10.0.0.71  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::221:91ff:fe19:965a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:45 errors:0 dropped:0 overruns:0 frame:0
          TX packets:53 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:9139 (8.9 KiB)  TX bytes:10259 (10.0 KiB)
          Interrupt:16 

eth2      Link encap:Ethernet  HWaddr 00:0E:0C:59:45:78  
          inet addr:10.0.1.71  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20e:cff:fe59:4578/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:45 errors:0 dropped:0 overruns:0 frame:0
          TX packets:62 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:9790 (9.5 KiB)  TX bytes:11102 (10.8 KiB)
          Base address:0xec00 Memory:febe0000-fec00000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:560 (560.0 b)  TX bytes:560 (560.0 b)

peth0     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:126 errors:0 dropped:0 overruns:0 frame:0
          TX packets:110 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:20923 (20.4 KiB)  TX bytes:18352 (17.9 KiB)
          Interrupt:252 Base address:0x6000 

vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:103 errors:0 dropped:0 overruns:0 frame:0
          TX packets:126 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:19306 (18.8 KiB)  TX bytes:20935 (20.4 KiB)

virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:49 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:9640 (9.4 KiB)

xenbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:148 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:24256 (23.6 KiB)  TX bytes:0 (0.0 b)

You'll notice that there is no peth1 or peth2 device, nor their associated virtual devices or bridges.

Create /etc/xen/scripts/an-network-script

This script will be used by Xen to create bridges for all NICs.

Please note two things;

  1. You don't need to use the name 'an-network-script'. I suggest this name mainly to keep in line with the rest of the 'AN!x' naming used here.
  2. If you install convirt, it will create it's own bridge script called convirt-xen-multibridge. Other tools may do something similar.
  3. Adding eth1 is optional, as we know ahead of time that eth1 will not be made available to any virtual machines as it is dedicated to DRBD and totem. I'm adding it here because I like having things consistent; Do whichever makes more sense to you.

First, touch the file and then chmod it to be executable. 

touch /etc/xen/scripts/an-network-script
chmod 755 /etc/xen/scripts/an-network-script

Now edit it to contain the following: 

vim /etc/xen/scripts/an-network-script

#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=xenbr0
"$dir/network-bridge" "$@" vifnum=1 netdev=eth1 bridge=xenbr1
"$dir/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=xenbr2

Now tell Xen to reference that script by editing /etc/xen/xend-config.sxp the network-script argument to point to this new script (this is line 91 in the default xend-config.sxp script): 

vim /etc/xen/xend-config.sxp

#(network-script network-bridge)
(network-script an-network-script)

Now restart xend 

/etc/init.d/xend restart

If everything worked, you should now be able to run ifconfig and see that all the ethX devices have matching pethX, virtual and bridge devices. 

ifconfig

eth0      Link encap:Ethernet  HWaddr 90:E6:BA:71:82:D8  
          inet addr:192.168.1.71  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::92e6:baff:fe71:82d8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:274 errors:0 dropped:0 overruns:0 frame:0
          TX packets:190 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:33479 (32.6 KiB)  TX bytes:33376 (32.5 KiB)

eth1      Link encap:Ethernet  HWaddr 00:21:91:19:96:5A  
          inet addr:10.0.0.71  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::221:91ff:fe19:965a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:33 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:10393 (10.1 KiB)

eth2      Link encap:Ethernet  HWaddr 00:0E:0C:59:45:78  
          inet addr:10.0.1.71  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20e:cff:fe59:4578/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:9964 (9.7 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:560 (560.0 b)  TX bytes:560 (560.0 b)

peth0     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:281 errors:0 dropped:0 overruns:0 frame:0
          TX packets:204 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:33929 (33.1 KiB)  TX bytes:35540 (34.7 KiB)
          Interrupt:252 Base address:0x6000 

peth1     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:45 errors:0 dropped:0 overruns:0 frame:0
          TX packets:86 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:9139 (8.9 KiB)  TX bytes:20652 (20.1 KiB)
          Interrupt:16 

peth2     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:45 errors:0 dropped:0 overruns:0 frame:0
          TX packets:90 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:9790 (9.5 KiB)  TX bytes:21066 (20.5 KiB)
          Base address:0xec00 Memory:febe0000-fec00000 

vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:200 errors:0 dropped:0 overruns:0 frame:0
          TX packets:281 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:37668 (36.7 KiB)  TX bytes:33941 (33.1 KiB)

vif0.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:33 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:10393 (10.1 KiB)  TX bytes:0 (0.0 b)

vif0.2    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:28 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:9964 (9.7 KiB)  TX bytes:0 (0.0 b)

virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:49 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:9640 (9.4 KiB)

xenbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:151 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:24426 (23.8 KiB)  TX bytes:0 (0.0 b)

xenbr1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:33 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:9931 (9.6 KiB)  TX bytes:0 (0.0 b)

xenbr2    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:28 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:9572 (9.3 KiB)  TX bytes:0 (0.0 b)

Fencing

Before proceeding with the cluster.conf file, you must understand what fencing is, how it is used in Red Hat/CentOS clusters and why it is so important.

  • The Cluster Admin's Mantra:
    • The only thing you don't know is what you don't know.

Just because one node loses communication with another node, it cannot be assume that the silent node is dead!

What is it?

"Fencing" is the act of isolating a malfunctioning node. The goal is to prevent a split-brain condition where two nodes think the other member is dead and continue to use a shared resource. When this happens, file system corruption is almost guaranteed. Another dangerous scenario would be if one node paused while writing to a disk, the other node decides it's dead and starts to replay the journal, then the first node recovers and completes the write. The results would be equally disastrous. If you are lucky enough to not lose the shared file system, you will be faced with the task of determining what data got written to which node, merging that data and/or overwriting the node you trust the least. This 'best case' is still pretty lousy.

Fencing, isolating a node from altering shared disks, can be accomplished in a couple of ways:

  • Power
    • Power fencing is where a device is used to cut the power to a malfunctioning node. This is probably the most common type.
  • Blocking
    • Blocking is often implemented at the network level. This type of fencing leaves the node alone, but disconnects it from the storage network. Often this is done by a switch which prevents traffic coming from the fenced node.

With power fencing, the term used is "STONITH", literally, Shoot The Other Node In The Head. Picture it like an old west dual. If one node is dead, the other node is going to win the dual by default and the dead node will just be shot again. When both nodes are alive, however, the faster node will win and will "kill" (power off or reset) the slower node before it has a chance to fire. Once this dual is over, the surviving node can then access the shared resource confident that it is the only one working on it.

Misconception

It is a very common mistake to ignore fencing when first starting to learn about clustering. Often people think "It's just for production systems, I don't need to worry about it yet because I don't care what happens to my test cluster.".

Wrong!

For the most practical reason; the cluster software will block all I/O transactions when it can't guarantee a fence operation succeeded. The result is that your cluster will essentially "lock up". Likewise, cman and related daemons will fail if they can't find a fence agent to use.

Secondly; Testing our cluster will involve inducing errors. Without proper fencing, there is a high probability that our shared file system will be corrupted. That would force the need to start over, making your learning take a lot longer than it needs to.

Implementation

In Red Hat's cluster software, the fence device (or devices) are configured in the main /etc/cluster.conf cluster configuration file. This configuration is then acted on via the fenced daemon. When the cluster determines that a node needs to be fenced, the fenced daemon will consult the cluster.conf file for information on how to access the fence device.

Given this cluster.conf snippet: 

<cluster name="an_san" config_version="1">
	<clusternodes>
		<clusternode name="an_san02.alteeve.com" nodeid="2">
			<fence>
				<method name="node_assassin">
					<device name="ariel" port="02" action="off"/>
				</method>
			</fence>
		</clusternode>
	</clusternodes>
	<fencedevices>
		<fencedevice name="ariel" agent="fence_na" quiet="true" ipaddr="ariel.alteeve.com" login="ariel" passwd="gr0tt0">
		</fencedevice>
	</fencedevices>
</cluster>

If the cluster manager determines that the node an_san02.alteeve.com needs to be fenced, it looks at the first (and only, in this case) <fence> entry's name, which is ariel in this case. It then looks in the <fencedevices> section for the device with the matching name. From there, it gets the information needed to find and access the fence device. Once it connects to the fence device, it then passes the options set in an_san02.alteeve.com's <fence> argument.

So in this example, fenced looks up the details on the ariel Node Assassin fence device. It calls the fence_na program, called a fence agent, and passes the following arguments:

  • ipaddr=ariel.alteeve.com
  • login=ariel
  • passwd=gr0tt0
  • quiet=true/span>
  • port=2
  • action=off

How the fence agent acts on these arguments varies depending on the fence device itself. In general terms, the 'fence_na' fence agent will create a connection to the device at the IP address (or resolvable name, as in this case) specified in the ipaddr argument. Once connected, it will authenticate using the login and passwd arguments. Once authenticated, it tells the device what port to act on, which could be a power jack, a power or reset button, a network switch port and so on. Finally, it tells the device what action to take.

Once the device completes, it returns a success or failed message. If the first attempt fails, the fence agent will try the next <fence> method, if a second exists. It will keep trying fence devices in the order they are found in the cluster.conf file until it runs out of devices. If it fails to fence the node, most daemons will "block", that is, lock up and stop responding until the issue is resolved. The logic for this is that a locked up cluster is better than a corrupted one.

If any of the fence devices succeed though, the cluster will know that it is safe to proceed and will reconfigure the cluster without the defective node.

Fence Devices

Many major OEMs have their own remote management devices that can serve as fence devices. Examples are Dell's 'DRAC' (Dell Remote Access Controller), HP's iLO (Integrate Lights Out), IBM's 'RSA' (Remote Supervisor Adapter), Sun's 'SSP' (System Service Processor) and so on. Smaller manufacturers implement remote management via IPMI, Intelligent Power Management Interface.

In the above devices, fencing is implemented via a build in or integrated device inside the server. These devices are usually accessible even when the host server is powered off or hard locked. Via these devices, the host server can be powered off, reset and powered on remotely, regardless of the state of the host server.

Block fencing is possible when the device connecting a node to shared resources, like a fiber-channel SAN switch, provides a method of logically "unplugging" a defective node from the shared resource, leaving the node itself alone.

Node Assassin

A cheap alternative is the Node Assassin, an open-hardware, open source fence device. It was built to allow the use of commodity system boards that lacked remote management support found on more expensive, server class hardware.

Full Disclosure: Node Assassin was created by me, with much help from others, for this paper.

cluster.conf

The core of the cluster is the /etc/cluster/cluster.conf XML configuration file. It contains information about the cluster itself, what nodes are to be used, how to fence each node, what fence devices exist plus miscellaneous other configuration options.

By default, there is no cluster.conf, so you need to start by creating it: 

touch /etc/cluster/cluster.conf

Here is the one AN!Cluster uses, with in-line comments, mostly from the man cluster.conf page.

Once you're comfortable with your changes to the file, you need to validate it. Run: 

xmllint --relaxng /usr/share/system-config-cluster/misc/cluster.ng /etc/cluster/cluster.conf

If there are errors, address them. Once you see /etc/cluster/cluster.conf validates, you can proceed to the next step.

Note: If you are using Node Assassin and the XML validation fails, be sure to get the updated cluster.ng validation file!

openais.conf

Where cluster.conf is the core configuration file for the cluster, OpenAIS is the master of ceremonies. It provides all the cluster functions using first it's own /etc/ais/openais.conf then the /etc/cluster/cluster.conf file for arguments. You can think of this file as a "low level" configuration file controlling the underlying mechanics of the cluster where cluster.conf contains the specific cluster configuration.

Unlike cluster.conf, there is a default openais.conf config file. It's a good habit to back default files up in case you need to start over.

When reviewing the openais.conf file below, please take the time to read the comments in the file. There are many aspects of clustering that will make sense if you understand the various OpenAIS configuration options.

Once you are comfortable, backup and edit openais.conf: 

cp /etc/ais/openais.conf /root/backups/
vim /etc/ais/openais.conf

Cluster First Start

If everything up until now was done right, you should be able to start your cluster for the first time. It can be useful to have a separate terminal window open with a tail watching /var/log/messages so that you can see if there are any problems.

On both nodes, in dedicated terminals, run: 

clear; tail -f -n 0 /var/log/messages

This next step must be run on both nodes as soon as possible. If you try to start one node and wait too long to start the other node, the first node will think there is a problem and it will fence the second node. Remember the <fence_daemon post_join_delay="60"></fence_daemon> line in cluster.conf? This is where it comes into play. The value you set it the "window" you have to start both nodes before a fence is issued. The default is 6 seconds, and the above line changed that to 60 seconds.

On both nodes, in different terminals, check that cman is indeed stopped, then start it up: 

/etc/init.d/cman status

ccsd is stopped

/etc/init.d/cman start

If all goes well, you should see something like this in each node's /var/log/messages file: 

Mar 14 13:17:29 an_san01 ccsd[6631]: Starting ccsd 2.0.115: 
Mar 14 13:17:29 an_san01 ccsd[6631]:  Built: Dec  8 2009 09:20:54 
Mar 14 13:17:29 an_san01 ccsd[6631]:  Copyright (C) Red Hat, Inc.  2004  All rights reserved. 
Mar 14 13:17:29 an_san01 ccsd[6631]: cluster.conf (cluster name = an_san, version = 1) found. 
Mar 14 13:17:32 an_san01 openais[6637]: [MAIN ] AIS Executive Service RELEASE 'subrev 1887 version 0.80.6' 
Mar 14 13:17:32 an_san01 openais[6637]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. 
Mar 14 13:17:32 an_san01 openais[6637]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
Mar 14 13:17:32 an_san01 openais[6637]: [MAIN ] AIS Executive Service: started and ready to provide service. 
Mar 14 13:17:32 an_san01 openais[6637]: [MAIN ] Using default multicast address of 239.192.24.13 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] send threads (0 threads) 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] RRP token expired timeout (495 ms) 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] RRP token problem counter (2000 ms) 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] RRP threshold (10 problem count) 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] RRP mode set to none. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] heartbeat_failures_allowed (0) 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] max_network_delay (50 ms) 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] The network interface [10.0.1.71] is now up. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Created or loaded sequence id 4.10.0.1.71 for this ring. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] entering GATHER state from 15. 
Mar 14 13:17:32 an_san01 openais[6637]: [CMAN ] CMAN 2.0.115 (built Dec  8 2009 09:20:58) started 
Mar 14 13:17:32 an_san01 openais[6637]: [MAIN ] Service initialized 'openais CMAN membership service 2.01' 
Mar 14 13:17:32 an_san01 openais[6637]: [SERV ] Service initialized 'openais extended virtual synchrony service' 
Mar 14 13:17:32 an_san01 openais[6637]: [SERV ] Service initialized 'openais cluster membership service B.01.01' 
Mar 14 13:17:32 an_san01 openais[6637]: [SERV ] Service initialized 'openais availability management framework B.01.01' 
Mar 14 13:17:32 an_san01 openais[6637]: [SERV ] Service initialized 'openais checkpoint service B.01.01' 
Mar 14 13:17:32 an_san01 openais[6637]: [SERV ] Service initialized 'openais event service B.01.01' 
Mar 14 13:17:32 an_san01 openais[6637]: [SERV ] Service initialized 'openais distributed locking service B.01.01' 
Mar 14 13:17:32 an_san01 openais[6637]: [SERV ] Service initialized 'openais message service B.01.01' 
Mar 14 13:17:32 an_san01 openais[6637]: [SERV ] Service initialized 'openais configuration service' 
Mar 14 13:17:32 an_san01 openais[6637]: [SERV ] Service initialized 'openais cluster closed process group service v1.01' 
Mar 14 13:17:32 an_san01 openais[6637]: [SERV ] Service initialized 'openais cluster config database access v1.01' 
Mar 14 13:17:32 an_san01 openais[6637]: [SYNC ] Not using a virtual synchrony filter. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Creating commit token because I am the rep. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Saving state aru 0 high seq received 0 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Storing new sequence id for ring 8 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] entering COMMIT state. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] entering RECOVERY state. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] position [0] member 10.0.1.71: 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] previous ring seq 4 rep 10.0.1.71 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] aru 0 high delivered 0 received flag 1 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Did not need to originate any messages in recovery. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Sending initial ORF token 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] CLM CONFIGURATION CHANGE 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] New Configuration: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] Members Left: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] Members Joined: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] CLM CONFIGURATION CHANGE 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] New Configuration: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] 	r(0) ip(10.0.1.71)  
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] Members Left: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] Members Joined: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] 	r(0) ip(10.0.1.71)  
Mar 14 13:17:32 an_san01 openais[6637]: [SYNC ] This node is within the primary component and will provide service. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] entering OPERATIONAL state. 
Mar 14 13:17:32 an_san01 openais[6637]: [CMAN ] quorum regained, resuming activity 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] got nodejoin message 10.0.1.71 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] entering GATHER state from 11. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Creating commit token because I am the rep. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Saving state aru a high seq received a 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Storing new sequence id for ring c 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] entering COMMIT state. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] entering RECOVERY state. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] position [0] member 10.0.1.71: 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] previous ring seq 8 rep 10.0.1.71 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] aru a high delivered a received flag 1 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] position [1] member 10.0.1.72: 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] previous ring seq 4 rep 10.0.1.72 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] aru a high delivered a received flag 1 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Did not need to originate any messages in recovery. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] Sending initial ORF token 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] CLM CONFIGURATION CHANGE 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] New Configuration: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] 	r(0) ip(10.0.1.71)  
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] Members Left: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] Members Joined: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] CLM CONFIGURATION CHANGE 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] New Configuration: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] 	r(0) ip(10.0.1.71)  
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] 	r(0) ip(10.0.1.72)  
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] Members Left: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] Members Joined: 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] 	r(0) ip(10.0.1.72)  
Mar 14 13:17:32 an_san01 openais[6637]: [SYNC ] This node is within the primary component and will provide service. 
Mar 14 13:17:32 an_san01 openais[6637]: [TOTEM] entering OPERATIONAL state. 
Mar 14 13:17:32 an_san01 ccsd[6631]: Initial status:: Quorate 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] got nodejoin message 10.0.1.71 
Mar 14 13:17:32 an_san01 openais[6637]: [CLM  ] got nodejoin message 10.0.1.72

If you do, then all systems are go! Congratulations, you now have an honest to goodness cluster up and running! Time to start using it.

DRBD

DRBD will be used to provide a real-time, redundant block device. On top of this, a new LVM PV will be created for a virtual machine that will be able to "float" between the two nodes. This way, should one of the nodes fail, the virtual machine would be able to quickly be brought back up on the surviving node with minimal interruption. When you have planned down time, you will be able to "hot migrate" the virtual machine from one node to the other with nothing more that a short pause while the virtual machine's RAM is frozen and copied over to the other node, a process that usually takes a few seconds to a minute.

Install

The drbd83 and kmod-drbd83-xen packages are not included in the default CentOS installation media, so we will need to install them now: 

yum -y install drbd83.x86_64 kmod-drbd83-xen.x86_64

Before we configure DRBD, we will need to create an LVM LV to host it.

Create the LVM Logical Volume

Most of the remaining space on either node's LVM PV will be allocated to a new LV. This new LV will host either node's side of the DRBD resource.

First, you need to see how much space you have left on you LVM PV: 

pvscan

  PV /dev/sda2   VG san01   lvm2 [465.50 GB / 443.97 GB free]
  Total: 1 [465.50 GB] / in use: 1 [465.50 GB] / in no VG: 0 [0   ]

On my nodes, each of which has a single 500GB drive, I've allocated only 20GB to dom0 so I've got over 440GB left free. I like to leave a bit of space unallocated because I never know where I might need it, so I will allocate 400GB even to DRBD and keep the remaining 44GB set aside for future growth. The space you have left and how you want to allocate is an exercise you must settle based on your own needs.

Next, check that the name you will give to the new LV isn't used yet: 

lvscan

  ACTIVE            '/dev/san01/lv01' [19.53 GB] inherit
  ACTIVE            '/dev/san01/lv00' [2.00 GB] inherit

I can see from the above output that lv00 and lv01 are used, so I will use lv02 for my DRBD partition. Of course, you can use drbd or pretty much anything else you want.

Now that I know I want to create a 400GB logical volume called lv02, I can proceed.

Create the Logical Volume for the DRBD device on each node. The next two commands show what I need to call on my nodes, and will match what you need to run if you used the AN!Cluster Install DVD. If you ran your own install, be sure to edit the following arguments to match your nodes:

On an_san01: 

lvcreate -L 400G -n lv02 /dev/san01

On an_san02: 

lvcreate -L 400G -n lv02 /dev/san02

  Logical volume "lv02" created

If I re-run lvscan now, I will see the new volume: 

lvscan

  ACTIVE            '/dev/san02/lv01' [19.53 GB] inherit
  ACTIVE            '/dev/san02/lv00' [2.00 GB] inherit
  ACTIVE            '/dev/san02/lv02' [400.00 GB] inherit

We can now proceed with the DRBD setup!

Create or Edit /etc/drbd.conf

DRBD is controlled from a single /etc/drbd.conf configuration file that must be identical on both nodes. This file tells DRBD what devices to use on each node, what interface to use and so on.

Full details on all the drbd.conf configuration file directives and arguments can be found here. 

global {
        usage-count yes;
}
common {
        protocol C;
        syncer { rate 33M; }
}
resource r0 {
        device    /dev/drbd0;
        meta-disk internal;
        net {
                allow-two-primaries;
        }
        startup { 
                become-primary-on both;
        }
	# The 'on' name must be the same as the output of 'uname -n'.
        on an_san01.alteeve.com {
                address   10.0.0.71:7789;
		disk      /dev/san01/lv02;
        }
        on an_san02.alteeve.com {
                address   10.0.0.72:7789;
        	disk      /dev/san02/lv02;
        }
}

The main things to note are:

  • The one argument must match the name returned by the 'uname -n' shell call.
  • 'Protocol C' tells DRBD to not tell the OS that a write was complete until both nodes have done so. This effects performance but is required for the later step when we will configure cluster-aware LVM.

With that file in place on both nodes, run the following command and make sure the output is the contents of the file above in a somewhat altered syntax. If you get an error, address it before proceeding. 

drbdadm dump

If it's all good, you should see something like this: 

# /etc/drbd.conf
common {
    protocol               C;
    syncer {
        rate             33M;
    }
}

# resource r0 on an_san02.alteeve.com: not ignored, not stacked
resource r0 {
    on an_san01.alteeve.com {
        device           /dev/drbd0 minor 0;
        disk             /dev/san01/lv02;
        address          ipv4 10.0.0.71:7789;
        meta-disk        internal;
    }
    on an_san02.alteeve.com {
        device           /dev/drbd0 minor 0;
        disk             /dev/san02/lv02;
        address          ipv4 10.0.0.72:7789;
        meta-disk        internal;
    }
    net {
        allow-two-primaries;
    }
    startup {
        become-primary-on both;
    }
}

Once you see this, you can proceed.

Setup the DRBD Resource r0

From the rest of this section, pay attention to whether you see

  • Primary
  • Secondary
  • Both

These indicate which node to run the following commands on. There is no functional difference between either node, so just randomly choose one to be Primary and the other will be Secondary. Once you've chosen which is which, be consistent with which node you run the commands on. Of course, if a command block is proceeded by Both, run the following code block on both nodes.

Both 

/etc/init.d/drbd restart

You should see output like this: 

  --==  Thank you for participating in the global usage survey  ==--
The server's response is:

you are the 2278th user to install this version
Restarting all DRBD resources: 0: Failure: (119) No valid meta-data signature found.

	==> Use 'drbdadm create-md res' to initialize meta-data area. <==

Command '/sbin/drbdsetup 0 disk /dev/san01/lv02 /dev/san02/lv02 internal --set-defaults --create-device' terminated with exit code 10

Don't worry about those errors.

Primary: Initiate the device by run the following commands one at a time: 

drbdadm create-md r0 
drbdadm attach r0 
drbdadm syncer r0 
drbdadm connect r0

Secondary: Configure and connect the device on the second node by run ningthe following commands one at a time: 

drbdadm create-md r0
drbdadm attach r0
drbdadm connect r0

Primary: Start the sync between the two nodes by calling: 

drbdadm -- --overwrite-data-of-peer primary r0

Secondary: At this point, we need to promote the secondary node to 'Primary' position. 

drbdadm primary r0

Both: Make sure that both nodes are sync'ing by watching the sync process by running: 

watch cat /proc/drbd

Both: You should see something like: 

Every 2.0s: cat /proc/drbd                                                                Tue Feb  2 23:53:29 2010

version: 8.3.2 (api:88/proto:86-90)
GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by mockbuild@v20z-x86-64.home.local, 2009-08-29 14:08:07
 0: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r----
    ns:366271552 nr:0 dw:0 dr:366271552 al:0 bm:22355 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:53146012
        [================>...] sync'ed: 87.4% (51900/409584)M
        finish: 0:22:27 speed: 39,364 (33,888) K/sec

You do not need to wait for the sync. to complete before proceeding.

LVM

LVM plays a roll at three levels of the AN!Cluster, and now is a good time to discuss why and how.

  • First; Hosts the dom0's OS.
    • Why? By using LVM over the bare block devices (or raid devices), we have an underlying disk system that can be expanded easily. Because the DRBD sits on top of this base level, we can grow the DRBD partitions later. This level was implemented by the kickstart script found at the beginning of this How-To.
  • Second; The base DRBD partition becomes a PV for the iSCSI VM to use. This way, once the DRBD is grown, or if a second DRBD devices is added, it can be added to the LVM's VG.
  • Third; Inside the Virtual Machines. This allows the virtual machines to grow in disk space as well.

By having all three levels using LVM, the cluster and all resources can be grown "on the fly" without ever taking the cluster offline.

Creating an LVM on the DRBD Partition

First, install the LVM programs if you haven't already. It's quite likely these packages are already installed because of the LVM partition under DRBD. However, let's double check: 

yum install -y lvm2 lvm2-cluster

From here on, the same Primary, Secondary, Both and Either prefixes will be shown before shell commands as was introduced above. Please be sure to continue using the same convention you selected earlier. That is, pick one node to be "Primary" and stick to it.

If you check now using pvdisplay, you should see only one LVM PV:

Either: 

pvdisplay

  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               san01
  PV Size               465.51 GB / not usable 14.52 MB
  Allocatable           yes 
  PE Size (KByte)       32768
  Total PE              14896
  Free PE               1407
  Allocated PE          13489
  PV UUID               PfKywt-t7or-ZfqD-f104-dDHR-8ILi-6ZswMq

Lets now create an LVM PV using the /dev/drbd0.

Primary: 

pvcreate /dev/drbd0

  Physical volume "/dev/drbd0" successfully created

If we check pvdisplay again, we should now see both PVs.

Both: 

pvdisplay

  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               san01
  PV Size               465.51 GB / not usable 14.52 MB
  Allocatable           yes 
  PE Size (KByte)       32768
  Total PE              14896
  Free PE               1407
  Allocated PE          13489
  PV UUID               PfKywt-t7or-ZfqD-f104-dDHR-8ILi-6ZswMq
   
  "/dev/san01/lv02" is a new physical volume of "399.99 GB"
  --- NEW Physical volume ---
  PV Name               /dev/san01/lv02
  VG Name               
  PV Size               399.99 GB
  Allocatable           NO
  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               alpyBY-ukAR-9KEz-CQt9-u86N-8fg8-W5DPvV

Run pvcreate from both nodes after creating the new PV on /dev/drbd0. Even though you only added the DRBD device on one node, you should see the PV on both. That's the first sign that your DRBD is working well!

Making the DRBD LVM Cluster Aware

Because this LVM exists on a Primary/Primary DRBD partition where both nodes can write at the same time, the LVM layer MUST be configured to be cluster-aware or else you WILL corrupt your data eventually.

To switch LVM to cluster aware mode, edit /etc/lvm/lvm.conf on both nodes:

Both: 

vim /etc/lvm/lvm.conf

Search for filter and locate the lines: 

    # By default we accept every block device:
    filter = [ "a/.*/" ]

Copy the filter = [ "a/.*/" ] line, comment out the first one and change the second one to [ "a|drbd.*|", "r|.*|" ]. The entry should now read: 

    # By default we accept every block device:
    #filter = [ "a/.*/" ]
    filter = [ "a|drbd.*|", "r|.*|" ]

In a similar manner, change the locking_type value from 1 to 3. The edited section should look like: 

    # Type of locking to use. Defaults to local file-based locking (1).
    # Turn locking off by setting to 0 (dangerous: risks metadata corruption
    # if LVM2 commands get run concurrently).
    # Type 2 uses the external shared library locking_library.
    # Type 3 uses built-in clustered locking.
    #locking_type = 1
    locking_type = 3

Now, tell LVM to become cluster aware. On Both nodes, run: 

lvmconf --enable-cluster

There will be no output if it worked, it will just return to the shell.

That's it for now.

Thanks

  • To HJ Lee from the OpenAIS list for helping me understand the mechanisms controlling the Redundant Ring Protocol's failure detection types.
  • To Steven Dake for clarifying the to_x vs. logoutput: x arguments in openais.conf.

 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche! Enterprise Support:
Alteeve Support 		Community Support
© Alteeve's Niche! Inc. 1997-2024   Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.
Retrieved from ‘https://alteeve.com/w/index.php?title=Abandoned_-_2-Node_EL5_Cluster&oldid=1073’