2-Node Red Hat KVM Cluster Tutorial - Archive

From Alteeve Wiki
Jump to navigation Jump to search

 AN!Wiki :: How To :: 2-Node Red Hat KVM Cluster Tutorial - Archive

This paper has one goal;

  • Creating a 2-node, high-availability cluster hosting KVM virtual machines using RHCS "stable 3" with DRBD and clustered LVM for synchronizing storage data. This is an updated version of the earlier Red Hat Cluster Service 2 Tutorial Tutorial. You will find much in common with that tutorial if you've previously followed that document. Please don't skip large sections though. There are some differences that are subtle but important.

Grab a coffee, a comfy chair, put on some nice music and settle in for some geekly fun.

The Task Ahead

Before we start, let's take a few minutes to discuss clustering and it's complexities.

Technologies We Will Use

  • Enterprise Linux 6; You can use a derivative like CentOS v6.
  • Red Hat Cluster Services "Stable" version 3. This describes the following core components:
    • Corosync; Provides cluster communications using the totem protocol.
    • Cluster Manager (cman); Manages the starting, stopping and managing of the cluster.
    • Resource Manager (rgmanager); Manages cluster resources and services. Handles service recovery during failures.
    • Clustered Logical Volume Manager (clvm); Cluster-aware (disk) volume manager. Backs GFS2 filesystems and KVM virtual machines.
    • Global File Systems version 2 (gfs2); Cluster-aware, concurrently mountable file system.
  • Distributed Redundant Block Device (DRBD); Keeps shared data synchronized across cluster nodes.
  • KVM; Hypervisor that controls and supports virtual machines.

A Note on Patience

There is nothing inherently hard about clustering. However, there are many components that you need to understand before you can begin. The result is that clustering has an inherently steep learning curve.

You must have patience. Lots of it.

Many technologies can be learned by creating a very simple base and then building on it. The classic "Hello, World!" script created when first learning a programming language is an example of this. Unfortunately, there is no real analogue to this in clustering. Even the most basic cluster requires several pieces be in place and working together. If you try to rush by ignoring pieces you think are not important, you will almost certainly waste time. A good example is setting aside fencing, thinking that your test cluster's data isn't important. The cluster software has no concept of "test". It treats everything as critical all the time and will shut down if anything goes wrong.

Take your time, work through these steps, and you will have the foundation cluster sooner than you realize. Clustering is fun because it is a challenge.

Network

The cluster will use three Class B networks, broken down as:

Purpose Subnet Notes
Internet-Facing Network (IFN) 10.255.0.0/16
  • Each node will use 10.255.0.x where x matches the node ID.
  • Virtual Machines in the cluster that need to be connected to the Internet will use 10.255.y.z where y corresponds to the cluster and z is a simple sequence number matching the VM ID.
Storage Network (SN) 10.10.0.0/16
  • Each node will use 10.10.0.x where x matches the node ID.
Back-Channel Network (BCN) 10.20.0.0/16
  • Each node will use 10.20.0.x where x matches the node ID.
  • Node-specific IPMI or other out-of-band management devices will use 10.20.1.x where x matches the node ID.
  • Multi-port fence devices will use 10.20.2.z where z is a simple sequence.

Miscellaneous equipment in the cluster, like managed switches, will use 10.20.3.z where z is a simple sequence.

We will be using six interfaces, bonded into three pairs of two NICs in Active/Passive (mode 1). Each link of each bond will be on alternate, unstacked switches. This configuration is the only configuration supported by Red Hat in clusters.

If you can not install six interfaces in your server, then four interfaces will do with the SN and BCN networks merged.

In this tutorial, we will use two D-Link DGS-3100-24, unstacked, using three VLANs to isolate the three networks.

You could just as easily use four or six unmanaged 5 port or 8 port switches. What matters is that the three subnets are isolated and that each link of each bond is on a separate switch. Lastly, only connect the IFN switches or VLANs to the Internet for security reasons.

The actual mapping of interfaces to bonds to networks will be:

Subnet Link 1 Link 2 Bond IP
BCN eth0 eth3 bond0 10.20.0.x
SN eth1 eth4 bond1 10.10.0.x
IFN eth2 eth5 bond2 10.255.0.x

Setting Up the Network

 Warning: The following steps can easily get confusing, given how many files we need to edit. Losing access to your server's network is a very real possibility! Do not continue without direct access to your servers! If you have out-of-band access via iKVM, console redirection or similar, be sure to test that it is working before proceeding.

Making Sure We Know Our Interfaces

When you installed the operating system, the network interfaces names are somewhat randomly assigned to the physical network interfaces. It more than likely that you will want to re-order.

Before you start moving interface names around, you will want to consider which physical interfaces you will want to use on which networks. At the end of the day, the names themselves have no meaning. At the very least though, make them consistent across nodes.

Some things to consider, in order of importance:

  • If you have a shared interface for your out-of-band management interface, like IPMI or iLO, you will want that interface to be on the Back-Channel Network.
  • For redundancy, you want to spread out which interfaces are paired up. In my case, I have three interfaces on my mainboard and three additional add-in cards. I will pair each onboard interface with an add-in interface. In my case, my IPMI interface physically piggy-backs on one of the onboard interfaces so this interface will need to be part of the BCN bond.
  • Your interfaces with the lowest latency should be used for the back-channel network.
  • Your two fastest interfaces should be used for your storage network.
  • The remaining two slowest interfaces should be used for the Internet-Facing Network bond.

In my case, all six interfaces are identical, so there is little to consider. The left-most interface on my system has IPMI, so it's paired network interface will be eth0. I simply work my way left, incrementing as I go. What you do will be whatever makes most sense to you.

There is a separate, short tutorial on re-ordering network interface;

Once you have the physical interfaces named the way you like, proceed to the next step.

Planning Our Network

To setup our network, we will need to edit the ifcfg-ethX, ifcfg-bondX and ifcfg-vbrX scripts. The last one will create bridges which will be used to route network connections to the virtual machines. We won't be creating an vbr1 bridge though, and bond1 will be dedicated to the storage and never used by a VM. The bridges will have the IP addresses, not the bonded interfaces. They will instead be slaved to their respective bridges.

We're going to be editing a lot of files. It's best to lay out what we'll be doing in a chart. So our setup will be:

Node BCN IP and Device SN IP and Device IFN IP and Device
an-node01 10.20.0.1 on vbr0 (bond0 slaved) 10.10.0.1 on bond1 10.255.0.1 on vbr2 (bond2 slaved)
an-node02 10.20.0.2 on vbr0 (bond0 slaved) 10.10.0.2 on bond1 10.255.0.2 on vbr2 (bond2 slaved)

Creating Some Network Configuration Files

Bridge configuration files must have a file name that sorts after the interfaces and bridges. The actual device name can be whatever you want though. If the system tries to start the bridge before it's interface is up, it will fail. I personally like to use the name vbrX for "virtual machine bridge". You can use whatever makes sense to you, with the above concern in mind.

Start by touching the configuration files we will need. 

touch /etc/sysconfig/network-scripts/ifcfg-bond{0,1,2}
touch /etc/sysconfig/network-scripts/ifcfg-vbr{0,2}

Now make a backup of your configuration files, in case something goes wrong and you want to start over. 

mkdir /root/backups/
rsync -av /etc/sysconfig/network-scripts/ifcfg-eth* /root/backups/

sending incremental file list
ifcfg-eth0
ifcfg-eth1
ifcfg-eth2
ifcfg-eth3
ifcfg-eth4
ifcfg-eth5

sent 1467 bytes  received 126 bytes  3186.00 bytes/sec
total size is 1119  speedup is 0.70

Configuring Our Bridges

Now lets start in reverse order. We'll write the bridge configuration, then the bond interfaces and finally alter the interface configuration files. The reason for doing this in reverse is to minimize the amount of time where a sudden restart would leave us without network access.

an-node01 BCN Bridge: 

vim /etc/sysconfig/network-scripts/ifcfg-vbr0

# Back-Channel Network - Bridge
DEVICE="vbr0"
TYPE="Bridge"
BOOTPROTO="static"
IPADDR="10.20.0.1"
NETMASK="255.255.0.0"

an-node01 IFN Bridge: 

vim /etc/sysconfig/network-scripts/ifcfg-vbr2

# Internet-Facing Network - Bridge
DEVICE="vbr2"
TYPE="Bridge"
BOOTPROTO="static"
IPADDR="10.255.0.1"
NETMASK="255.255.0.0"
GATEWAY="10.255.255.254"
DNS1="192.139.81.117"
DNS2="192.139.81.1"
DEFROUTE="yes"

Creating the Bonded Interfaces

Now we can create the actual bond configuration files.

To explain the BONDING_OPTS options;

  • mode=1 sets the bonding mode to active-backup.
  • The miimon=100 tells the bonding driver to check if the network cable has been unplugged or plugged in every 100 milliseconds.
  • The use_carrier=1 tells the driver to use the driver to maintain the link state. Some drivers don't support that. If you run into trouble, try changing this to 0.
  • The updelay=0 and downdelay=0 tells the driver not to wait before changing the state of an interface when the link goes up or down. There are some cases where the link will report that it is ready before the switch can push data. Other times, you might have the switch drop the link for a very brief time. Altering these values gives you a chance to either delay the use of an interface until the switch is ready or prevent dropping a link when the link is just bouncing.

an-node01 BCN Bond: 

vim /etc/sysconfig/network-scripts/ifcfg-bond0

# Back-Channel Network - Bond
DEVICE="bond0"
BRIDGE="vbr0"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=0 downdelay=0"

an-node01 SN Bond: 

vim /etc/sysconfig/network-scripts/ifcfg-bond1

# Storage Network - Bond
DEVICE="bond1"
BOOTPROTO="static"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=0 downdelay=0"
IPADDR="10.10.0.1"
NETMASK="255.255.0.0"

an-node01 IFN Bond: 

vim /etc/sysconfig/network-scripts/ifcfg-bond2

# Internet-Facing Network - Bond
DEVICE="bond2"
BRIDGE="vbr2"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=0 downdelay=0"

Alter The Interface Configurations

Now, finally, alter the interfaces themselves to join their respective bonds.

an-node01's eth0, the BCN bond0, Link 1: 

vim /etc/sysconfig/network-scripts/ifcfg-eth0

# Back-Channel Network - Link 1
HWADDR="00:E0:81:C7:EC:49"
DEVICE="eth0"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond0"
SLAVE="yes"

an-node01's eth1, the SN bond1, Link 1: 

vim /etc/sysconfig/network-scripts/ifcfg-eth0

# Storage Network - Link 1
HWADDR="00:E0:81:C7:EC:48"
DEVICE="eth1"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond1"
SLAVE="yes"

an-node01's eth2, the IFN bond2, Link 1: 

vim /etc/sysconfig/network-scripts/ifcfg-eth0

# Internet-Facing Network - Link 1
HWADDR="00:E0:81:C7:EC:47"
DEVICE="eth2"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond2"
SLAVE="yes"

an-node01's eth3, the BCN bond0, Link 2: 

vim /etc/sysconfig/network-scripts/ifcfg-eth3

# Back-Channel Network - Link 2
DEVICE="eth3"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond0"
SLAVE="yes"

an-node01's eth4, the SN bond1, Link 2: 

vim /etc/sysconfig/network-scripts/ifcfg-eth0

# Storage Network - Link 2
HWADDR="00:1B:21:BF:6F:FE"
DEVICE="eth4"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond1"
SLAVE="yes"

an-node01's eth5, the IFN bond2, Link 2: 

vim /etc/sysconfig/network-scripts/ifcfg-eth0

# Internet-Facing Network - Link 2
HWADDR="00:1B:21:BF:70:02"
DEVICE="eth5"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
MASTER="bond2"
SLAVE="yes"



Thanks

 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche! Enterprise Support:
Alteeve Support 		Community Support
© Alteeve's Niche! Inc. 1997-2024   Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.
Retrieved from ‘https://alteeve.com/w/index.php?title=2-Node_Red_Hat_KVM_Cluster_Tutorial_-_Archive&oldid=3657’