Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

From Alteeve Wiki
Revision as of 00:16, 9 September 2010 by Digimer (talk | contribs) (Created page with '{{howto_header}} '''Warning''': This is currently a dumping ground for notes. '''''DO NOT FOLLOW THIS DOCUMENT'S INSTRUCTIONS'''''. Seriously, it could blow up your computer or …')
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

 AN!Wiki :: How To :: Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

Warning: This is currently a dumping ground for notes. DO NOT FOLLOW THIS DOCUMENT'S INSTRUCTIONS. Seriously, it could blow up your computer or cause winter to come early.

This HowTo will walk you through setting up Xen VMs using DRBD and CLVM for high availability.

Prerequisite =

This talk is an extension of the Two Node Fedora 13 Cluster HowTo. As such, you will be expected to have a freshly built two-node cluster with spare disk space on either node.

Please do not proceed until you have completed the first tutorial.

Overview

This tutorial will cover several topics; DRBD, CLVM, GFS2, Xen dom0 and domU VMs and rgmanager. Their relationship is thus:

  • DRBD provides a mechanism to replicate data across both nodes in real time and guarantees a consistent view of that data from either node. Think of it like RAID level 1, but across machines.
  • CLVM sits on the DRBD partition and provides the underlying mechanism for allowing both nodes to access shared data in a clustered environment. It will host a shared filesystem by way of GFS2 as well as LVs that Xen's domU VMs will use as their disk space.
  • GFS2 will be the clustered file system used on one of the DBRD-backed, CLVM-managed partitions. Files that need to be shared between nodes, like the Xen VM configuration files, will exist on this partition.
  • Xen will be the hypervisor in use that will manage the various virtual machines. Each virtual machine will exist in an LVM LV.
    • Xen's dom0 is the special "host" virtual machine. In this case, dom0 will be the OS installed in the first HowTo.
    • Xen's domU virtual machines will be the "floating", highly available servers.
  • Lastly, rgmanager will be the component of cman that will be configured to manage the automatic migration of the virtual machines when failures occur and when nodes recover.

Setting up Xen's dom0

It may seem odd to start with Xen at this stage, but it is going to rather fundamentally alter each node's "host" operating system.

At this point, each node's host OS is a traditional operating system operating on the bare metal. When we install a dom0 kernel though, we tell Xen to boot a mini operating system first, and then to boot our "host" operating system. In effect, this converts the host node's operating system into just another virtual machine, albeit with a special view of the underlying hardware and Xen hypervisor.

This conversion is somewhat disruptive, so I like to get it out of the way right away. We will then do the rest of the setup before returning to Xen later on to create the floating virtual machines.

A Note On The State Of Xen dom0 Support In Fedora

As of Fedora 8, support for Xen dom0 has been removed. This is temporary, and dom0 is expected to be restored as a supported option in Fedora 15 or 16.

The reason for the removal is that, at this time, much of the code needed to create a dom0 kernel needs to be applied as patches against a vanilla Linux kernel. This is very time-consuming task that will be resolves when many of these patches are moved into the kernel proper later. Once that happens, dom0 support will become native and the overhead will be significantly reduced for the Fedora developers.

What this means for us is that we need to use a non-standard dom0 kernel. Specifically, we will use a kernel created by myoung for Fedora 12. This kernel does not directly support DRBD, so be aware that we will need to build new DRBD kernel modules for his kernel and then rebuild the DRBD modules each time his kernel is updated.

Install The Hypervisor

The Xen hypervisor is the program that manages the virtual servers, provides the virtual hardware, routes access to the real hardware and so on. To install it, simply install the xen RPM package. 

yum install xen.x86_64

Install myoung's dom0

This uses a kernel built for Fedora 12, but it works on Fedora 13. This step involves adding and enabling his repository.

To add the repository, download the myoung.dom0.repo into the /etc/yum.repos.d/ directory. 

cd /etc/yum.repos.d/
wget -c http://myoung.fedorapeople.org/dom0/myoung.dom0.repo

To enable his repository, edit the repository file and change the two enabled=0 entries to enabled=1. 

vim /etc/yum.repos.d/myoung.dom0.repo

[myoung-dom0]
name=myoung's repository of Fedora based dom0 kernels - $basearch
baseurl=http://fedorapeople.org/~myoung/dom0/$basearch/
enabled=1
gpgcheck=0

[myoung-dom0-source]
name=myoung's repository of Fedora based dom0 kernels - Source
baseurl=http://fedorapeople.org/~myoung/dom0/src/
enabled=1
gpgcheck=0

Install the Xen dom0 kernel (edit the version number with the updated version if it has changed). 

yum install kernel-2.6.32.21-167.xendom0.fc12.x86_64

The entry in grub's /boot/grub/menu.lst won't work. You will need to edit it so that it calls the existing installed operating system as a module.

Note: Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your root= is likely different and your rd_MD_UUID= will definitely be different, even on the same machine across installs. Generally speaking, what follows the kernel /vmlinuz-2.6.32.21-167.xendom0.fc12.x86_64 ... entry made by the dom0 kernel can be copied after the module /vmlinuz-2.6.32.21-167.xendom0.fc12.x86_64 ... entry in the example below. 

vim /boot/grub/menu.lst

title Xen 3.4.x, Linux kernel 2.6.32.21-167.xendom0.fc12.x86_64
	root   (hd0,2)
	kernel /xen.gz dom0_mem=1024M
	module /vmlinuz-2.6.32.21-167.xendom0.fc12.x86_64 ...
	module /initramfs-2.6.32.21-167.xendom0.fc12.x86_64.img

Lastly, we need to tell fstab to mount the virtual /proc/xen file system on boot. Do this by appending the following line to /etc/fstab 

echo "xenfs                   /proc/xen               xenfs   defaults        0 0" >> /etc/fstab


Make xend play nice with clustering

By default under Fedora 13, cman will start before xend. This is a problem because xend takes the network down as part of it's setup. This causes totem communication to fail which leads to fencing.

To avoid this, edit /etc/init.d/xend and tell it to start earlier than position 98. This is done by changing the line chkconfig: 2345 98 01 to chkconfig: 2345 11 98.

We also don't want it to stop until cman has stopped. We accomplish this by adding cman to the Required-Stop line. 

vim /etc/init.d/xend

#!/bin/bash
#
# xend          Script to start and stop the Xen control daemon.
#
# Author:       Keir Fraser <keir.fraser@cl.cam.ac.uk>
#
# chkconfig: 2345 11 98
# description: Starts and stops the Xen control daemon.
### BEGIN INIT INFO
# Provides:          xend
# Required-Start:    $syslog $remote_fs
# Should-Start:
# Required-Stop:     $syslog $remote_fs cman
# Should-Stop:
# Default-Start:     3 4 5
# Default-Stop:      0 1 2 6
# Default-Enabled:   yes
# Short-Description: Start/stop xend
# Description:       Starts and stops the Xen control daemon.
### END INIT INFO

With xend set to start at a position lower than 98, we now have room for chkconfig to put other daemons after it in the start order, which will be needed a little later. First and foremost, we now need to tell cman to not start until after xend is up.

As above, we will now edit cman's /etc/init.d/cman script. This time though, we will not edit it's chkconfig line. Instead, we will simply add xend to the Required-Start line. 

vim /etc/init.d/cman

#!/bin/bash
#
# cman - Cluster Manager init script
#
# chkconfig: - 21 79
# description: Starts and stops cman
#
#
### BEGIN INIT INFO
# Provides:             cman
# Required-Start:       $network $time xend
# Required-Stop:        $network $time
# Default-Start:
# Default-Stop:
# Short-Description:    Starts and stops cman
# Description:          Starts and stops the Cluster Manager set of daemons
### END INIT INFO

Finally, remove and re-add the xend and cman daemons to re-order them in the start list: 

chkconfig xend off; chkconfig cman off; chkconfig xend on; chkconfig cman on

Confirm that the order has changed so that xend is earlier in the boot sequence than cman. Assuming you've switched to run-level 3, run: 

ls -lah /etc/rc3.d/

Your start sequence should now look like: 

lrwxrwxrwx.  1 root root   14 Sep  1 19:26 S26xend -> ../init.d/xend
lrwxrwxrwx.  1 root root   14 Sep  1 19:26 S27cman -> ../init.d/cman

Booting Into The New dom0

If everything went well, you should be able to boot the new dom0 operating system. If you watch the boot process closely, you will see that the boot process is different. You should now see the Xen hypervisor boot prior to handing off to the "host" operating system. This can be confirmed once the dom0 operating system has booted by checking that the file /proc/xen/capabilities exists. What it contains doesn't matter at this stage, only that it exists at all. 

cat /proc/xen/capabilities

control_d

If you see something like this, then you are ready to proceed!



 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche! Enterprise Support:
Alteeve Support 		Community Support
© Alteeve's Niche! Inc. 1997-2024   Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.
Retrieved from ‘https://alteeve.com/w/index.php?title=Two_Node_Fedora_13_Cluster_-_Xen-Based_Virtual_Machine_Host_on_DRBD%2BCLVM&oldid=2081’