Alteeve Wiki - User contributions [en-gb]

Setting Up a PXE Server on an RPM-based OS

2012-03-11T22:27:06Z

SRSullivan: /* The Configuration Files */ Description of Colour hash (#) codes had the alpha on the wrong end. Verified against syslinux docs. Please update static download file.

{{howto_header}}

= About this Tutorial =

This tutorial covers the steps needed to make a [http://en.wikipedia.org/wiki/Preboot_Execution_Environment PXE] server. It will be used for hosting multiple [[Operating Systems]] that can be booted from a machine's network card. The main reason for this setup is to host installation media removing the requirement for have optical drives in machines. It also saves you from having a pile of optical discs kicking around.

== Prerequisite ==

This tutorial assumes that you have a fresh install of [[EL6]] and that the machine's eth1 device has been statically set to 10.255.255.254 and is the interface and IP address machines will use to talk to the PXE server.

== Parts Needed ==

A PXE boot server is fairly strait forward. You need:
* dhcp; This answers a workstation's request for an IP during the boot process.
* tftp-server; This is a PXE compliant FTP server than handles passing the core boot files to the remote machine.
* syslinux; This handles those special boot files that the remote machine needs to boot.
* httpd; Once the boot files start up the remote machine, generally you will tell it to pull the main files from a webserver. This is the Apache webserver that will server that purpose.

= Configuring Needed Services =

== dhcp ==

First you will need a DHCP server. Here is a good little tutorial to follow. Come back once you finish.

* [[DHCP on an RPM-based OS]]

=== Addition to the DHCP Configuration File ===

We need to add a section to the DHCP server configuration file, dhcpd.conf. We need to add an option to the relevant subnet directive(s) indicating where interested clients can go to get the PXE boot configuration. In this example, the PXE server and DHCP server are one in the same, but this is by choice only.

The section to add is below.

<source lang="text">
# These two options tell clients where to go to get the file needed to
# start the boot process.
next-server 192.168.1.254;
filename "pxelinux.0";
</source>

The updated dhcpd.conf file should look something like the example below.

'''Note''': [[EL5]] distributions store this file at /etc/dhcpd.conf.

<source lang="bash">
vim /etc/dhcp/dhcpd.conf
</source>
<source lang="bash">
#
# DHCP Server Configuration file.
# see /usr/share/doc/dhcp*/dhcpd.conf.sample
# see 'man 5 dhcpd.conf'
#

### Global options
# General domain information
option domain-name "alteeve.com";
option domain-name-servers 192.139.81.117, 192.139.81.1;

# Tell the server that it's authoritive on our network.
authoritive;

# This is required for EL5 operating systems but is optional on EL6 and newer
# Fedoras (F13+, at least). It controls how dynamic DNS updating is handled. In
# our case, we aren't concerned about DDNS so we'll set it to 'none'.
ddns-update-style none;

### Subnet options
subnet 10.255.0.0 netmask 255.255.0.0 {
# These two options tell clients where to go to get the file needed to
# start the boot process.
next-server 10.255.255.254;
filename "pxelinux.0";

# This is the DHCP server, but not the actual Internet gateway. So this
# Argument points our clients to the right box.
option routers 10.255.255.254;

# Set our range. This can be whatever you want so long as it fits in
# your netmask.
range 10.255.0.10 10.255.0.250;

# If clients don't ask, make the lease available for the following
# number of seconds. If the client does ask, allow up to this number of
# seconds. 86,400s = 24h.
default-lease-time 86400;
max-lease-time 86400;
}
</source>

Restart the DHCP server and you're done.

<source lang="bash">
/etc/init.d/dhcpd restart
</source>
<source lang="text">
Starting dhcpd: [ OK ]
</source>

== tftp ==

We will use the 'trivial FTP' program as it is a PXE-compliant FTP program that can transfers the boot files from the PXE server to the client. It runs as an xinetd package, so we will need to make sure it is installed as well. All told, it lives up to it's name as it is quite trivial to setup.

First, install them.

<source lang="bash">
yum install tftp-server xinetd
</source>

Then edit /etc/xinted.d/tftp and change disable = yes to disable = no. The edited file should now look like this:

<source lang="bash">
vim /etc/xinetd.d/tftp
</source>
<source lang="text">
# default: off
# description: The tftp server serves files using the trivial file transfer \
# protocol. The tftp protocol is often used to boot diskless \
# workstations, download configuration files to network-aware printers, \
# and to start the installation process for some operating systems.
service tftp
{
disable = no
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -s /var/lib/tftpboot
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}
</source>

Being an xinetd service, that is what we need to enable at boot time and then start.

<source lang="bash">
chkconfig xinetd on
/etc/init.d/xinetd restart
</source>
<source lang="text">
Stopping xinetd: [FAILED]
Starting xinetd: [ OK ]
</source>

Done!

= pxelinux =

In the xintd file for tftp was this line:

<source lang="text">
server_args = -s /var/lib/tftpboot
</source>

This determines where the PXE boot files will be setup. Some people like to change this to be /tftpboot, but we'll keep it there to keep things simple. In Fedora, this directory already exists and should be world-readable. If it isn't for some reason, create it and set the permissions to 0755 or 0777, depending on your security requirements.

'''Note''': On [[EL5]], the server_args value is -s /tftpboot. You can leave this as-is and adapt the rest of the tutorial to this directory if you wish. Otherwise, change this to /var/lib/tftpboot. Changing it is recommended as it will facilitate future upgrades or migrations of your configuration.

So now we need to install a package called syslinux.

<source lang="bash">
yum install syslinux
</source>

== Setting Up the Boot Environment ==

Next up, we need to copy a couple files into the tftpboot directory. These files are provided by the syslinux package we installed earlier and can be found in the /usr/share/syslinux/ directory.

The main files are:
* pxelinux.0: This is the actual kernel that is passed to the client to begin the boot process. You'll notice this is the file specified in the dhcpd.conf file earlier in our setup. It boots the client far enough so that it can see the boot menu, if any, and then move on to find the main system to boot. This "main system" could be a boot DVD or a full [[OS]].
* vesamenu.c32: This is the 32-bit [http://syslinux.zytor.com/wiki/index.php/SYSLINUX#COMBOOT_image comboot] file. This enables 32-bit colour images to be used for the boot menu (alpha+rr+gg+bb). This replaces the older 16-colour menu.c32 of earlier version that allow for very basic images.

<source lang="bash">
rsync -a /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
rsync -a /usr/share/syslinux/vesamenu.c32 /var/lib/tftpboot/
</source>

'''Note''': On [[EL5]], only pxelinux.0 exists and it is in /usr/lib/syslinux/. The vesamenu.c32 can be copied from an [[EL6]] machine or downloaded from this server

For [[EL5]] users;

<source lang="bash">
rsync -a /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/
wget -c https://alteeve.com/files/pxe/tftpboot/vesamenu.c32 -P /var/lib/tftpboot/
</source>

=== Client Configuration Files ===

Client configuration files will be placed in a new directory under tftpboot called pxelinux.cfg. So to start, we need to create it:

<source lang="bash">
mkdir /var/lib/tftpboot/pxelinux.cfg
</source>

Before we talk about the contents of the configuration files, it is important to understand how the PXE server decides which one to use for a given client.

When a client connects, the PXE server looks at the client's [[MAC]] address and checks to see if there is a matching hyphen-separated configuration file. If that isn't found, it then looks at the client's IP address, as set by the DHCP server. It looks for configuration files matching the [[hexadecimal]] representation of the IP address. For example, If the client was given the IP address 192.168.1.200, the PXE server will start by looking for a configuration file called C0A801C8. If it doesn't find a matching file, it will then knock-off the right-most nibble and check again. It will do this until all the possible file names are checked.

If the PXE server finds no configuration file matching the MAC address or any variant on the IP address, it falls back to a configuration file called default.

Let's show this series using the example of a client with MAC address 00:24:7e:69:6f:0e and having been assigned the IP address 192.168.1.200. The PXE server will then look for the following configuration file names in the following order:

<source lang="text">
/var/lib/tftpboot/pxelinux.cfg/00-24-7E-69-6F-0E
/var/lib/tftpboot/pxelinux.cfg/C0A801C8
/var/lib/tftpboot/pxelinux.cfg/C0A801C
/var/lib/tftpboot/pxelinux.cfg/C0A801
/var/lib/tftpboot/pxelinux.cfg/C0A80
/var/lib/tftpboot/pxelinux.cfg/C0A8
/var/lib/tftpboot/pxelinux.cfg/C0A
/var/lib/tftpboot/pxelinux.cfg/C0
/var/lib/tftpboot/pxelinux.cfg/C
/var/lib/tftpboot/pxelinux.cfg/default
</source>

I've made a little script to convert decimal-type IP addresses into hexadecimal-type specifically to help in naming these configuration files. I am sure there are many others out there.
* [[ip_to_hex]]

=== The Configuration Files ===

All of the possible configuration files can be setup using the same set of options and can be setup in similar ways. There is nothing special about any given configuration file. For this reason, we will cover the contents of the default configuration file only.

Below is the default configuration file we will use. Comments are embedded explaining each option.

* Direct download: /var/lib/tftpboot/pxelinux.cfg/[https://alteeve.com/files/pxe/tftpboot/pxelinux.cfg/default default]
* Direct download: /var/lib/tftpboot/[https://alteeve.com/files/pxe/tftpboot/an-pxe_splash_1024_768.png an-pxe_splash_1024_768.png]

<source lang="bash">
vim /var/lib/tftpboot/pxelinux.cfg/default
</source>
<source lang="perl">
# Use the high-colour menu system. This file, and the low-colour 'menu.c32'
# version, are provided by the syslinux package and can be found in the
# '/var/lib/tftpboot' directory. Copy it to '/var/lib/tftpboot'.
UI vesamenu.c32

# Time out and use the default menu option. Defined as tenths of a second.
TIMEOUT 600

# Prompt the user. Set to '1' to automatically choose the default option. This
# is really meant for files matched to MAC addresses.
PROMPT 0

# Set the boot menu to be 1024x768 with a nice background image. Be careful to
# ensure that all your user's can see this resolution! Default is 640x480.
MENU RESOLUTION 1024 768
# This file must be in or under the '/var/lib/tftpboot' folder.
MENU BACKGROUND an-pxe_splash_1024_768.png

# These do not need to be set. I set them here to show how you can customize or
# localize your PXE server's dialogue.
MENU TITLE AN!PXE Boot Server
# Below, the hash (#) character is replaced with the countdown timer. The
# '{,s}' allows for pluralizing a word and is used when the value is >= '2'.
MENU AUTOBOOT Will boot the next device as configured in your BIOS in # second{,s}.
MENU TABMSG Press the <tab> key to edit the boot parameters of the highlighted option.
MENU NOTABMSG Editing of this option is disabled.

# The following options set the various colours used in the menu. All possible
# options are specified except for F# help options. The colour is expressed as
# two hex characters between '00' and 'ff' for alpha, red, green and blue
# respectively (#AARRGGBB).
# Format is: MENU COLOR <Item> <ANSI Seq.> <foreground> <background> <shadow type>
MENU COLOR screen 0 #80ffffff #00000000 std # background colour not covered by the splash image
MENU COLOR border 0 #ffffffff #ee000000 std # The wire-frame border
MENU COLOR title 0 #ffff3f7f #ee000000 std # Menu title text
MENU COLOR sel 0 #ff00dfdf #ee000000 std # Selected menu option
MENU COLOR hotsel 0 #ff7f7fff #ee000000 std # The selected hotkey (set with ^ in MENU LABEL)
MENU COLOR unsel 0 #ffffffff #ee000000 std # Unselected menu options
MENU COLOR hotkey 0 #ff7f7fff #ee000000 std # Unselected hotkeys (set with ^ in MENU LABEL)
MENU COLOR tabmsg 0 #c07f7fff #00000000 std # Tab text
MENU COLOR timeout_msg 0 #8000dfdf #00000000 std # Timout text
MENU COLOR timeout 0 #c0ff3f7f #00000000 std # Timout counter
MENU COLOR disabled 0 #807f7f7f #ee000000 std # Disabled menu options, including SEPARATORs
MENU COLOR cmdmark 0 #c000ffff #ee000000 std # Command line marker - The '> ' on the left when editing an option
MENU COLOR cmdline 0 #c0ffffff #ee000000 std # Command line - The text being edited
# Options below haven't been tested, descriptions may be lacking.
MENU COLOR scrollbar 0 #40000000 #00000000 std # Scroll bar
MENU COLOR pwdborder 0 #80ffffff #20ffffff std # Password box wire-frame border
MENU COLOR pwdheader 0 #80ff8080 #20ffffff std # Password box header
MENU COLOR pwdentry 0 #80ffffff #20ffffff std # Password entry field
MENU COLOR help 0 #c0ffffff #00000000 std # Help text, if set via 'TEXT HELP ... ENDTEXT'

## To keep the menu from getting out of hand, I like to create sub-menus.
## The sub-menus are simply additional files in the same directory as this
## file with one or more boot options or, if you wish, further sub-menus.

# As I build a lot of clusters, I like to have a section dedicated to boot
# options used to build cluster nodes. These are the EL6 + RHCS 3 nodes.
MENU BEGIN cluster_rhel
MENU TITLE Cluster 3 nodes - RHEL 6
LABEL Previous
MENU LABEL ^B) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/cluster3-rhel.menu
MENU END

# As I build a lot of clusters, I like to have a section dedicated to boot
# options used to build cluster nodes. These are the EL6 + RHCS 3 nodes.
MENU BEGIN cluster_centos
MENU TITLE Cluster 3 nodes - CentOS 6
LABEL Previous
MENU LABEL ^C) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/cluster3-centos.menu
MENU END

# As I build a lot of clusters, I like to have a section dedicated to boot
# options used to build cluster nodes. These are the EL6 + RHCS 3 nodes.
MENU BEGIN cluster_fedora
MENU TITLE Cluster 3 nodes - Fedora
LABEL Previous
MENU LABEL ^D) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/cluster3-fedora.menu
MENU END

# This section covers base options without pre-defined kickstart (or other)
# automation scripts.
MENU BEGIN base
MENU TITLE Base installations (as if booting from media)
LABEL Previous
MENU LABEL ^E) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/base.menu
MENU END

# This section covers generic server installs
MENU BEGIN stock
MENU TITLE Stock installations (common, generic server types)
LABEL Previous
MENU LABEL ^F) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/stock.menu
MENU END
</source>

The default file above points to four sub-menu files. How many you use, if you use any at all, is up to you. You can also use any file name you want, or any directory under /var/lib/tftpboot/.

To save space on this page, The sub-menu files mentioned above are their own pages and are linked below. These will hopefully act as useful references for your own sub-menu files, should you wish to use menus.

* [[pxe_generic.menu|generic.menu]]
* [[pxe_cluster.menu|cluster.menu]]
* [[pxe_stock.menu|stock.menu]]
* [[pxe_live.menu|live.menu]]

=== Options ===

Let's now take a look at the various parts. This is an overview only, for a complete list of options please read /usr/share/doc/syslinux-3.84/menu.txt. Update the version number to match your installed version.

Wherever a colour can be specified, the format is #AARRGGBB. That is, alpha (transparency), red, green and blue intensity expressed as a 256 range specified using two [[hexadecimal]] characters per option.

* UI vesamenu.c32

This loads the 32-bit colour [http://syslinux.zytor.com/wiki/index.php/SYSLINUX#COMBOOT_image COMBOOT] image. This allows for the full colour range plus 8-bit alpha (transparency) to be available at boot time. It's what allows for a much more attractive boot menu. To read details on the innards of the COMBOOT format, read /usr/share/doc/syslinux-3.84/comboot.txt. Update the version number to match your installed version.

* TIMEOUT 600

This is the amount of time, in tenths of a second, that the PXE loader will wait before performing the default action. If the user navigates through the menu. the timer will stop and wait for the user to make a selection. To prevent an install from running accidentally, the default option should always be a non-damaging option.

* prompt 0

This tells the PXE server to prompt the user to make a choice. If this is set to 1 then the PXE server will use the default option.

* MENU RESOLUTION 1024 768

This tells the boot loader to run at the set resolution, 1024x768 in this example. The default is to run up as vga (640x480). Be sure when pushing a higher resolution that all of your potential users have machines that support the given resolution.

* MENU BACKGROUND an-pxe_splash_1024_768.png

This is the background image to use. This must exist in or under the /var/lib/tftpboot/ directory. If you want to use a subdirectory, specify it as a relative path. For example, if you want to store images in /var/lib/tftpboot/images/, then this would be set to images/an-pxe_splash_1024_768.png. The format of the image can be JPEG or The format of the image can be JPNG (other image formats may work). The image itself should match the size of the screen.

* MENU TITLE AN!PXE Boot Server
* MENU AUTOBOOT Will boot the next device as configured in your BIOS in # second{,s}.
* MENU TABMSG Press the <tab> key to edit the boot parameters of the highlighted option.
* MENU NOTABMSG Editing of this option is disabled.

These options allow you to customize or localize the text. None need to be specified.

The AUTOBOOT option is a little special to accommodate the count-down. The # will be replaced by the time remaining. The {,s} tells the string to add an s to the end of seconds when the time remaining is greater than one.

* MENU COLOR x

These options control the colour and transparency of the various text-elements, borders and backgrounds. None of them need to be specified and all have sane default values. Most of the available options are shown in the example above.

Each entry is formatted like so:

MENU COLOR <type> <ansi> <foreground> <background> <shadow>

The <type> is the name of the element you are manipulating. The only 'type' not show in the example above is msgXX where XX is a number between 1 and 12. These are used to control "help windows" that can be shown to the user when they press an F[1-12] key.

The <ansi> number(s) tell the boot loader how to format the text using [[ANSI]] code. This is set to 0, "reset", in our example to clear any ANSI formatting. You can specify multiple ANSI codes by separating them with ; (semi-colons).

The <foreground and <background> values are the [[hexadecimal]] notation for the transparency and colour in the format #AARRGGBB as mentioned earlier.

The <shadow> is the type of drop-shadow to render for the <type>. Valid options are:
* none: No shadowing.
* std, standard: Foreground pixels are raised.
* all: both foreground and background pixels are raised.
* rev, reverse: Background pixels are raised.

=== Labels ===

Label blocks are the actual boot options made available within the PXE menu. There are many options available here that are not shown in this example. If you are curious, please read /usr/share/doc/syslinux-3.84/menu.txt which was installed on your system when you installed syslinux. Of course, replace 3.84 with the version you have installed.

First example;

<source lang="text">
LABEL next
MENU LABEL ^A) Boot the next device as configured in your BIOS
MENU DEFAULT
localboot
</source>

This example is a good one to always use as the first, and default, option.

The LABEL next defines this option group. The name next has no special meaning, but can be used for more complex setups later. For now, just be sure to keep this name simple with no spaces.

The MENU LABEL ... option control two things. First, it's the text shown to the user. Second, the ^A sets the keyboard key that the user can press to select the option. Whatever character comes immediately after the (caret), A in this example, becomes the mapped key. Please note that the ^ can come anywhere in the title text. Ensure that no two entries have the same character mapped!

The MENU DEFAULT command tells the boot loader to automatically boot this option if the TIMEOUT expires. Only one entry may have this option.

The localboot is a special command the skips the PXE boot loader. Your BIOS should proceed to boot the next device in it's boot order list. Generally this will boot the local hard drive.

Another example, from the cluster3-rhel.menu file:

<source lang="text">
LABEL rhel6_an-node01
MENU LABEL ^1) Install AN!Node01; Storage Node 01 - Cluster 3
KERNEL boot/rhel6/x86_64/vmlinuz
APPEND initrd=boot/rhel6/x86_64/initrd.img ks=http://192.168.1.254/rhel6/x86_64/ks/an-node01.ks #ksdevice=eth1
</source>

This is a pretty typical entry. The two new options to note are:

* KERNEL ...

This is the path to the boot kernel used to start the OS or it's installer. Most distributions make use of a small kernel to run up their installer or Live CD/DVD environment. The PXE server will search for the kernel relative to the tftpboot directory. Where to get the proper boot kernel will depend on the distribution you are setting up.

Some examples will be [[#Setting Up The Boot Environment|shown below]].

* APPEND ...

This argument allows you to pass arguments to the KERNEL specified above. Which options can be passed will, again, depend on the distribution you are setting up. In this example, an initrd image is specified, a kickstart script is set and the network device to use to search for the kickstart script are defined.

The [[Setting_Up_a_PXE_Server_in_Fedora#Making_The_Install_Files_For_An_OS_Available|Apache configuration]] section below covers in more detail how to make the kickstart and OS installation files available.

=== Menus ===

Menus are very useful when you have more than a few entries in your PXE server. They allow for creating groups of bootable options based on whatever given criteria works for you.

Let's look at the "cluster" menu from above.

<source lang="text">
MENU BEGIN cluster_rhel
MENU TITLE Cluster 3 nodes - RHEL 6
LABEL Previous
MENU LABEL ^B) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/cluster3-rhel.menu
MENU END
</source>

* MENU BEGIN ...

This begins the section and is followed by a name that should (must?) be unique.

* MENU TITLE ...

This is the text shown at the top of the sub-menu once selected.

* LABEL ...

This is the text shown as the first option in the sub-menu that returns to the previous screen.

* MENU LABEL ...

This is the text that the user selects to descend into the sub-menu.

'''Note''': I use the ^ character to enable a letter to select the menu option but it doesn't seem to work. Please let me know if you have further information on how, or if, sub-menu options are selected with a key press.

* MENU EXIT ...

I am not sure, exactly, what this option does.

* MENU SEPARATOR ...

I am not sure, exactly, what this option does.

* MENU INCLUDE ...

This is the path and file to include when building this sub-menu. The path is relative to the tftpboot server_args directory, as set in /etc/xinetd.d/tftp file.

* MENU END

This closes the sub-menu section.

There are other options, though none seem to be well documented at the moment. I got this far thanks to [https://help.ubuntu.com/community/PXEInstallMultiDistro this page]. Any additional information on these and other options would be welcome!

= Setting Up Apache =

[[Image:c6_httpd_default_page_01.png|thumb|right|400px|Default Apache2 web page on CentOS 6.]]

We will use the Apache web server to make our operating system images available. First, install the server.

<source lang="bash">
yum install httpd
</source>

The default install of Apache2 will work fine for us, so we won't adjust it. Let's start it now.

<source lang="bash">
/etc/init.d/httpd start
</source>

Test access by loading [http://10.255.255.254 http://10.255.255.254] a machine on your network, replacing 10.255.255.254 with a resolvable name or IP address of your machine. If this doesn't work, make sure [[TCP]] port 80 is open on your firewall/PXE server.

== Setting Up The DocumentRoot ==

Apache has a variable called DocumentRoot in it's main configuration file /etc/httpd/conf/httpd.conf. Anything in or under the directory specified in this variable will be exposed on your web server. The default directory is /var/www/html and this will be where we put our various kickstart files, ISO files and under which we will mount those ISO files.

== The Default Page ==

There is no need to create an index.html page, as the image kickstart scripts and image files will be called directly. Personally though, I like to put together a simple page listing the various installations I've made available on the PXE server. If nothing else, it helps identify the machine when you call it in a browser.

If you decide to create one, please use whatever website tool, text editor or [[Integrated Development Environment|IDE]] you prefer. For reference, below are the very simple, probably not [[W3C]] compliant static [[HTML]] page I use along with a trivial [[CSS]] file.

* [[PXE Server index.html]]
* [[PXE Server common.css]]

Put the index.html (and associated files) directly in the DocumentRoot directory. By default, Apache2 will look for a file named index.html and load it if it is found.

== Making The Install Files For An OS Available ==

At the end of the day, the only thing that matters here is that all of the installation files and kickstart scripts are ''somewhere'' under the DocumentRoot directory. I will describe here the layout that I like, but feel free to adjust it however you see fit.

Further, different operating systems will need to be configured differently. For the purpose of this tutorial, I will be showing how RPM-based [[Linux]] distributions are configured. If you are using different operating systems, you should be able to build on the steps below.

=== Planning Your Directory Structure ===

Under the DocumentRoot directory, we will need to create a set of directories to store the operating system files and associated kickstart scripts in. I've found the following structure works well.

* <DocumentRoot>/<os_name>/<architecture>/[ks|iso|img].

Using [[CentOS]] v6, [[x86_64]] with the default Apache DocumentRoot as an example, you would create the following directories. Note that I like to shorten Operating System names to the shortest version practical to save typing. Feel free to use the full names if you prefer verbose [[URL]]s.

* /var/www/html/c6/x86_64/ks: Kickstart scripts go here
* /var/www/html/c6/x86_64/iso: The [http://mirror.its.sfu.ca/mirror/CentOS/6.2/isos/x86_64/CentOS-6.2-x86_64-bin-DVD1.iso CentOS-6.2-x86_64-bin-DVD1.iso] file is saved here. Alternatively, you can find [http://www.centos.org/modules/tinycontent/index.php?id=30 a mirror] closer to you.
* /var/www/html/c6/x86_64/img: The [[ISO]] file above is mounted here.

If you wanted to also make the [[i386]] version of CentOS available, simply replace x86_64 with i386 above.

=== CentOS 6 x86_64 ===

Let's walk through the actual steps needed to make CentOS 6.2 x86_64 available on Apache now.

First, create the directory structure.

<source lang="bash">
mkdir -p /var/www/html/c6/x86_64/{iso,img,ks}
</source>

Now, download the ISO if you have not yet done so. If you have, move it into place.

<source lang="bash">
cd /var/www/html/c6/x86_64/iso
wget -c http://centos.alteeve.com/6.2/isos/x86_64/CentOS-6.2-x86_64-bin-DVD1.iso
</source>

Now we mount the ISO using a [[loop-back]] device.

<source lang="bash">
mount -o loop /var/www/html/c6/x86_64/iso/CentOS-6.2-x86_64-bin-DVD1.iso /var/www/html/c6/x86_64/img
df -hP
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/luks-51fbeed6-2efc-4eef-a8d6-8431a4fcceb8 454G 18G 414G 5% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 1008M 92M 866M 10% /boot
/var/www/html/c6/x86_64/iso/CentOS-6.2-x86_64-bin-DVD1.iso 4.2G 4.2G 0 100% /var/www/html/c6/x86_64/img
</source>

Lastly, copy any kickstart scripts into the /var/www/html/c6/x86_64/ks/ directory. These scripts will be called by the user after they select an automated install option from the PXE menu.

That's it! You can now use make CentOS 6.2 available as a PXE boot option in your menu.

== Mounting And Unmounting Many ISOs ==

If you make any real use of your PXE server, you will probably find that you will have a lot of ISO files to mount and unmount. To simplify this task, I like to create a couple of bash scripts called, simply, mount_iso.sh and umount_iso.sh.

These are simple shell scripts meant to mount and unmount all ISOs with one command. You need to be sure that you keep them up to date as you add and remove operating systems.

You will certainly want to adjust these to suit your setup. With that said, here are my scripts that you can use as templates.

Mount;

<source lang="bash">
vim /root/mount_iso.sh
</source>
<source lang="bash">
#!/bin/bash
# This is used to mount the various ISO used by the PXE server on this server.

echo -n "Mounting RHEL 6.2, x86_64... "
mount -o loop /var/www/html/rhel6/x86_64/iso/rhel-server-6.2-x86_64-dvd.iso /var/www/html/rhel6/x86_64/img
echo "Done!"

echo -n "Mounting CentOS 6.2, x86_64... "
mount -o loop /var/www/html/c6/x86_64/iso/CentOS-6.2-x86_64-bin-DVD1.iso /var/www/html/c6/x86_64/img
echo "Done!"

echo -n "Mounting Fedora 16, x86_64... "
mount -o loop /var/www/html/f16/x86_64/iso/Fedora-16-x86_64-DVD.iso /var/www/html/f16/x86_64/img
echo "Done!"
</source>

Unmount;

<source lang="bash">
vim /root/umount_iso.sh
</source>
<source lang="bash">
#!/bin/bash
# This is used to unmount the various ISO used by the PXE server on this server.

echo -n "Unmounting RHEL 6.2, x86_64... "
umount /var/www/html/rhel6/x86_64/img
echo "Done!"

echo -n "Unmounting CentOS 6.2, x86_64... "
umount /var/www/html/c6/x86_64/img
echo "Done!"

echo -n "Unmounting Fedora 16, x86_64... "
umount /var/www/html/f16/x86_64/img
echo "Done!"
</source>

=== Increasing Maximum number of Mountable ISOs ===

By default the loop driver only creates 8 loop back devices. Each mounted [[ISO]] takes one of those and there for limits the total number of ISOs that can be mounted at any one time.

There exist two places to change the maximum number of loop back devices.
The recommend method is to add max_loop option to a new config file in /etc/modprobe.d/.

<source lang="bash">
vim /etc/modprobe.d/loop.conf
</source>
<source lang="text">
options loop max_loop=64
</source>

The second options is to add max_loop=64 to the kernel boot line.

Adjust the number after max_loop= to suit your installations needs.

= Setting Up The Boot Environment =

All boot options, except localboot, need to have a kernel made available via the PXE server. Specifically, these kernels must exist somewhere under the /var/lib/tftpboot directory. What kernel or boot image a given operating system will need depends entirely on that operating system. For this section, we will continue to use CentOS 6 x86_64 as an example.

== Planning Your Directory Structure, Take Two ==

As with [[#Planning Your Directory Structure|Apache]], we're potentially going to copy a lot of boot images and other files. It makes sense to have an organized directory structure to put them all in. As before, I will share what I do, but please feel free to do what you find works best for you.

I like to create a dedicated directory called boot and then mimic the layout from Apache, minus the last set of three directories.

* <tftpboot>/boot/<os_name>/<architecture>/

Using CentOS 6 x86_64 with the default tftpboot install, I will create the following directory.

<source lang="bash">
mkdir -p /var/lib/tftpboot/boot/c6/x86_64
</source>

== The Bootable Kernel ==

Remember in the earlier LABEL c6_an-node01 example there was the APPEND initrd=boot/c6/x86_64/initrd.img ... line? The boot/c6/x86_64/initrd.img is the kernel, in the directory we just created!

The last piece of the puzzle then is; Where does the initrd.img file come from?

With most Linux distributions, there is an isolinux directory in the root of the installation DVD. The files needed to boot the DVD's installer (or "rescue mode") are in this directory. These are the same files we need to boot over PXE. If you have followed this tutorial up to this point, then you should be able to see the contents of this directory in your file browser. My PXE server uses the IP address 192.168.1.254, so I can go to
http://10.255.255.254/c6/x86_64/img/isolinux and see, among other files, initrd.img.

With the directory created and the ISO mounted, all that is left is to copy the contents of the isolinux files.

<source lang="bash">
rsync -av /var/www/html/c6/x86_64/img/isolinux/* /var/lib/tftpboot/boot/c6/x86_64/
</source>
<source lang="text">
sending incremental file list
TRANS.TBL
boot.cat
boot.msg
grub.conf
initrd.img
isolinux.bin
isolinux.cfg
memtest
splash.jpg
vesamenu.c32
vmlinuz

sent 35199717 bytes received 221 bytes 23466625.33 bytes/sec
total size is 35194744 speedup is 1.00
</source>

And we're done!

Now repeat the relevant steps for the rest of the operating systems you want to support via PXE.

= Credits =

References used:
* [http://linux-sxs.org/internet_serving/pxeboot.html Linux-SXS]
* [https://help.ubuntu.com/community/PXEInstallMultiDistro Ubuntu]
* [http://www.c3.hu/docs/oreilly/tcpip/tcpip/appd_03.htm O'Reilly]
* [http://www.linuxdoc.org/HOWTO/CDServer-HOWTO/addloops.html linuxdoc.org]

{{footer}}

Setting Up a PXE Server on an RPM-based OS

2011-09-29T17:32:46Z

SRSullivan: Undo revision 3698 by SRSullivan (talk), Symlinks trying to leave tftp's chroot fail.

{{howto_header}}

= About this Tutorial =
'''Dec. 30, 2010''': This tutorial has been confirmed to work on [[RHEL]] 6.0 and Fedora 13+.

'''Nov. 08, 2010''': This is the first draft of the final tutorial. As such, should be fairly safe to follow, though mistakes and omissions may exist. Suggestions for clarification are also appreciated.

This tutorial covers the steps needed to make a [http://en.wikipedia.org/wiki/Preboot_Execution_Environment PXE] server. It will be used for hosting multiple [[Operating Systems]] that can be booted from a machine's network card. The main reason for this setup is to host installation media removing the requirement for have optical drives in machines. It also saves you from having a pile of optical discs kicking around.

== Prerequisite ==

This tutorial assumes that you have a fresh install of [[Fedora]] 13+ and/or RHEL 6.0 and that the machine's eth0 device has been statically set to 192.168.1.10 and the subnet's router is set to 192.168.1.1. This should be easy to adapt to other distributions and network configurations.

== Parts Needed ==

A PXE boot server is fairly strait forward. You need:
* dhcp; This answers a workstation's request for an IP during the boot process.
* tftp-server; This is a PXE compliant FTP server than handles passing the core boot files to the remote machine.
* syslinux; This handles those special boot files that the remote machine needs to boot.
* httpd; Once the boot files start up the remote machine, generally you will tell it to pull the main files from a webserver. This is the Apache webserver that will server that purpose.

= Configuring Needed Services =

== dhcp ==

First you will need a DHCP server. Here is a good little tutorial to follow. Come back once you finish.

* [[DHCP on an RPM-based OS]]

=== Addition to the DHCP Configuration File ===

We need to add a section to the DHCP server configuration file, dhcpd.conf. We need to add an option to the relevant subnet directive(s) indicating where interested clients can go to get the PXE boot configuration. In this example, the PXE server and DHCP server are one in the same, but this is by choice only.

The section to add is below.

<source lang="text">
# These two options tell clients where to go to get the file needed to
# start the boot process.
next-server 192.168.1.254;
filename "pxelinux.0";
</source>

The updated dhcpd.conf file should look something like the example below.

'''Note''': [[EL5]] distributions store this file at /etc/dhcpd.conf.

<source lang="bash">
vim /etc/dhcp/dhcpd.conf
</source>
<source lang="bash">
### Global options
# General domain information
option domain-name "alteeve.com";
option domain-name-servers 192.139.81.117, 192.139.81.1;

# Tell the server that it's authoritive on our network.
authoritive;

# This is required for EL5 operating systems but is optional on EL6 and newer
# Fedoras (F13+, at least). It controls how dynamic DNS updating is handled. In
# our case, we aren't concerned about DDNS so we'll set it to 'none'.
ddns-update-style none;

### Subnet options
subnet 192.168.1.0 netmask 255.255.255.0 {
# This is the DHCP server, but not the actual Internet gateway. So this
# Argument points our clients to the right box.
option routers 192.168.1.254;

# Set our range. This can be whatever you want so long as it fits in
# your netmask.
range 192.168.1.100 192.168.1.220;

# If clients don't ask, make the lease available for the following
# number of seconds. If the client does ask, allow up to this number of
# seconds. 86,400s = 24h.
default-lease-time 86400;
max-lease-time 86400;

# These two options tell clients where to go to get the file needed to
# start the boot process.
next-server 192.168.1.254;
filename "pxelinux.0";
}
</source>

Restart the DHCP server and you're done.

<source lang="bash">
/etc/init.d/dhcpd restart
</source>
<source lang="text">
Starting dhcpd: [ OK ]
</source>

== tftp ==

We will use the 'trivial FTP' program as it is a PXE-compliant FTP program that can transfers the boot files from the PXE server to the client. It runs as an xinetd package, so we will need to make sure it is installed as well. All told, it lives up to it's name as it is quite trivial to setup.

First, install them.

<source lang="bash">
yum install tftp-server xinetd
</source>

Then edit /etc/xinted.d/tftp and change disable = yes to disable = no. The edited file should now look like this:

<source lang="bash">
vim /etc/xinetd.d/tftp
</source>
<source lang="bash">
# default: off
# description: The tftp server serves files using the trivial file transfer \
# protocol. The tftp protocol is often used to boot diskless \
# workstations, download configuration files to network-aware printers, \
# and to start the installation process for some operating systems.
service tftp
{
disable = no
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -s /var/lib/tftpboot
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}
</source>

Being an xinetd service, that is what we need to enable at boot time and then start.

<source lang="bash">
chkconfig dhcpd on
chkconfig xinetd on
/etc/init.d/xinetd restart
</source>
<source lang="text">
Stopping xinetd: [FAILED]
Starting xinetd: [ OK ]
</source>

Done!

= pxelinux =

In the xintd file for tftp was this line:

<source lang="text">
server_args = -s /var/lib/tftpboot
</source>

This determines where the PXE boot files will be setup. Some people like to change this to be /tftpboot, but we'll keep it there to keep things simple. In Fedora, this directory already exists and should be world-readable. If it isn't for some reason, create it and set the permissions to 0755 or 0777, depending on your security requirements.

'''Note''': On [[EL5]], the server_args value is -s /tftpboot. You can leave this as-is and adapt the rest of the tutorial to this directory if you wish. Otherwise, change this to /var/lib/tftpboot. Changing it is recommended as it will facilitate future upgrades or migrations of your configuration.

So now we need to install a package called syslinux.

<source lang="bash">
yum install syslinux
</source>

== Setting Up the Boot Environment ==

Next up, we need to copy a couple files into the tftpboot directory. These files are provided by the syslinux package we installed earlier and can be found in the /usr/share/syslinux/ directory.

The main files are:
* pxelinux.0: This is the actual kernel that is passed to the client to begin the boot process. You'll notice this is the file specified in the dhcpd.conf file earlier in our setup. It boots the client far enough so that it can see the boot menu, if any, and then move on to find the main system to boot. This "main system" could be a boot DVD or a full [[OS]].
* vesamenu.c32: This is the 32-bit [http://syslinux.zytor.com/wiki/index.php/SYSLINUX#COMBOOT_image comboot] file. This enables 32-bit colour images to be used for the boot menu (alpha+rr+gg+bb). This replaces the older 16-colour menu.c32 of earlier version that allow for very basic images.

<source lang="bash">
rsync -av /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
rsync -av /usr/share/syslinux/vesamenu.c32 /var/lib/tftpboot/
</source>

'''Note''': On [[EL5]], only pxelinux.0 exists and it is in /usr/lib/syslinux/. The vesamenu.c32 can be copied from an [[EL6]] machine or downloaded from this server.

<source lang="bash">
rsync -av /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/
wget -c https://alteeve.com/files/pxe/tftpboot/vesamenu.c32 -P /var/lib/tftpboot/
</source>

=== Client Configuration Files ===

Client configuration files will be placed in a new directory under tftpboot called pxelinux.cfg. So to start, we need to create it:

<source lang="bash">
mkdir /var/lib/tftpboot/pxelinux.cfg
</source>

Before we talk about the contents of the configuration files, it is important to understand how the PXE server decides which one to use for a given client.

When a client connects, the PXE server looks at the client's [[MAC]] address and checks to see if there is a matching hyphen-separated configuration file. If that isn't found, it then looks at the client's IP address, as set by the DHCP server. It looks for configuration files matching the [[hexadecimal]] representation of the IP address. For example, If the client was given the IP address 192.168.1.200, the PXE server will start by looking for a configuration file called C0A801C8. If it doesn't find a matching file, it will then knock-off the right-most nibble and check again. It will do this until all the possible file names are checked.

If the PXE server finds no configuration file matching the MAC address or any variant on the IP address, it falls back to a configuration file called default.

Let's show this series using the example of a client with MAC address 00:24:7e:69:6f:0e and having been assigned the IP address 192.168.1.200. The PXE server will then look for the following configuration file names in the following order:

<source lang="text">
/var/lib/tftpboot/pxelinux.cfg/00-24-7E-69-6F-0E
/var/lib/tftpboot/pxelinux.cfg/C0A801C8
/var/lib/tftpboot/pxelinux.cfg/C0A801C
/var/lib/tftpboot/pxelinux.cfg/C0A801
/var/lib/tftpboot/pxelinux.cfg/C0A80
/var/lib/tftpboot/pxelinux.cfg/C0A8
/var/lib/tftpboot/pxelinux.cfg/C0A
/var/lib/tftpboot/pxelinux.cfg/C0
/var/lib/tftpboot/pxelinux.cfg/C
/var/lib/tftpboot/pxelinux.cfg/default
</source>

I've made a little script to convert decimal-type IP addresses into hexadecimal-type specifically to help in naming these configuration files. I am sure there are many others out there.
* [[ip_to_hex]]

=== The Configuration Files ===

All of the possible configuration files can be setup using the same set of options and can be setup in similar ways. There is nothing special about any given configuration file. For this reason, we will cover the contents of the default configuration file only.

Below is the default configuration file we will use. Comments are embedded explaining each option.

* Direct download: /var/lib/tftpboot/pxelinux.cfg/[https://alteeve.com/files/pxe/tftpboot/pxelinux.cfg/default default]
* Direct download: /var/lib/tftpboot/[https://alteeve.com/files/pxe/tftpboot/an-pxe_splash_1024_768.png an-pxe_splash_1024_768.png]

<source lang="bash">
vim /var/lib/tftpboot/pxelinux.cfg/default
</source>
<source lang="perl">
# Use the high-colour menu system. This file, and the low-colour 'menu.c32'
# version, are provided by the syslinux package and can be found in the
# '/var/lib/tftpboot' directory. Copy it to '/var/lib/tftpboot'.
UI vesamenu.c32

# Time out and use the default menu option. Defined as tenths of a second.
TIMEOUT 600

# Prompt the user. Set to '1' to automatically choose the default option. This
# is really meant for files matched to MAC addresses.
PROMPT 0

# Set the boot menu to be 1024x768 with a nice background image. Be careful to
# ensure that all your user's can see this resolution! Default is 640x480.
MENU RESOLUTION 1024 768
# This file must be in or under the '/var/lib/tftpboot' folder.
MENU BACKGROUND an-pxe_splash_1024_768.png

# These do not need to be set. I set them here to show how you can customize or
# localize your PXE server's dialogue.
MENU TITLE AN!PXE Boot Server
# Below, the hash (#) character is replaced with the countdown timer. The
# '{,s}' allows for pluralizing a word and is used when the value is >= '2'.
MENU AUTOBOOT Will boot the next device as configured in your BIOS in # second{,s}.
MENU TABMSG Press the <tab> key to edit the boot parameters of the highlighted option.
MENU NOTABMSG Editing of this option is disabled.

# The following options set the various colours used in the menu. All possible
# options are specified except for F# help options. The colour is expressed as
# two hex characters between '00' and 'ff' for red, green, blue and alpha,
# respectively (#RRGGBBAA).
# Format is: MENU COLOR <Item> <ANSI Seq.> <foreground> <background> <shadow type>
MENU COLOR screen 0 #80ffffff #00000000 std # background colour not covered by the splash image
MENU COLOR border 0 #ffffffff #ee000000 std # The wire-frame border
MENU COLOR title 0 #ffff3f7f #ee000000 std # Menu title text
MENU COLOR sel 0 #ff00dfdf #ee000000 std # Selected menu option
MENU COLOR hotsel 0 #ff7f7fff #ee000000 std # The selected hotkey (set with ^ in MENU LABEL)
MENU COLOR unsel 0 #ffffffff #ee000000 std # Unselected menu options
MENU COLOR hotkey 0 #ff7f7fff #ee000000 std # Unselected hotkeys (set with ^ in MENU LABEL)
MENU COLOR tabmsg 0 #c07f7fff #00000000 std # Tab text
MENU COLOR timeout_msg 0 #8000dfdf #00000000 std # Timout text
MENU COLOR timeout 0 #c0ff3f7f #00000000 std # Timout counter
MENU COLOR disabled 0 #807f7f7f #ee000000 std # Disabled menu options, including SEPARATORs
MENU COLOR cmdmark 0 #c000ffff #ee000000 std # Command line marker - The '> ' on the left when editing an option
MENU COLOR cmdline 0 #c0ffffff #ee000000 std # Command line - The text being edited
# Options below haven't been tested, descriptions may be lacking.
MENU COLOR scrollbar 0 #40000000 #00000000 std # Scroll bar
MENU COLOR pwdborder 0 #80ffffff #20ffffff std # Password box wire-frame border
MENU COLOR pwdheader 0 #80ff8080 #20ffffff std # Password box header
MENU COLOR pwdentry 0 #80ffffff #20ffffff std # Password entry field
MENU COLOR help 0 #c0ffffff #00000000 std # Help text, if set via 'TEXT HELP ... ENDTEXT'

### Now define the menu options

# I feel it is safest to return booting to the client as the first and default
# option. This entry below will do just that.
LABEL next
MENU LABEL ^A) Boot the next device as configured in your BIOS
MENU DEFAULT
localboot

## To keep the menu from getting out of hand, I like to create sub-menus.
## The sub-menus are simply additional files in the same directory as this
## file with one or more boot options or, if you wish, further sub-menus.
#
# This section covers boot options without pre-defined kickstart (or other)
# automation scripts.
MENU BEGIN generic
MENU TITLE Generic installations (as if booting from media)
LABEL Previous
MENU LABEL ^B) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/generic.menu
MENU END

# As I build a lot of clusters, I like to have a section dedicated to boot
# options used to build cluster nodes.
MENU BEGIN cluster
MENU TITLE Cluster nodes
LABEL Previous
MENU LABEL ^C) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/cluster.menu
MENU END

# This section is where I put generic automated install scripts. These
# generally have no specific use and are meant to create installations that
# can be easily adapted or built upon.
MENU BEGIN stock
MENU TITLE Stock installs (generic kickstart-based installs)
LABEL Previous
MENU LABEL ^D) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/stock.menu
MENU END

# Lastly, I like to make "Live" operating systems available. These are very
# useful for diagnostics and recovery work where you need more than a 'rescue'
# boot provides and where you do not want to effect the underlying system OS.
MENU BEGIN live
MENU TITLE Live Distros (bootable without affecting the underlying system)
LABEL Previous
MENU LABEL ^E) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/live.menu
MENU END
</source>

The default file above points to four sub-menu files. How many you use, if you use any at all, is up to you. You can also use any file name you want, or any directory under /var/lib/tftpboot/.

To save space on this page, The sub-menu files mentioned above are their own pages and are linked below. These will hopefully act as useful references for your own sub-menu files, should you wish to use menus.

* [[pxe_generic.menu|generic.menu]]
* [[pxe_cluster.menu|cluster.menu]]
* [[pxe_stock.menu|stock.menu]]
* [[pxe_live.menu|live.menu]]

=== Options ===

Let's now take a look at the various parts. This is an overview only, for a complete list of options please read /usr/share/doc/syslinux-3.84/menu.txt. Update the version number to match your installed version.

Wherever a colour can be specified, the format is #AARRGGBB. That is, alpha (transparency), red, green and blue intensity expressed as a 256 range specified using two [[hexadecimal]] characters per option.

* UI vesamenu.c32

This loads the 32-bit colour [http://syslinux.zytor.com/wiki/index.php/SYSLINUX#COMBOOT_image COMBOOT] image. This allows for the full colour range plus 8-bit alpha (transparency) to be available at boot time. It's what allows for a much more attractive boot menu. To read details on the innards of the COMBOOT format, read /usr/share/doc/syslinux-3.84/comboot.txt. Update the version number to match your installed version.

* TIMEOUT 600

This is the amount of time, in tenths of a second, that the PXE loader will wait before performing the default action. If the user navigates through the menu. the timer will stop and wait for the user to make a selection. To prevent an install from running accidentally, the default option should always be a non-damaging option.

* prompt 0

This tells the PXE server to prompt the user to make a choice. If this is set to 1 then the PXE server will use the default option.

* MENU RESOLUTION 1024 768

This tells the boot loader to run at the set resolution, 1024x768 in this example. The default is to run up as vga (640x480). Be sure when pushing a higher resolution that all of your potential users have machines that support the given resolution.

* MENU BACKGROUND an-pxe_splash_1024_768.png

This is the background image to use. This must exist in or under the /var/lib/tftpboot/ directory. If you want to use a subdirectory, specify it as a relative path. For example, if you want to store images in /var/lib/tftpboot/images/, then this would be set to images/an-pxe_splash_1024_768.png. The format of the image can be JPEG or The format of the image can be JPNG (other image formats may work). The image itself should match the size of the screen.

* MENU TITLE AN!PXE Boot Server
* MENU AUTOBOOT Will boot the next device as configured in your BIOS in # second{,s}.
* MENU TABMSG Press the <tab> key to edit the boot parameters of the highlighted option.
* MENU NOTABMSG Editing of this option is disabled.

These options allow you to customize or localize the text. None need to be specified.

The AUTOBOOT option is a little special to accommodate the count-down. The # will be replaced by the time remaining. The {,s} tells the string to add an s to the end of seconds when the time remaining is greater than one.

* MENU COLOR x

These options control the colour and transparency of the various text-elements, borders and backgrounds. None of them need to be specified and all have sane default values. Most of the available options are shown in the example above.

Each entry is formatted like so:

MENU COLOR <type> <ansi> <foreground> <background> <shadow>

The <type> is the name of the element you are manipulating. The only 'type' not show in the example above is msgXX where XX is a number between 1 and 12. These are used to control "help windows" that can be shown to the user when they press an F[1-12] key.

The <ansi> number(s) tell the boot loader how to format the text using [[ANSI]] code. This is set to 0, "reset", in our example to clear any ANSI formatting. You can specify multiple ANSI codes by separating them with ; (semi-colons).

The <foreground and <background> values are the [[hexadecimal]] notation for the transparency and colour in the format #AARRGGBB as mentioned earlier.

The <shadow> is the type of drop-shadow to render for the <type>. Valid options are:
* none: No shadowing.
* std, standard: Foreground pixels are raised.
* all: both foreground and background pixels are raised.
* rev, reverse: Background pixels are raised.

=== Labels ===

Label blocks are the actual boot options made available within the PXE menu. There are many options available here that are not shown in this example. If you are curious, please read /usr/share/doc/syslinux-3.84/menu.txt which was installed on your system when you installed syslinux. Of course, replace 3.84 with the version you have installed.

First example;

<source lang="text">
LABEL next
MENU LABEL ^A) Boot the next device as configured in your BIOS
MENU DEFAULT
localboot
</source>

This example is a good one to always use as the first, and default, option.

The LABEL next defines this option group. The name next has no special meaning, but can be used for more complex setups later. For now, just be sure to keep this name simple with no spaces.

The MENU LABEL ... option control two things. First, it's the text shown to the user. Second, the ^A sets the keyboard key that the user can press to select the option. Whatever character comes immediately after the (caret), A in this example, becomes the mapped key. Please note that the ^ can come anywhere in the title text. Ensure that no two entries have the same character mapped!

The MENU DEFAULT command tells the boot loader to automatically boot this option if the TIMEOUT expires. Only one entry may have this option.

The localboot is a special command the skips the PXE boot loader. Your BIOS should proceed to boot the next device in it's boot order list. Generally this will boot the local hard drive.

Another example, from the cluster.menu file:

<source lang="text">
LABEL f14_an-node01
MENU LABEL ^1) Install AN!Node01; Storage Node 01
KERNEL boot/f14/x86_64/vmlinuz
APPEND initrd=boot/f14/x86_64/initrd.img ks=http://192.168.1.254/f14/x86_64/ks/an-node01.ks ksdevice=eth0
</source>

This is a pretty typical entry. The two new options to note are:

* KERNEL ...

This is the path to the boot kernel used to start the OS or it's installer. Most distributions make use of a small kernel to run up their installer or Live CD/DVD environment. The PXE server will search for the kernel relative to the tftpboot directory. Where to get the proper boot kernel will depend on the distribution you are setting up.

Some examples will be [[#Setting Up The Boot Environment|shown below]].

* APPEND ...

This argument allows you to pass arguments to the KERNEL specified above. Which options can be passed will, again, depend on the distribution you are setting up. In this example, an initrd image is specified, a kickstart script is set and the network device to use to search for the kickstart script are defined.

The [[Setting_Up_a_PXE_Server_in_Fedora#Making_The_Install_Files_For_An_OS_Available|Apache configuration]] section below covers in more detail how to make the kickstart and OS installation files available.

=== Menus ===

Menus are very useful when you have more than a few entries in your PXE server. They allow for creating groups of bootable options based on whatever given criteria works for you.

Let's look at the "cluster" menu from above.

<source lang="text">
MENU BEGIN cluster
MENU TITLE Cluster nodes
LABEL Previous
MENU LABEL ^C) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/cluster.menu
MENU END
</source>

* MENU BEGIN ...

This begins the section and is followed by a name that should (must?) be unique.

* MENU TITLE ...

This is the text shown at the top of the sub-menu once selected.

* LABEL ...

This is the text shown as the first option in the sub-menu that returns to the previous screen.

* MENU LABEL ...

This is the text that the user selects to descend into the sub-menu.

'''Note''': I use the ^ character to enable a letter to select the menu option but it doesn't seem to work. Please let me know if you have further information on how, or if, sub-menu options are selected with a key press.

* MENU EXIT ...

I am not sure, exactly, what this option does.

* MENU SEPARATOR ...

I am not sure, exactly, what this option does.

* MENU INCLUDE ...

This is the path and file to include when building this sub-menu. The path is relative to the tftpboot server_args directory, as set in /etc/xinetd.d/tftp file.

* MENU END

This closes the sub-menu section.

There are other options, though none seem to be well documented at the moment. I got this far thanks to [https://help.ubuntu.com/community/PXEInstallMultiDistro this page]. Any additional information on these and other options would be welcome!

= Setting Up Apache =

[[Image:f13_httpd_default_page_01.png|thumb|right|400px|Default Apache2.2 web page on Fedora 13]]

We will use the Apache web server to make our operating system images available. First, install the server.

<source lang="bash">
yum install httpd
</source>

The default install of Apache2 will work fine for us, so we won't adjust it. Let's start it now.

<source lang="bash">
/etc/init.d/httpd start
</source>

Test access by loading [http://localhost http://localhost] from the PXE server. If that works, then try connecting to the PXE server from another computer using it's [[IP]] address. In either case, you should see a page like the one to the right.

== Setting Up The DocumentRoot ==

Apache has a variable called DocumentRoot in it's main configuration file /etc/httpd/conf/httpd.conf. Anything in or under the directory specified in this variable will be exposed on your web server. The default directory is /var/www/html and this will be where we put our various kickstart files, ISO files and under which we will mount those ISO files.

== The Default Page ==

There is no need to create an index.html page, as the image kickstart scripts and image files will be called directly. Personally though, I like to put together a simple page listing the various installations I've made available on the PXE server. If nothing else, it helps identify the machine when you call it in a browser.

If you decide to create one, please use whatever website tool, text editor or [[Integrated Development Environment|IDE]] you prefer. For reference, below are the very simple, probably not [[W3C]] compliant static [[HTML]] page I use along with a trivial [[CSS]] file.

* [[PXE Server index.html]]
* [[PXE Server common.css]]

Put the index.html (and associated files) directly in the DocumentRoot directory. By default, Apache2 will look for a file named index.html and load it if it is found.

== Making The Install Files For An OS Available ==

At the end of the day, the only thing that matters here is that all of the installation files and kickstart scripts are ''somewhere'' under the DocumentRoot directory. I will describe here the layout that I like, but feel free to adjust it however you see fit.

Further, different operating systems will need to be configured differently. For the purpose of this tutorial, I will be showing how RPM-based [[Linux]] distributions are configured. If you are using different operating systems, you should be able to build on the steps below.

=== Planning Your Directory Structure ===

Under the DocumentRoot directory, we will need to create a set of directories to store the operating system files and associated kickstart scripts in. I've found the following structure works well.

* <DocumentRoot>/<os_name>/<architecture>/[ks|iso|img].

Using Fedora 14 [[x86_64]] with the default Apache DocumentRoot as an example, you would create the following directories. Note that I like to shorten Operating System names to the shortest version practical to save typing. Feel free to use the full names if you prefer verbose [[URL]]s.

* /var/www/html/f14/x86_64/ks: Kickstart scripts go here
* /var/www/html/f14/x86_64/iso: The [http://download.fedoraproject.org/pub/fedora/linux/releases/14/Fedora/x86_64/iso/Fedora-14-x86_64-DVD.iso Fedora-14-x86_64-DVD.iso] file is saved here
* /var/www/html/f14/x86_64/img: The [[ISO]] file above is mounted here.

If you wanted to also make the [[i386]] version of Fedora available, simply replace x86_64 with i386 above. If you want to also make the live version for Fedora 14 x86_64 available, replace f13 with something like f13_live.

=== Fedora 14 x86_64 ===

Let's walk through the actual steps needed to make Fedora 14 x86_64 available on Apache now.

First, create the directory structure.

<source lang="bash">
mkdir -p /var/www/html/f14/x86_64/{iso,img,ks}
</source>

Now, download the ISO if you have not yet done so. If you have, move it into place.

<source lang="bash">
cd /var/www/html/f14/x86_64/iso
wget -c http://download.fedoraproject.org/pub/fedora/linux/releases/14/Fedora/x86_64/iso/Fedora-14-x86_64-DVD.iso
</source>

Now we mount the ISO using a loopback device.

<source lang="bash">
mount -o loop /var/www/html/f14/x86_64/iso/Fedora-14-x86_64-DVD.iso /var/www/html/f14/x86_64/img
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 69G 7.1G 59G 11% /
tmpfs 497M 88K 497M 1% /dev/shm
/dev/sda1 485M 62M 398M 14% /boot
/dev/sdb1 1.4T 678G 629G 52% /media/disk
/var/www/html/f14/x86_64/iso/Fedora-14-x86_64-DVD.iso
3.3G 3.3G 0 100% /var/www/html/f14/x86_64/img
</source>

Lastly, copy any kickstart scripts into the /var/www/html/f14/x86_64/ks/ directory. These scripts will be called by the user after they select an automated install option from the PXE menu.

Here is a generic Fedora 14 x86_64 kickstart I use as a server that you can adapt if you would like. It creates a quite stripped down install.
* [[kickstart_f14_generic_server.ks]]

That's it! You can now use make Fedora 14 available as a PXE boot option in your menu.

== Mounting And Unmounting Many ISOs ==

If you make any real use of your PXE server, you will probably find that you will have a lot of ISO files to mount and unmount. To simplify this task, I like to create a couple of bash scripts called, simply, mount_iso.sh and umount_iso.sh.

These are simple shell scripts meant to mount and unmount all ISOs with one command. You need to be sure that you keep them up to date as you add and remove operating systems.

You will certainly want to adjust these to suit your setup. With that said, here are my scripts that you can use as templates.

* Direct download: [https://alteeve.com/files/pxe/mount_iso.sh mount_iso.sh]

<source lang="bash">
#!/bin/bash
# This is used to mount the various ISO used by the PXE server on this server.

echo -n "Mounting Fedora 14, x86_64... "
mount -o loop /var/www/html/f14/x86_64/iso/Fedora-14-x86_64-DVD.iso /var/www/html/f14/x86_64/img
echo "Done!"

echo -n "Mounting Fedora 14, i386... "
mount -o loop /var/www/html/f14/i386/iso/Fedora-14-i386-DVD.iso /var/www/html/f14/i386/img/
echo "Done!"

echo -n "Mounting Fedora 14 Live DVD, x86_64... "
mount -o loop /var/www/html/f14_live/x86_64/iso/Fedora-14-x86_64-Live-Desktop.iso /var/www/html/f14_live/x86_64/img/
echo "Done!"

echo -n "Mounting Fedora 14 Live DVD, i686... "
mount -o loop /var/www/html/f14_live/i686/iso/Fedora-14-i686-Live-Desktop.iso /var/www/html/f14_live/i686/img/
echo "Done!"
</source>

* Direct download: [https://alteeve.com/files/pxe/umount_iso.sh umount_iso.sh]

<source lang="bash">
#!/bin/bash
# This is used to unmount the various ISO used by the PXE server on this server.

echo -n "Unmounting Fedora 14, x86_64... "
umount /var/www/html/f14/x86_64/img
echo "Done!"

echo -n "Unmounting Fedora 14, i386... "
umount /var/www/html/f14/i386/img/
echo "Done!"

echo -n "Unmounting Fedora 14 Live DVD, x86_64... "
umount /var/www/html/f14_live/x86_64/img/
echo "Done!"

echo -n "Unmounting Fedora 14 Live DVD, i686... "
umount /var/www/html/f14_live/i686/img/
echo "Done!"
</source>

=== Increasing Maximum number of Mountable ISOs ===

By default the loop driver only creates 8 loop back devices. Each mounted [[ISO]] takes one of those and there for limits the total number of ISOs that can be mounted at any one time.

There exist two places to change the maximum number of loop back devices.
The recommend method is to add max_loop option to a new config file in /etc/modprobe.d/.

<source lang="bash">
vim /etc/modprobe.d/loop.conf
</source>
<source lang="text">
options loop max_loop=64
</source>

The second options is to add max_loop=64 to the kernel boot line.

Adjust the number after max_loop= to suit your installations needs.

= Setting Up The Boot Environment =

All boot options, except localboot, need to have a kernel made available via the PXE server. Specifically, these kernels must exist somewhere under the /var/lib/tftpboot directory. What kernel or boot image a given operating system will need depends entirely on that operating system. For this section, we will continue to use Fedora 14 x86_64 as an example.

== Planning Your Directory Structure, Take Two ==

As with [[#Planning Your Directory Structure|Apache]], we're potentially going to copy a lot of boot images and other files. It makes sense to have an organized directory structure to put them all in. As before, I will share what I do, but please feel free to do what you find works best for you.

I like to create a dedicated directory called boot and then mimic the layout from Apache, minus the last set of three directories.

* <tftpboot>/boot/<os_name>/<architecture>/

Using Fedora 14 x86_64 with the default tftpboot install, I will create the following directory.

<source lang="bash">
mkdir -p /var/lib/tftpboot/boot/f14/x86_64
</source>

== The Bootable Kernel ==

Remember in the earlier LABEL f14_an-node01 example there was the APPEND initrd=boot/f14/x86_64/initrd.img ... line? The boot/f14/x86_64/initrd.img is the kernel, in the directory we just created!

The last piece of the puzzle then is; Where does the initrd.img file come from?

With most Linux distributions, there is an isolinux directory in the root of the installation DVD. The files needed to boot the DVD's installer (or "rescue mode") are in this directory. These are the same files we need to boot over PXE. If you have followed this tutorial up to this point, then you should be able to see the contents of this directory in your file browser. My PXE server uses the IP address 192.168.1.254, so I can go to
http://192.168.1.254/f14/x86_64/img/isolinux and see, among other files, initrd.img.

With the directory created and the ISO mounted, all that is left is to copy the contents of the isolinux files.

<source lang="bash">
cp -v /var/www/html/f14/x86_64/img/isolinux/* /var/lib/tftpboot/boot/f14/x86_64/
</source>
<source lang="text">
`/var/www/html/f14/x86_64/img/isolinux/boot.cat' -> `/var/lib/tftpboot/boot/f14/x86_64/boot.cat'
`/var/www/html/f14/x86_64/img/isolinux/boot.msg' -> `/var/lib/tftpboot/boot/f14/x86_64/boot.msg'
`/var/www/html/f14/x86_64/img/isolinux/grub.conf' -> `/var/lib/tftpboot/boot/f14/x86_64/grub.conf'
`/var/www/html/f14/x86_64/img/isolinux/initrd.img' -> `/var/lib/tftpboot/boot/f14/x86_64/initrd.img'
`/var/www/html/f14/x86_64/img/isolinux/isolinux.bin' -> `/var/lib/tftpboot/boot/f14/x86_64/isolinux.bin'
`/var/www/html/f14/x86_64/img/isolinux/isolinux.cfg' -> `/var/lib/tftpboot/boot/f14/x86_64/isolinux.cfg'
`/var/www/html/f14/x86_64/img/isolinux/memtest' -> `/var/lib/tftpboot/boot/f14/x86_64/memtest'
`/var/www/html/f14/x86_64/img/isolinux/splash.jpg' -> `/var/lib/tftpboot/boot/f14/x86_64/splash.jpg'
`/var/www/html/f14/x86_64/img/isolinux/splash.lss' -> `/var/lib/tftpboot/boot/f14/x86_64/splash.lss'
`/var/www/html/f14/x86_64/img/isolinux/syslinux-vesa-splash.jpg' -> `/var/lib/tftpboot/boot/f14/x86_64/syslinux-vesa-splash.jpg'
`/var/www/html/f14/x86_64/img/isolinux/TRANS.TBL' -> `/var/lib/tftpboot/boot/f14/x86_64/TRANS.TBL'
`/var/www/html/f14/x86_64/img/isolinux/vesamenu.c32' -> `/var/lib/tftpboot/boot/f14/x86_64/vesamenu.c32'
`/var/www/html/f14/x86_64/img/isolinux/vmlinuz' -> `/var/lib/tftpboot/boot/f14/x86_64/vmlinuz'
</source>

And we're done!

Now repeat the relevant steps for the rest of the operating systems you want to support via PXE.

= Credits =

References used:
* [http://linux-sxs.org/internet_serving/pxeboot.html Linux-SXS]
* [https://help.ubuntu.com/community/PXEInstallMultiDistro Ubuntu]
* [http://www.c3.hu/docs/oreilly/tcpip/tcpip/appd_03.htm O'Reilly]
* [http://www.linuxdoc.org/HOWTO/CDServer-HOWTO/addloops.html linuxdoc.org]

{{footer}}

Setting Up a PXE Server on an RPM-based OS

2011-09-29T16:56:13Z

SRSullivan: /* The Bootable Kernel */

{{howto_header}}

= About this Tutorial =
'''Dec. 30, 2010''': This tutorial has been confirmed to work on [[RHEL]] 6.0 and Fedora 13+.

'''Nov. 08, 2010''': This is the first draft of the final tutorial. As such, should be fairly safe to follow, though mistakes and omissions may exist. Suggestions for clarification are also appreciated.

This tutorial covers the steps needed to make a [http://en.wikipedia.org/wiki/Preboot_Execution_Environment PXE] server. It will be used for hosting multiple [[Operating Systems]] that can be booted from a machine's network card. The main reason for this setup is to host installation media removing the requirement for have optical drives in machines. It also saves you from having a pile of optical discs kicking around.

== Prerequisite ==

This tutorial assumes that you have a fresh install of [[Fedora]] 13+ and/or RHEL 6.0 and that the machine's eth0 device has been statically set to 192.168.1.10 and the subnet's router is set to 192.168.1.1. This should be easy to adapt to other distributions and network configurations.

== Parts Needed ==

A PXE boot server is fairly strait forward. You need:
* dhcp; This answers a workstation's request for an IP during the boot process.
* tftp-server; This is a PXE compliant FTP server than handles passing the core boot files to the remote machine.
* syslinux; This handles those special boot files that the remote machine needs to boot.
* httpd; Once the boot files start up the remote machine, generally you will tell it to pull the main files from a webserver. This is the Apache webserver that will server that purpose.

= Configuring Needed Services =

== dhcp ==

First you will need a DHCP server. Here is a good little tutorial to follow. Come back once you finish.

* [[DHCP on an RPM-based OS]]

=== Addition to the DHCP Configuration File ===

We need to add a section to the DHCP server configuration file, dhcpd.conf. We need to add an option to the relevant subnet directive(s) indicating where interested clients can go to get the PXE boot configuration. In this example, the PXE server and DHCP server are one in the same, but this is by choice only.

The section to add is below.

<source lang="text">
# These two options tell clients where to go to get the file needed to
# start the boot process.
next-server 192.168.1.254;
filename "pxelinux.0";
</source>

The updated dhcpd.conf file should look something like the example below.

'''Note''': [[EL5]] distributions store this file at /etc/dhcpd.conf.

<source lang="bash">
vim /etc/dhcp/dhcpd.conf
</source>
<source lang="bash">
### Global options
# General domain information
option domain-name "alteeve.com";
option domain-name-servers 192.139.81.117, 192.139.81.1;

# Tell the server that it's authoritive on our network.
authoritive;

# This is required for EL5 operating systems but is optional on EL6 and newer
# Fedoras (F13+, at least). It controls how dynamic DNS updating is handled. In
# our case, we aren't concerned about DDNS so we'll set it to 'none'.
ddns-update-style none;

### Subnet options
subnet 192.168.1.0 netmask 255.255.255.0 {
# This is the DHCP server, but not the actual Internet gateway. So this
# Argument points our clients to the right box.
option routers 192.168.1.254;

# Set our range. This can be whatever you want so long as it fits in
# your netmask.
range 192.168.1.100 192.168.1.220;

# If clients don't ask, make the lease available for the following
# number of seconds. If the client does ask, allow up to this number of
# seconds. 86,400s = 24h.
default-lease-time 86400;
max-lease-time 86400;

# These two options tell clients where to go to get the file needed to
# start the boot process.
next-server 192.168.1.254;
filename "pxelinux.0";
}
</source>

Restart the DHCP server and you're done.

<source lang="bash">
/etc/init.d/dhcpd restart
</source>
<source lang="text">
Starting dhcpd: [ OK ]
</source>

== tftp ==

We will use the 'trivial FTP' program as it is a PXE-compliant FTP program that can transfers the boot files from the PXE server to the client. It runs as an xinetd package, so we will need to make sure it is installed as well. All told, it lives up to it's name as it is quite trivial to setup.

First, install them.

<source lang="bash">
yum install tftp-server xinetd
</source>

Then edit /etc/xinted.d/tftp and change disable = yes to disable = no. The edited file should now look like this:

<source lang="bash">
vim /etc/xinetd.d/tftp
</source>
<source lang="bash">
# default: off
# description: The tftp server serves files using the trivial file transfer \
# protocol. The tftp protocol is often used to boot diskless \
# workstations, download configuration files to network-aware printers, \
# and to start the installation process for some operating systems.
service tftp
{
disable = no
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -s /var/lib/tftpboot
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}
</source>

Being an xinetd service, that is what we need to enable at boot time and then start.

<source lang="bash">
chkconfig dhcpd on
chkconfig xinetd on
/etc/init.d/xinetd restart
</source>
<source lang="text">
Stopping xinetd: [FAILED]
Starting xinetd: [ OK ]
</source>

Done!

= pxelinux =

In the xintd file for tftp was this line:

<source lang="text">
server_args = -s /var/lib/tftpboot
</source>

This determines where the PXE boot files will be setup. Some people like to change this to be /tftpboot, but we'll keep it there to keep things simple. In Fedora, this directory already exists and should be world-readable. If it isn't for some reason, create it and set the permissions to 0755 or 0777, depending on your security requirements.

'''Note''': On [[EL5]], the server_args value is -s /tftpboot. You can leave this as-is and adapt the rest of the tutorial to this directory if you wish. Otherwise, change this to /var/lib/tftpboot. Changing it is recommended as it will facilitate future upgrades or migrations of your configuration.

So now we need to install a package called syslinux.

<source lang="bash">
yum install syslinux
</source>

== Setting Up the Boot Environment ==

Next up, we need to copy a couple files into the tftpboot directory. These files are provided by the syslinux package we installed earlier and can be found in the /usr/share/syslinux/ directory.

The main files are:
* pxelinux.0: This is the actual kernel that is passed to the client to begin the boot process. You'll notice this is the file specified in the dhcpd.conf file earlier in our setup. It boots the client far enough so that it can see the boot menu, if any, and then move on to find the main system to boot. This "main system" could be a boot DVD or a full [[OS]].
* vesamenu.c32: This is the 32-bit [http://syslinux.zytor.com/wiki/index.php/SYSLINUX#COMBOOT_image comboot] file. This enables 32-bit colour images to be used for the boot menu (alpha+rr+gg+bb). This replaces the older 16-colour menu.c32 of earlier version that allow for very basic images.

<source lang="bash">
rsync -av /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
rsync -av /usr/share/syslinux/vesamenu.c32 /var/lib/tftpboot/
</source>

'''Note''': On [[EL5]], only pxelinux.0 exists and it is in /usr/lib/syslinux/. The vesamenu.c32 can be copied from an [[EL6]] machine or downloaded from this server.

<source lang="bash">
rsync -av /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/
wget -c https://alteeve.com/files/pxe/tftpboot/vesamenu.c32 -P /var/lib/tftpboot/
</source>

=== Client Configuration Files ===

Client configuration files will be placed in a new directory under tftpboot called pxelinux.cfg. So to start, we need to create it:

<source lang="bash">
mkdir /var/lib/tftpboot/pxelinux.cfg
</source>

Before we talk about the contents of the configuration files, it is important to understand how the PXE server decides which one to use for a given client.

When a client connects, the PXE server looks at the client's [[MAC]] address and checks to see if there is a matching hyphen-separated configuration file. If that isn't found, it then looks at the client's IP address, as set by the DHCP server. It looks for configuration files matching the [[hexadecimal]] representation of the IP address. For example, If the client was given the IP address 192.168.1.200, the PXE server will start by looking for a configuration file called C0A801C8. If it doesn't find a matching file, it will then knock-off the right-most nibble and check again. It will do this until all the possible file names are checked.

If the PXE server finds no configuration file matching the MAC address or any variant on the IP address, it falls back to a configuration file called default.

Let's show this series using the example of a client with MAC address 00:24:7e:69:6f:0e and having been assigned the IP address 192.168.1.200. The PXE server will then look for the following configuration file names in the following order:

<source lang="text">
/var/lib/tftpboot/pxelinux.cfg/00-24-7E-69-6F-0E
/var/lib/tftpboot/pxelinux.cfg/C0A801C8
/var/lib/tftpboot/pxelinux.cfg/C0A801C
/var/lib/tftpboot/pxelinux.cfg/C0A801
/var/lib/tftpboot/pxelinux.cfg/C0A80
/var/lib/tftpboot/pxelinux.cfg/C0A8
/var/lib/tftpboot/pxelinux.cfg/C0A
/var/lib/tftpboot/pxelinux.cfg/C0
/var/lib/tftpboot/pxelinux.cfg/C
/var/lib/tftpboot/pxelinux.cfg/default
</source>

I've made a little script to convert decimal-type IP addresses into hexadecimal-type specifically to help in naming these configuration files. I am sure there are many others out there.
* [[ip_to_hex]]

=== The Configuration Files ===

All of the possible configuration files can be setup using the same set of options and can be setup in similar ways. There is nothing special about any given configuration file. For this reason, we will cover the contents of the default configuration file only.

Below is the default configuration file we will use. Comments are embedded explaining each option.

* Direct download: /var/lib/tftpboot/pxelinux.cfg/[https://alteeve.com/files/pxe/tftpboot/pxelinux.cfg/default default]
* Direct download: /var/lib/tftpboot/[https://alteeve.com/files/pxe/tftpboot/an-pxe_splash_1024_768.png an-pxe_splash_1024_768.png]

<source lang="bash">
vim /var/lib/tftpboot/pxelinux.cfg/default
</source>
<source lang="perl">
# Use the high-colour menu system. This file, and the low-colour 'menu.c32'
# version, are provided by the syslinux package and can be found in the
# '/var/lib/tftpboot' directory. Copy it to '/var/lib/tftpboot'.
UI vesamenu.c32

# Time out and use the default menu option. Defined as tenths of a second.
TIMEOUT 600

# Prompt the user. Set to '1' to automatically choose the default option. This
# is really meant for files matched to MAC addresses.
PROMPT 0

# Set the boot menu to be 1024x768 with a nice background image. Be careful to
# ensure that all your user's can see this resolution! Default is 640x480.
MENU RESOLUTION 1024 768
# This file must be in or under the '/var/lib/tftpboot' folder.
MENU BACKGROUND an-pxe_splash_1024_768.png

# These do not need to be set. I set them here to show how you can customize or
# localize your PXE server's dialogue.
MENU TITLE AN!PXE Boot Server
# Below, the hash (#) character is replaced with the countdown timer. The
# '{,s}' allows for pluralizing a word and is used when the value is >= '2'.
MENU AUTOBOOT Will boot the next device as configured in your BIOS in # second{,s}.
MENU TABMSG Press the <tab> key to edit the boot parameters of the highlighted option.
MENU NOTABMSG Editing of this option is disabled.

# The following options set the various colours used in the menu. All possible
# options are specified except for F# help options. The colour is expressed as
# two hex characters between '00' and 'ff' for red, green, blue and alpha,
# respectively (#RRGGBBAA).
# Format is: MENU COLOR <Item> <ANSI Seq.> <foreground> <background> <shadow type>
MENU COLOR screen 0 #80ffffff #00000000 std # background colour not covered by the splash image
MENU COLOR border 0 #ffffffff #ee000000 std # The wire-frame border
MENU COLOR title 0 #ffff3f7f #ee000000 std # Menu title text
MENU COLOR sel 0 #ff00dfdf #ee000000 std # Selected menu option
MENU COLOR hotsel 0 #ff7f7fff #ee000000 std # The selected hotkey (set with ^ in MENU LABEL)
MENU COLOR unsel 0 #ffffffff #ee000000 std # Unselected menu options
MENU COLOR hotkey 0 #ff7f7fff #ee000000 std # Unselected hotkeys (set with ^ in MENU LABEL)
MENU COLOR tabmsg 0 #c07f7fff #00000000 std # Tab text
MENU COLOR timeout_msg 0 #8000dfdf #00000000 std # Timout text
MENU COLOR timeout 0 #c0ff3f7f #00000000 std # Timout counter
MENU COLOR disabled 0 #807f7f7f #ee000000 std # Disabled menu options, including SEPARATORs
MENU COLOR cmdmark 0 #c000ffff #ee000000 std # Command line marker - The '> ' on the left when editing an option
MENU COLOR cmdline 0 #c0ffffff #ee000000 std # Command line - The text being edited
# Options below haven't been tested, descriptions may be lacking.
MENU COLOR scrollbar 0 #40000000 #00000000 std # Scroll bar
MENU COLOR pwdborder 0 #80ffffff #20ffffff std # Password box wire-frame border
MENU COLOR pwdheader 0 #80ff8080 #20ffffff std # Password box header
MENU COLOR pwdentry 0 #80ffffff #20ffffff std # Password entry field
MENU COLOR help 0 #c0ffffff #00000000 std # Help text, if set via 'TEXT HELP ... ENDTEXT'

### Now define the menu options

# I feel it is safest to return booting to the client as the first and default
# option. This entry below will do just that.
LABEL next
MENU LABEL ^A) Boot the next device as configured in your BIOS
MENU DEFAULT
localboot

## To keep the menu from getting out of hand, I like to create sub-menus.
## The sub-menus are simply additional files in the same directory as this
## file with one or more boot options or, if you wish, further sub-menus.
#
# This section covers boot options without pre-defined kickstart (or other)
# automation scripts.
MENU BEGIN generic
MENU TITLE Generic installations (as if booting from media)
LABEL Previous
MENU LABEL ^B) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/generic.menu
MENU END

# As I build a lot of clusters, I like to have a section dedicated to boot
# options used to build cluster nodes.
MENU BEGIN cluster
MENU TITLE Cluster nodes
LABEL Previous
MENU LABEL ^C) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/cluster.menu
MENU END

# This section is where I put generic automated install scripts. These
# generally have no specific use and are meant to create installations that
# can be easily adapted or built upon.
MENU BEGIN stock
MENU TITLE Stock installs (generic kickstart-based installs)
LABEL Previous
MENU LABEL ^D) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/stock.menu
MENU END

# Lastly, I like to make "Live" operating systems available. These are very
# useful for diagnostics and recovery work where you need more than a 'rescue'
# boot provides and where you do not want to effect the underlying system OS.
MENU BEGIN live
MENU TITLE Live Distros (bootable without affecting the underlying system)
LABEL Previous
MENU LABEL ^E) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/live.menu
MENU END
</source>

The default file above points to four sub-menu files. How many you use, if you use any at all, is up to you. You can also use any file name you want, or any directory under /var/lib/tftpboot/.

To save space on this page, The sub-menu files mentioned above are their own pages and are linked below. These will hopefully act as useful references for your own sub-menu files, should you wish to use menus.

* [[pxe_generic.menu|generic.menu]]
* [[pxe_cluster.menu|cluster.menu]]
* [[pxe_stock.menu|stock.menu]]
* [[pxe_live.menu|live.menu]]

=== Options ===

Let's now take a look at the various parts. This is an overview only, for a complete list of options please read /usr/share/doc/syslinux-3.84/menu.txt. Update the version number to match your installed version.

Wherever a colour can be specified, the format is #AARRGGBB. That is, alpha (transparency), red, green and blue intensity expressed as a 256 range specified using two [[hexadecimal]] characters per option.

* UI vesamenu.c32

This loads the 32-bit colour [http://syslinux.zytor.com/wiki/index.php/SYSLINUX#COMBOOT_image COMBOOT] image. This allows for the full colour range plus 8-bit alpha (transparency) to be available at boot time. It's what allows for a much more attractive boot menu. To read details on the innards of the COMBOOT format, read /usr/share/doc/syslinux-3.84/comboot.txt. Update the version number to match your installed version.

* TIMEOUT 600

This is the amount of time, in tenths of a second, that the PXE loader will wait before performing the default action. If the user navigates through the menu. the timer will stop and wait for the user to make a selection. To prevent an install from running accidentally, the default option should always be a non-damaging option.

* prompt 0

This tells the PXE server to prompt the user to make a choice. If this is set to 1 then the PXE server will use the default option.

* MENU RESOLUTION 1024 768

This tells the boot loader to run at the set resolution, 1024x768 in this example. The default is to run up as vga (640x480). Be sure when pushing a higher resolution that all of your potential users have machines that support the given resolution.

* MENU BACKGROUND an-pxe_splash_1024_768.png

This is the background image to use. This must exist in or under the /var/lib/tftpboot/ directory. If you want to use a subdirectory, specify it as a relative path. For example, if you want to store images in /var/lib/tftpboot/images/, then this would be set to images/an-pxe_splash_1024_768.png. The format of the image can be JPEG or The format of the image can be JPNG (other image formats may work). The image itself should match the size of the screen.

* MENU TITLE AN!PXE Boot Server
* MENU AUTOBOOT Will boot the next device as configured in your BIOS in # second{,s}.
* MENU TABMSG Press the <tab> key to edit the boot parameters of the highlighted option.
* MENU NOTABMSG Editing of this option is disabled.

These options allow you to customize or localize the text. None need to be specified.

The AUTOBOOT option is a little special to accommodate the count-down. The # will be replaced by the time remaining. The {,s} tells the string to add an s to the end of seconds when the time remaining is greater than one.

* MENU COLOR x

These options control the colour and transparency of the various text-elements, borders and backgrounds. None of them need to be specified and all have sane default values. Most of the available options are shown in the example above.

Each entry is formatted like so:

MENU COLOR <type> <ansi> <foreground> <background> <shadow>

The <type> is the name of the element you are manipulating. The only 'type' not show in the example above is msgXX where XX is a number between 1 and 12. These are used to control "help windows" that can be shown to the user when they press an F[1-12] key.

The <ansi> number(s) tell the boot loader how to format the text using [[ANSI]] code. This is set to 0, "reset", in our example to clear any ANSI formatting. You can specify multiple ANSI codes by separating them with ; (semi-colons).

The <foreground and <background> values are the [[hexadecimal]] notation for the transparency and colour in the format #AARRGGBB as mentioned earlier.

The <shadow> is the type of drop-shadow to render for the <type>. Valid options are:
* none: No shadowing.
* std, standard: Foreground pixels are raised.
* all: both foreground and background pixels are raised.
* rev, reverse: Background pixels are raised.

=== Labels ===

Label blocks are the actual boot options made available within the PXE menu. There are many options available here that are not shown in this example. If you are curious, please read /usr/share/doc/syslinux-3.84/menu.txt which was installed on your system when you installed syslinux. Of course, replace 3.84 with the version you have installed.

First example;

<source lang="text">
LABEL next
MENU LABEL ^A) Boot the next device as configured in your BIOS
MENU DEFAULT
localboot
</source>

This example is a good one to always use as the first, and default, option.

The LABEL next defines this option group. The name next has no special meaning, but can be used for more complex setups later. For now, just be sure to keep this name simple with no spaces.

The MENU LABEL ... option control two things. First, it's the text shown to the user. Second, the ^A sets the keyboard key that the user can press to select the option. Whatever character comes immediately after the (caret), A in this example, becomes the mapped key. Please note that the ^ can come anywhere in the title text. Ensure that no two entries have the same character mapped!

The MENU DEFAULT command tells the boot loader to automatically boot this option if the TIMEOUT expires. Only one entry may have this option.

The localboot is a special command the skips the PXE boot loader. Your BIOS should proceed to boot the next device in it's boot order list. Generally this will boot the local hard drive.

Another example, from the cluster.menu file:

<source lang="text">
LABEL f14_an-node01
MENU LABEL ^1) Install AN!Node01; Storage Node 01
KERNEL boot/f14/x86_64/vmlinuz
APPEND initrd=boot/f14/x86_64/initrd.img ks=http://192.168.1.254/f14/x86_64/ks/an-node01.ks ksdevice=eth0
</source>

This is a pretty typical entry. The two new options to note are:

* KERNEL ...

This is the path to the boot kernel used to start the OS or it's installer. Most distributions make use of a small kernel to run up their installer or Live CD/DVD environment. The PXE server will search for the kernel relative to the tftpboot directory. Where to get the proper boot kernel will depend on the distribution you are setting up.

Some examples will be [[#Setting Up The Boot Environment|shown below]].

* APPEND ...

This argument allows you to pass arguments to the KERNEL specified above. Which options can be passed will, again, depend on the distribution you are setting up. In this example, an initrd image is specified, a kickstart script is set and the network device to use to search for the kickstart script are defined.

The [[Setting_Up_a_PXE_Server_in_Fedora#Making_The_Install_Files_For_An_OS_Available|Apache configuration]] section below covers in more detail how to make the kickstart and OS installation files available.

=== Menus ===

Menus are very useful when you have more than a few entries in your PXE server. They allow for creating groups of bootable options based on whatever given criteria works for you.

Let's look at the "cluster" menu from above.

<source lang="text">
MENU BEGIN cluster
MENU TITLE Cluster nodes
LABEL Previous
MENU LABEL ^C) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/cluster.menu
MENU END
</source>

* MENU BEGIN ...

This begins the section and is followed by a name that should (must?) be unique.

* MENU TITLE ...

This is the text shown at the top of the sub-menu once selected.

* LABEL ...

This is the text shown as the first option in the sub-menu that returns to the previous screen.

* MENU LABEL ...

This is the text that the user selects to descend into the sub-menu.

'''Note''': I use the ^ character to enable a letter to select the menu option but it doesn't seem to work. Please let me know if you have further information on how, or if, sub-menu options are selected with a key press.

* MENU EXIT ...

I am not sure, exactly, what this option does.

* MENU SEPARATOR ...

I am not sure, exactly, what this option does.

* MENU INCLUDE ...

This is the path and file to include when building this sub-menu. The path is relative to the tftpboot server_args directory, as set in /etc/xinetd.d/tftp file.

* MENU END

This closes the sub-menu section.

There are other options, though none seem to be well documented at the moment. I got this far thanks to [https://help.ubuntu.com/community/PXEInstallMultiDistro this page]. Any additional information on these and other options would be welcome!

= Setting Up Apache =

[[Image:f13_httpd_default_page_01.png|thumb|right|400px|Default Apache2.2 web page on Fedora 13]]

We will use the Apache web server to make our operating system images available. First, install the server.

<source lang="bash">
yum install httpd
</source>

The default install of Apache2 will work fine for us, so we won't adjust it. Let's start it now.

<source lang="bash">
/etc/init.d/httpd start
</source>

Test access by loading [http://localhost http://localhost] from the PXE server. If that works, then try connecting to the PXE server from another computer using it's [[IP]] address. In either case, you should see a page like the one to the right.

== Setting Up The DocumentRoot ==

Apache has a variable called DocumentRoot in it's main configuration file /etc/httpd/conf/httpd.conf. Anything in or under the directory specified in this variable will be exposed on your web server. The default directory is /var/www/html and this will be where we put our various kickstart files, ISO files and under which we will mount those ISO files.

== The Default Page ==

There is no need to create an index.html page, as the image kickstart scripts and image files will be called directly. Personally though, I like to put together a simple page listing the various installations I've made available on the PXE server. If nothing else, it helps identify the machine when you call it in a browser.

If you decide to create one, please use whatever website tool, text editor or [[Integrated Development Environment|IDE]] you prefer. For reference, below are the very simple, probably not [[W3C]] compliant static [[HTML]] page I use along with a trivial [[CSS]] file.

* [[PXE Server index.html]]
* [[PXE Server common.css]]

Put the index.html (and associated files) directly in the DocumentRoot directory. By default, Apache2 will look for a file named index.html and load it if it is found.

== Making The Install Files For An OS Available ==

At the end of the day, the only thing that matters here is that all of the installation files and kickstart scripts are ''somewhere'' under the DocumentRoot directory. I will describe here the layout that I like, but feel free to adjust it however you see fit.

Further, different operating systems will need to be configured differently. For the purpose of this tutorial, I will be showing how RPM-based [[Linux]] distributions are configured. If you are using different operating systems, you should be able to build on the steps below.

=== Planning Your Directory Structure ===

Under the DocumentRoot directory, we will need to create a set of directories to store the operating system files and associated kickstart scripts in. I've found the following structure works well.

* <DocumentRoot>/<os_name>/<architecture>/[ks|iso|img].

Using Fedora 14 [[x86_64]] with the default Apache DocumentRoot as an example, you would create the following directories. Note that I like to shorten Operating System names to the shortest version practical to save typing. Feel free to use the full names if you prefer verbose [[URL]]s.

* /var/www/html/f14/x86_64/ks: Kickstart scripts go here
* /var/www/html/f14/x86_64/iso: The [http://download.fedoraproject.org/pub/fedora/linux/releases/14/Fedora/x86_64/iso/Fedora-14-x86_64-DVD.iso Fedora-14-x86_64-DVD.iso] file is saved here
* /var/www/html/f14/x86_64/img: The [[ISO]] file above is mounted here.

If you wanted to also make the [[i386]] version of Fedora available, simply replace x86_64 with i386 above. If you want to also make the live version for Fedora 14 x86_64 available, replace f13 with something like f13_live.

=== Fedora 14 x86_64 ===

Let's walk through the actual steps needed to make Fedora 14 x86_64 available on Apache now.

First, create the directory structure.

<source lang="bash">
mkdir -p /var/www/html/f14/x86_64/{iso,img,ks}
</source>

Now, download the ISO if you have not yet done so. If you have, move it into place.

<source lang="bash">
cd /var/www/html/f14/x86_64/iso
wget -c http://download.fedoraproject.org/pub/fedora/linux/releases/14/Fedora/x86_64/iso/Fedora-14-x86_64-DVD.iso
</source>

Now we mount the ISO using a loopback device.

<source lang="bash">
mount -o loop /var/www/html/f14/x86_64/iso/Fedora-14-x86_64-DVD.iso /var/www/html/f14/x86_64/img
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 69G 7.1G 59G 11% /
tmpfs 497M 88K 497M 1% /dev/shm
/dev/sda1 485M 62M 398M 14% /boot
/dev/sdb1 1.4T 678G 629G 52% /media/disk
/var/www/html/f14/x86_64/iso/Fedora-14-x86_64-DVD.iso
3.3G 3.3G 0 100% /var/www/html/f14/x86_64/img
</source>

Lastly, copy any kickstart scripts into the /var/www/html/f14/x86_64/ks/ directory. These scripts will be called by the user after they select an automated install option from the PXE menu.

Here is a generic Fedora 14 x86_64 kickstart I use as a server that you can adapt if you would like. It creates a quite stripped down install.
* [[kickstart_f14_generic_server.ks]]

That's it! You can now use make Fedora 14 available as a PXE boot option in your menu.

== Mounting And Unmounting Many ISOs ==

If you make any real use of your PXE server, you will probably find that you will have a lot of ISO files to mount and unmount. To simplify this task, I like to create a couple of bash scripts called, simply, mount_iso.sh and umount_iso.sh.

These are simple shell scripts meant to mount and unmount all ISOs with one command. You need to be sure that you keep them up to date as you add and remove operating systems.

You will certainly want to adjust these to suit your setup. With that said, here are my scripts that you can use as templates.

* Direct download: [https://alteeve.com/files/pxe/mount_iso.sh mount_iso.sh]

<source lang="bash">
#!/bin/bash
# This is used to mount the various ISO used by the PXE server on this server.

echo -n "Mounting Fedora 14, x86_64... "
mount -o loop /var/www/html/f14/x86_64/iso/Fedora-14-x86_64-DVD.iso /var/www/html/f14/x86_64/img
echo "Done!"

echo -n "Mounting Fedora 14, i386... "
mount -o loop /var/www/html/f14/i386/iso/Fedora-14-i386-DVD.iso /var/www/html/f14/i386/img/
echo "Done!"

echo -n "Mounting Fedora 14 Live DVD, x86_64... "
mount -o loop /var/www/html/f14_live/x86_64/iso/Fedora-14-x86_64-Live-Desktop.iso /var/www/html/f14_live/x86_64/img/
echo "Done!"

echo -n "Mounting Fedora 14 Live DVD, i686... "
mount -o loop /var/www/html/f14_live/i686/iso/Fedora-14-i686-Live-Desktop.iso /var/www/html/f14_live/i686/img/
echo "Done!"
</source>

* Direct download: [https://alteeve.com/files/pxe/umount_iso.sh umount_iso.sh]

<source lang="bash">
#!/bin/bash
# This is used to unmount the various ISO used by the PXE server on this server.

echo -n "Unmounting Fedora 14, x86_64... "
umount /var/www/html/f14/x86_64/img
echo "Done!"

echo -n "Unmounting Fedora 14, i386... "
umount /var/www/html/f14/i386/img/
echo "Done!"

echo -n "Unmounting Fedora 14 Live DVD, x86_64... "
umount /var/www/html/f14_live/x86_64/img/
echo "Done!"

echo -n "Unmounting Fedora 14 Live DVD, i686... "
umount /var/www/html/f14_live/i686/img/
echo "Done!"
</source>

=== Increasing Maximum number of Mountable ISOs ===

By default the loop driver only creates 8 loop back devices. Each mounted [[ISO]] takes one of those and there for limits the total number of ISOs that can be mounted at any one time.

There exist two places to change the maximum number of loop back devices.
The recommend method is to add max_loop option to a new config file in /etc/modprobe.d/.

<source lang="bash">
vim /etc/modprobe.d/loop.conf
</source>
<source lang="text">
options loop max_loop=64
</source>

The second options is to add max_loop=64 to the kernel boot line.

Adjust the number after max_loop= to suit your installations needs.

= Setting Up The Boot Environment =

All boot options, except localboot, need to have a kernel made available via the PXE server. Specifically, these kernels must exist somewhere under the /var/lib/tftpboot directory. What kernel or boot image a given operating system will need depends entirely on that operating system. For this section, we will continue to use Fedora 14 x86_64 as an example.

== Planning Your Directory Structure, Take Two ==

As with [[#Planning Your Directory Structure|Apache]], we're potentially going to copy a lot of boot images and other files. It makes sense to have an organized directory structure to put them all in. As before, I will share what I do, but please feel free to do what you find works best for you.

I like to create a dedicated directory called boot and then mimic the layout from Apache, minus the last set of three directories.

* <tftpboot>/boot/<os_name>/<architecture>/

Using Fedora 14 x86_64 with the default tftpboot install, I will create the following directory.

<source lang="bash">
mkdir -p /var/lib/tftpboot/boot/f14/x86_64
</source>

== The Bootable Kernel ==

Remember in the earlier LABEL f14_an-node01 example there was the APPEND initrd=boot/f14/x86_64/initrd.img ... line? The boot/f14/x86_64/initrd.img is the kernel, in the directory we just created!

The last piece of the puzzle then is; Where does the initrd.img file come from?

With most Linux distributions, there is an isolinux directory in the root of the installation DVD. The files needed to boot the DVD's installer (or "rescue mode") are in this directory. These are the same files we need to boot over PXE. If you have followed this tutorial up to this point, then you should be able to see the contents of this directory in your file browser. My PXE server uses the IP address 192.168.1.254, so I can go to
http://192.168.1.254/f14/x86_64/img/isolinux and see, among other files, initrd.img.

With the directory created and the ISO mounted, all that is left the make the contents of the isolinux files avaliable in the tftp /boot/ directory.

<source lang="bash">
ln -s /var/www/html/f14/x86_64/img/isolinux/ /var/lib/tftpboot/boot/f14/x86_64/
</source>

And we're done!

'''Note:'''

The Reason we use a symlink is so that as the img directory is mounted with an updated ISO (eg. CentOS 5.6 instead of 5.5), the boot directory will always have the correct up to date files.

Now repeat the relevant steps for the rest of the operating systems you want to support via PXE.

= Credits =

References used:
* [http://linux-sxs.org/internet_serving/pxeboot.html Linux-SXS]
* [https://help.ubuntu.com/community/PXEInstallMultiDistro Ubuntu]
* [http://www.c3.hu/docs/oreilly/tcpip/tcpip/appd_03.htm O'Reilly]
* [http://www.linuxdoc.org/HOWTO/CDServer-HOWTO/addloops.html linuxdoc.org]

{{footer}}

Setting Up a PXE Server on an RPM-based OS

2011-05-31T20:40:51Z

SRSullivan: Added documenation on increasing maximum number of loop devices. Updates to Header to induce Section Table.

{{howto_header}}

= About this Tutorial =
'''Dec. 30, 2010''': This tutorial has been confirmed to work on [[RHEL]] 6.0 and Fedora 13+.

'''Nov. 08, 2010''': This is the first draft of the final tutorial. As such, should be fairly safe to follow, though mistakes and omissions may exist. Suggestions for clarification are also appreciated.

This tutorial covers the steps needed to make a [http://en.wikipedia.org/wiki/Preboot_Execution_Environment PXE] server. It will be used for hosting multiple [[Operating Systems]] that can be booted from a machine's network card. The main reason for this setup is to host installation media removing the requirement for have optical drives in machines. It also saves you from having a pile of optical discs kicking around.

== Prerequisite ==

This tutorial assumes that you have a fresh install of [[Fedora]] 13+ and/or RHEL 6.0 and that the machine's eth0 device has been statically set to 192.168.1.10 and the subnet's router is set to 192.168.1.1. This should be easy to adapt to other distributions and network configurations.

== Parts Needed ==

A PXE boot server is fairly strait forward. You need:
* dhcp; This answers a workstation's request for an IP during the boot process.
* tftp-server; This is a PXE compliant FTP server than handles passing the core boot files to the remote machine.
* syslinux; This handles those special boot files that the remote machine needs to boot.
* httpd; Once the boot files start up the remote machine, generally you will tell it to pull the main files from a webserver. This is the Apache webserver that will server that purpose.

= Configuring Needed Services =

== dhcp ==

First you will need a DHCP server. Here is a good little tutorial to follow. Come back once you finish.

* [[DHCP on an RPM-based OS]]

=== Addition to the DHCP Configuration File ===

We need to add a section to the DHCP server configuration file, dhcpd.conf. We need to add an option to the relevant subnet directive(s) indicating where interested clients can go to get the PXE boot configuration. In this example, the PXE server and DHCP server are one in the same, but this is by choice only.

The section to add is below.

<source lang="text">
# These two options tell clients where to go to get the file needed to
# start the boot process.
next-server 192.168.1.254;
filename "pxelinux.0";
</source>

The updated dhcpd.conf file should look something like the example below.

'''Note''': [[EL5]] distributions store this file at /etc/dhcpd.conf.

<source lang="bash">
vim /etc/dhcp/dhcpd.conf
</source>
<source lang="bash">
### Global options
# General domain information
option domain-name "alteeve.com";
option domain-name-servers 192.139.81.117, 192.139.81.1;

# Tell the server that it's authoritive on our network.
authoritive;

# This is required for EL5 operating systems but is optional on EL6 and newer
# Fedoras (F13+, at least). It controls how dynamic DNS updating is handled. In
# our case, we aren't concerned about DDNS so we'll set it to 'none'.
ddns-update-style none;

### Subnet options
subnet 192.168.1.0 netmask 255.255.255.0 {
# This is the DHCP server, but not the actual Internet gateway. So this
# Argument points our clients to the right box.
option routers 192.168.1.254;

# Set our range. This can be whatever you want so long as it fits in
# your netmask.
range 192.168.1.100 192.168.1.220;

# If clients don't ask, make the lease available for the following
# number of seconds. If the client does ask, allow up to this number of
# seconds. 86,400s = 24h.
default-lease-time 86400;
max-lease-time 86400;

# These two options tell clients where to go to get the file needed to
# start the boot process.
next-server 192.168.1.254;
filename "pxelinux.0";
}
</source>

Restart the DHCP server and you're done.

<source lang="bash">
/etc/init.d/dhcpd restart
</source>
<source lang="text">
Starting dhcpd: [ OK ]
</source>

== tftp ==

We will use the 'trivial FTP' program as it is a PXE-compliant FTP program that can transfers the boot files from the PXE server to the client. It runs as an xinetd package, so we will need to make sure it is installed as well. All told, it lives up to it's name as it is quite trivial to setup.

First, install them.

<source lang="bash">
yum install tftp-server xinetd
</source>

Then edit /etc/xinted.d/tftp and change disable = yes to disable = no. The edited file should now look like this:

<source lang="bash">
vim /etc/xinetd.d/tftp
</source>
<source lang="bash">
# default: off
# description: The tftp server serves files using the trivial file transfer \
# protocol. The tftp protocol is often used to boot diskless \
# workstations, download configuration files to network-aware printers, \
# and to start the installation process for some operating systems.
service tftp
{
disable = no
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -s /var/lib/tftpboot
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}
</source>

Being an xinetd service, that is what we need to enable at boot time and then start.

<source lang="bash">
chkconfig dhcpd on
chkconfig xinetd on
/etc/init.d/xinetd restart
</source>
<source lang="text">
Stopping xinetd: [FAILED]
Starting xinetd: [ OK ]
</source>

Done!

= pxelinux =

In the xintd file for tftp was this line:

<source lang="text">
server_args = -s /var/lib/tftpboot
</source>

This determines where the PXE boot files will be setup. Some people like to change this to be /tftpboot, but we'll keep it there to keep things simple. In Fedora, this directory already exists and should be world-readable. If it isn't for some reason, create it and set the permissions to 0755 or 0777, depending on your security requirements.

'''Note''': On [[EL5]], the server_args value is -s /tftpboot. You can leave this as-is and adapt the rest of the tutorial to this directory if you wish. Otherwise, change this to /var/lib/tftpboot. Changing it is recommended as it will facilitate future upgrades or migrations of your configuration.

So now we need to install a package called syslinux.

<source lang="bash">
yum install syslinux
</source>

== Setting Up the Boot Environment ==

Next up, we need to copy a couple files into the tftpboot directory. These files are provided by the syslinux package we installed earlier and can be found in the /usr/share/syslinux/ directory.

The main files are:
* pxelinux.0: This is the actual kernel that is passed to the client to begin the boot process. You'll notice this is the file specified in the dhcpd.conf file earlier in our setup. It boots the client far enough so that it can see the boot menu, if any, and then move on to find the main system to boot. This "main system" could be a boot DVD or a full [[OS]].
* vesamenu.c32: This is the 32-bit [http://syslinux.zytor.com/wiki/index.php/SYSLINUX#COMBOOT_image comboot] file. This enables 32-bit colour images to be used for the boot menu (alpha+rr+gg+bb). This replaces the older 16-colour menu.c32 of earlier version that allow for very basic images.

<source lang="bash">
rsync -av /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
rsync -av /usr/share/syslinux/vesamenu.c32 /var/lib/tftpboot/
</source>

'''Note''': On [[EL5]], only pxelinux.0 exists and it is in /usr/lib/syslinux/. The vesamenu.c32 can be copied from an [[EL6]] machine or downloaded from this server.

<source lang="bash">
rsync -av /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/
wget -c https://alteeve.com/files/pxe/tftpboot/vesamenu.c32 -P /var/lib/tftpboot/
</source>

=== Client Configuration Files ===

Client configuration files will be placed in a new directory under tftpboot called pxelinux.cfg. So to start, we need to create it:

<source lang="bash">
mkdir /var/lib/tftpboot/pxelinux.cfg
</source>

Before we talk about the contents of the configuration files, it is important to understand how the PXE server decides which one to use for a given client.

When a client connects, the PXE server looks at the client's [[MAC]] address and checks to see if there is a matching hyphen-separated configuration file. If that isn't found, it then looks at the client's IP address, as set by the DHCP server. It looks for configuration files matching the [[hexadecimal]] representation of the IP address. For example, If the client was given the IP address 192.168.1.200, the PXE server will start by looking for a configuration file called C0A801C8. If it doesn't find a matching file, it will then knock-off the right-most nibble and check again. It will do this until all the possible file names are checked.

If the PXE server finds no configuration file matching the MAC address or any variant on the IP address, it falls back to a configuration file called default.

Let's show this series using the example of a client with MAC address 00:24:7e:69:6f:0e and having been assigned the IP address 192.168.1.200. The PXE server will then look for the following configuration file names in the following order:

<source lang="text">
/var/lib/tftpboot/pxelinux.cfg/00-24-7E-69-6F-0E
/var/lib/tftpboot/pxelinux.cfg/C0A801C8
/var/lib/tftpboot/pxelinux.cfg/C0A801C
/var/lib/tftpboot/pxelinux.cfg/C0A801
/var/lib/tftpboot/pxelinux.cfg/C0A80
/var/lib/tftpboot/pxelinux.cfg/C0A8
/var/lib/tftpboot/pxelinux.cfg/C0A
/var/lib/tftpboot/pxelinux.cfg/C0
/var/lib/tftpboot/pxelinux.cfg/C
/var/lib/tftpboot/pxelinux.cfg/default
</source>

I've made a little script to convert decimal-type IP addresses into hexadecimal-type specifically to help in naming these configuration files. I am sure there are many others out there.
* [[ip_to_hex]]

=== The Configuration Files ===

All of the possible configuration files can be setup using the same set of options and can be setup in similar ways. There is nothing special about any given configuration file. For this reason, we will cover the contents of the default configuration file only.

Below is the default configuration file we will use. Comments are embedded explaining each option.

* Direct download: /var/lib/tftpboot/pxelinux.cfg/[https://alteeve.com/files/pxe/tftpboot/pxelinux.cfg/default default]
* Direct download: /var/lib/tftpboot/[https://alteeve.com/files/pxe/tftpboot/an-pxe_splash_1024_768.png an-pxe_splash_1024_768.png]

<source lang="bash">
vim /var/lib/tftpboot/pxelinux.cfg/default
</source>
<source lang="perl">
# Use the high-colour menu system. This file, and the low-colour 'menu.c32'
# version, are provided by the syslinux package and can be found in the
# '/var/lib/tftpboot' directory. Copy it to '/var/lib/tftpboot'.
UI vesamenu.c32

# Time out and use the default menu option. Defined as tenths of a second.
TIMEOUT 600

# Prompt the user. Set to '1' to automatically choose the default option. This
# is really meant for files matched to MAC addresses.
PROMPT 0

# Set the boot menu to be 1024x768 with a nice background image. Be careful to
# ensure that all your user's can see this resolution! Default is 640x480.
MENU RESOLUTION 1024 768
# This file must be in or under the '/var/lib/tftpboot' folder.
MENU BACKGROUND an-pxe_splash_1024_768.png

# These do not need to be set. I set them here to show how you can customize or
# localize your PXE server's dialogue.
MENU TITLE AN!PXE Boot Server
# Below, the hash (#) character is replaced with the countdown timer. The
# '{,s}' allows for pluralizing a word and is used when the value is >= '2'.
MENU AUTOBOOT Will boot the next device as configured in your BIOS in # second{,s}.
MENU TABMSG Press the <tab> key to edit the boot parameters of the highlighted option.
MENU NOTABMSG Editing of this option is disabled.

# The following options set the various colours used in the menu. All possible
# options are specified except for F# help options. The colour is expressed as
# two hex characters between '00' and 'ff' for red, green, blue and alpha,
# respectively (#RRGGBBAA).
# Format is: MENU COLOR <Item> <ANSI Seq.> <foreground> <background> <shadow type>
MENU COLOR screen 0 #80ffffff #00000000 std # background colour not covered by the splash image
MENU COLOR border 0 #ffffffff #ee000000 std # The wire-frame border
MENU COLOR title 0 #ffff3f7f #ee000000 std # Menu title text
MENU COLOR sel 0 #ff00dfdf #ee000000 std # Selected menu option
MENU COLOR hotsel 0 #ff7f7fff #ee000000 std # The selected hotkey (set with ^ in MENU LABEL)
MENU COLOR unsel 0 #ffffffff #ee000000 std # Unselected menu options
MENU COLOR hotkey 0 #ff7f7fff #ee000000 std # Unselected hotkeys (set with ^ in MENU LABEL)
MENU COLOR tabmsg 0 #c07f7fff #00000000 std # Tab text
MENU COLOR timeout_msg 0 #8000dfdf #00000000 std # Timout text
MENU COLOR timeout 0 #c0ff3f7f #00000000 std # Timout counter
MENU COLOR disabled 0 #807f7f7f #ee000000 std # Disabled menu options, including SEPARATORs
MENU COLOR cmdmark 0 #c000ffff #ee000000 std # Command line marker - The '> ' on the left when editing an option
MENU COLOR cmdline 0 #c0ffffff #ee000000 std # Command line - The text being edited
# Options below haven't been tested, descriptions may be lacking.
MENU COLOR scrollbar 0 #40000000 #00000000 std # Scroll bar
MENU COLOR pwdborder 0 #80ffffff #20ffffff std # Password box wire-frame border
MENU COLOR pwdheader 0 #80ff8080 #20ffffff std # Password box header
MENU COLOR pwdentry 0 #80ffffff #20ffffff std # Password entry field
MENU COLOR help 0 #c0ffffff #00000000 std # Help text, if set via 'TEXT HELP ... ENDTEXT'

### Now define the menu options

# I feel it is safest to return booting to the client as the first and default
# option. This entry below will do just that.
LABEL next
MENU LABEL ^A) Boot the next device as configured in your BIOS
MENU DEFAULT
localboot

## To keep the menu from getting out of hand, I like to create sub-menus.
## The sub-menus are simply additional files in the same directory as this
## file with one or more boot options or, if you wish, further sub-menus.
#
# This section covers boot options without pre-defined kickstart (or other)
# automation scripts.
MENU BEGIN generic
MENU TITLE Generic installations (as if booting from media)
LABEL Previous
MENU LABEL ^B) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/generic.menu
MENU END

# As I build a lot of clusters, I like to have a section dedicated to boot
# options used to build cluster nodes.
MENU BEGIN cluster
MENU TITLE Cluster nodes
LABEL Previous
MENU LABEL ^C) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/cluster.menu
MENU END

# This section is where I put generic automated install scripts. These
# generally have no specific use and are meant to create installations that
# can be easily adapted or built upon.
MENU BEGIN stock
MENU TITLE Stock installs (generic kickstart-based installs)
LABEL Previous
MENU LABEL ^D) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/stock.menu
MENU END

# Lastly, I like to make "Live" operating systems available. These are very
# useful for diagnostics and recovery work where you need more than a 'rescue'
# boot provides and where you do not want to effect the underlying system OS.
MENU BEGIN live
MENU TITLE Live Distros (bootable without affecting the underlying system)
LABEL Previous
MENU LABEL ^E) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/live.menu
MENU END
</source>

The default file above points to four sub-menu files. How many you use, if you use any at all, is up to you. You can also use any file name you want, or any directory under /var/lib/tftpboot/.

To save space on this page, The sub-menu files mentioned above are their own pages and are linked below. These will hopefully act as useful references for your own sub-menu files, should you wish to use menus.

* [[pxe_generic.menu|generic.menu]]
* [[pxe_cluster.menu|cluster.menu]]
* [[pxe_stock.menu|stock.menu]]
* [[pxe_live.menu|live.menu]]

=== Options ===

Let's now take a look at the various parts. This is an overview only, for a complete list of options please read /usr/share/doc/syslinux-3.84/menu.txt. Update the version number to match your installed version.

Wherever a colour can be specified, the format is #AARRGGBB. That is, alpha (transparency), red, green and blue intensity expressed as a 256 range specified using two [[hexadecimal]] characters per option.

* UI vesamenu.c32

This loads the 32-bit colour [http://syslinux.zytor.com/wiki/index.php/SYSLINUX#COMBOOT_image COMBOOT] image. This allows for the full colour range plus 8-bit alpha (transparency) to be available at boot time. It's what allows for a much more attractive boot menu. To read details on the innards of the COMBOOT format, read /usr/share/doc/syslinux-3.84/comboot.txt. Update the version number to match your installed version.

* TIMEOUT 600

This is the amount of time, in tenths of a second, that the PXE loader will wait before performing the default action. If the user navigates through the menu. the timer will stop and wait for the user to make a selection. To prevent an install from running accidentally, the default option should always be a non-damaging option.

* prompt 0

This tells the PXE server to prompt the user to make a choice. If this is set to 1 then the PXE server will use the default option.

* MENU RESOLUTION 1024 768

This tells the boot loader to run at the set resolution, 1024x768 in this example. The default is to run up as vga (640x480). Be sure when pushing a higher resolution that all of your potential users have machines that support the given resolution.

* MENU BACKGROUND an-pxe_splash_1024_768.png

This is the background image to use. This must exist in or under the /var/lib/tftpboot/ directory. If you want to use a subdirectory, specify it as a relative path. For example, if you want to store images in /var/lib/tftpboot/images/, then this would be set to images/an-pxe_splash_1024_768.png. The format of the image can be JPEG or The format of the image can be JPNG (other image formats may work). The image itself should match the size of the screen.

* MENU TITLE AN!PXE Boot Server
* MENU AUTOBOOT Will boot the next device as configured in your BIOS in # second{,s}.
* MENU TABMSG Press the <tab> key to edit the boot parameters of the highlighted option.
* MENU NOTABMSG Editing of this option is disabled.

These options allow you to customize or localize the text. None need to be specified.

The AUTOBOOT option is a little special to accommodate the count-down. The # will be replaced by the time remaining. The {,s} tells the string to add an s to the end of seconds when the time remaining is greater than one.

* MENU COLOR x

These options control the colour and transparency of the various text-elements, borders and backgrounds. None of them need to be specified and all have sane default values. Most of the available options are shown in the example above.

Each entry is formatted like so:

MENU COLOR <type> <ansi> <foreground> <background> <shadow>

The <type> is the name of the element you are manipulating. The only 'type' not show in the example above is msgXX where XX is a number between 1 and 12. These are used to control "help windows" that can be shown to the user when they press an F[1-12] key.

The <ansi> number(s) tell the boot loader how to format the text using [[ANSI]] code. This is set to 0, "reset", in our example to clear any ANSI formatting. You can specify multiple ANSI codes by separating them with ; (semi-colons).

The <foreground and <background> values are the [[hexadecimal]] notation for the transparency and colour in the format #AARRGGBB as mentioned earlier.

The <shadow> is the type of drop-shadow to render for the <type>. Valid options are:
* none: No shadowing.
* std, standard: Foreground pixels are raised.
* all: both foreground and background pixels are raised.
* rev, reverse: Background pixels are raised.

=== Labels ===

Label blocks are the actual boot options made available within the PXE menu. There are many options available here that are not shown in this example. If you are curious, please read /usr/share/doc/syslinux-3.84/menu.txt which was installed on your system when you installed syslinux. Of course, replace 3.84 with the version you have installed.

First example;

<source lang="text">
LABEL next
MENU LABEL ^A) Boot the next device as configured in your BIOS
MENU DEFAULT
localboot
</source>

This example is a good one to always use as the first, and default, option.

The LABEL next defines this option group. The name next has no special meaning, but can be used for more complex setups later. For now, just be sure to keep this name simple with no spaces.

The MENU LABEL ... option control two things. First, it's the text shown to the user. Second, the ^A sets the keyboard key that the user can press to select the option. Whatever character comes immediately after the (caret), A in this example, becomes the mapped key. Please note that the ^ can come anywhere in the title text. Ensure that no two entries have the same character mapped!

The MENU DEFAULT command tells the boot loader to automatically boot this option if the TIMEOUT expires. Only one entry may have this option.

The localboot is a special command the skips the PXE boot loader. Your BIOS should proceed to boot the next device in it's boot order list. Generally this will boot the local hard drive.

Another example, from the cluster.menu file:

<source lang="text">
LABEL f14_an-node01
MENU LABEL ^1) Install AN!Node01; Storage Node 01
KERNEL boot/f14/x86_64/vmlinuz
APPEND initrd=boot/f14/x86_64/initrd.img ks=http://192.168.1.254/f14/x86_64/ks/an-node01.ks ksdevice=eth0
</source>

This is a pretty typical entry. The two new options to note are:

* KERNEL ...

This is the path to the boot kernel used to start the OS or it's installer. Most distributions make use of a small kernel to run up their installer or Live CD/DVD environment. The PXE server will search for the kernel relative to the tftpboot directory. Where to get the proper boot kernel will depend on the distribution you are setting up.

Some examples will be [[#Setting Up The Boot Environment|shown below]].

* APPEND ...

This argument allows you to pass arguments to the KERNEL specified above. Which options can be passed will, again, depend on the distribution you are setting up. In this example, an initrd image is specified, a kickstart script is set and the network device to use to search for the kickstart script are defined.

The [[Setting_Up_a_PXE_Server_in_Fedora#Making_The_Install_Files_For_An_OS_Available|Apache configuration]] section below covers in more detail how to make the kickstart and OS installation files available.

=== Menus ===

Menus are very useful when you have more than a few entries in your PXE server. They allow for creating groups of bootable options based on whatever given criteria works for you.

Let's look at the "cluster" menu from above.

<source lang="text">
MENU BEGIN cluster
MENU TITLE Cluster nodes
LABEL Previous
MENU LABEL ^C) Previous Menu
MENU EXIT
MENU SEPARATOR
MENU INCLUDE pxelinux.cfg/cluster.menu
MENU END
</source>

* MENU BEGIN ...

This begins the section and is followed by a name that should (must?) be unique.

* MENU TITLE ...

This is the text shown at the top of the sub-menu once selected.

* LABEL ...

This is the text shown as the first option in the sub-menu that returns to the previous screen.

* MENU LABEL ...

This is the text that the user selects to descend into the sub-menu.

'''Note''': I use the ^ character to enable a letter to select the menu option but it doesn't seem to work. Please let me know if you have further information on how, or if, sub-menu options are selected with a key press.

* MENU EXIT ...

I am not sure, exactly, what this option does.

* MENU SEPARATOR ...

I am not sure, exactly, what this option does.

* MENU INCLUDE ...

This is the path and file to include when building this sub-menu. The path is relative to the tftpboot server_args directory, as set in /etc/xinetd.d/tftp file.

* MENU END

This closes the sub-menu section.

There are other options, though none seem to be well documented at the moment. I got this far thanks to [https://help.ubuntu.com/community/PXEInstallMultiDistro this page]. Any additional information on these and other options would be welcome!

= Setting Up Apache =

[[Image:f13_httpd_default_page_01.png|thumb|right|400px|Default Apache2.2 web page on Fedora 13]]

We will use the Apache web server to make our operating system images available. First, install the server.

<source lang="bash">
yum install httpd
</source>

The default install of Apache2 will work fine for us, so we won't adjust it. Let's start it now.

<source lang="bash">
/etc/init.d/httpd start
</source>

Test access by loading [http://localhost http://localhost] from the PXE server. If that works, then try connecting to the PXE server from another computer using it's [[IP]] address. In either case, you should see a page like the one to the right.

== Setting Up The DocumentRoot ==

Apache has a variable called DocumentRoot in it's main configuration file /etc/httpd/conf/httpd.conf. Anything in or under the directory specified in this variable will be exposed on your web server. The default directory is /var/www/html and this will be where we put our various kickstart files, ISO files and under which we will mount those ISO files.

== The Default Page ==

There is no need to create an index.html page, as the image kickstart scripts and image files will be called directly. Personally though, I like to put together a simple page listing the various installations I've made available on the PXE server. If nothing else, it helps identify the machine when you call it in a browser.

If you decide to create one, please use whatever website tool, text editor or [[Integrated Development Environment|IDE]] you prefer. For reference, below are the very simple, probably not [[W3C]] compliant static [[HTML]] page I use along with a trivial [[CSS]] file.

* [[PXE Server index.html]]
* [[PXE Server common.css]]

Put the index.html (and associated files) directly in the DocumentRoot directory. By default, Apache2 will look for a file named index.html and load it if it is found.

== Making The Install Files For An OS Available ==

At the end of the day, the only thing that matters here is that all of the installation files and kickstart scripts are ''somewhere'' under the DocumentRoot directory. I will describe here the layout that I like, but feel free to adjust it however you see fit.

Further, different operating systems will need to be configured differently. For the purpose of this tutorial, I will be showing how RPM-based [[Linux]] distributions are configured. If you are using different operating systems, you should be able to build on the steps below.

=== Planning Your Directory Structure ===

Under the DocumentRoot directory, we will need to create a set of directories to store the operating system files and associated kickstart scripts in. I've found the following structure works well.

* <DocumentRoot>/<os_name>/<architecture>/[ks|iso|img].

Using Fedora 14 [[x86_64]] with the default Apache DocumentRoot as an example, you would create the following directories. Note that I like to shorten Operating System names to the shortest version practical to save typing. Feel free to use the full names if you prefer verbose [[URL]]s.

* /var/www/html/f14/x86_64/ks: Kickstart scripts go here
* /var/www/html/f14/x86_64/iso: The [http://download.fedoraproject.org/pub/fedora/linux/releases/14/Fedora/x86_64/iso/Fedora-14-x86_64-DVD.iso Fedora-14-x86_64-DVD.iso] file is saved here
* /var/www/html/f14/x86_64/img: The [[ISO]] file above is mounted here.

If you wanted to also make the [[i386]] version of Fedora available, simply replace x86_64 with i386 above. If you want to also make the live version for Fedora 14 x86_64 available, replace f13 with something like f13_live.

=== Fedora 14 x86_64 ===

Let's walk through the actual steps needed to make Fedora 14 x86_64 available on Apache now.

First, create the directory structure.

<source lang="bash">
mkdir -p /var/www/html/f14/x86_64/{iso,img,ks}
</source>

Now, download the ISO if you have not yet done so. If you have, move it into place.

<source lang="bash">
cd /var/www/html/f14/x86_64/iso
wget -c http://download.fedoraproject.org/pub/fedora/linux/releases/14/Fedora/x86_64/iso/Fedora-14-x86_64-DVD.iso
</source>

Now we mount the ISO using a loopback device.

<source lang="bash">
mount -o loop /var/www/html/f14/x86_64/iso/Fedora-14-x86_64-DVD.iso /var/www/html/f14/x86_64/img
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 69G 7.1G 59G 11% /
tmpfs 497M 88K 497M 1% /dev/shm
/dev/sda1 485M 62M 398M 14% /boot
/dev/sdb1 1.4T 678G 629G 52% /media/disk
/var/www/html/f14/x86_64/iso/Fedora-14-x86_64-DVD.iso
3.3G 3.3G 0 100% /var/www/html/f14/x86_64/img
</source>

Lastly, copy any kickstart scripts into the /var/www/html/f14/x86_64/ks/ directory. These scripts will be called by the user after they select an automated install option from the PXE menu.

Here is a generic Fedora 14 x86_64 kickstart I use as a server that you can adapt if you would like. It creates a quite stripped down install.
* [[kickstart_f14_generic_server.ks]]

That's it! You can now use make Fedora 14 available as a PXE boot option in your menu.

== Mounting And Unmounting Many ISOs ==

If you make any real use of your PXE server, you will probably find that you will have a lot of ISO files to mount and unmount. To simplify this task, I like to create a couple of bash scripts called, simply, mount_iso.sh and umount_iso.sh.

These are simple shell scripts meant to mount and unmount all ISOs with one command. You need to be sure that you keep them up to date as you add and remove operating systems.

You will certainly want to adjust these to suit your setup. With that said, here are my scripts that you can use as templates.

* Direct download: [https://alteeve.com/files/pxe/mount_iso.sh mount_iso.sh]

<source lang="bash">
#!/bin/bash
# This is used to mount the various ISO used by the PXE server on this server.

echo -n "Mounting Fedora 14, x86_64... "
mount -o loop /var/www/html/f14/x86_64/iso/Fedora-14-x86_64-DVD.iso /var/www/html/f14/x86_64/img
echo "Done!"

echo -n "Mounting Fedora 14, i386... "
mount -o loop /var/www/html/f14/i386/iso/Fedora-14-i386-DVD.iso /var/www/html/f14/i386/img/
echo "Done!"

echo -n "Mounting Fedora 14 Live DVD, x86_64... "
mount -o loop /var/www/html/f14_live/x86_64/iso/Fedora-14-x86_64-Live-Desktop.iso /var/www/html/f14_live/x86_64/img/
echo "Done!"

echo -n "Mounting Fedora 14 Live DVD, i686... "
mount -o loop /var/www/html/f14_live/i686/iso/Fedora-14-i686-Live-Desktop.iso /var/www/html/f14_live/i686/img/
echo "Done!"
</source>

* Direct download: [https://alteeve.com/files/pxe/umount_iso.sh umount_iso.sh]

<source lang="bash">
#!/bin/bash
# This is used to unmount the various ISO used by the PXE server on this server.

echo -n "Unmounting Fedora 14, x86_64... "
umount /var/www/html/f14/x86_64/img
echo "Done!"

echo -n "Unmounting Fedora 14, i386... "
umount /var/www/html/f14/i386/img/
echo "Done!"

echo -n "Unmounting Fedora 14 Live DVD, x86_64... "
umount /var/www/html/f14_live/x86_64/img/
echo "Done!"

echo -n "Unmounting Fedora 14 Live DVD, i686... "
umount /var/www/html/f14_live/i686/img/
echo "Done!"
</source>

=== Increasing Maximum number of Mountable ISOs ===

By default the loop driver only creates 8 loop back devices. Each mounted ISO takes one of those and there for limits the total number of ISOs that can be mounted at any one time.

There exist two places to change the maximum number of loop back devices.
The recommend method is to add max_loop option to a new config file in /etc/modprobe.d/.

/etc/modprobe.d/loop.conf
<source lang="text">
options loop max_loop=64
</source>

The second options is to add "max_loop=64" to the kernel boot line.

Adjust the number after max_loop= to suit your installations needs.

= Setting Up The Boot Environment =

All boot options, except localboot, need to have a kernel made available via the PXE server. Specifically, these kernels must exist somewhere under the /var/lib/tftpboot directory. What kernel or boot image a given operating system will need depends entirely on that operating system. For this section, we will continue to use Fedora 14 x86_64 as an example.

== Planning Your Directory Structure, Take Two ==

As with [[#Planning Your Directory Structure|Apache]], we're potentially going to copy a lot of boot images and other files. It makes sense to have an organized directory structure to put them all in. As before, I will share what I do, but please feel free to do what you find works best for you.

I like to create a dedicated directory called boot and then mimic the layout from Apache, minus the last set of three directories.

* <tftpboot>/boot/<os_name>/<architecture>/

Using Fedora 14 x86_64 with the default tftpboot install, I will create the following directory.

<source lang="bash">
mkdir -p /var/lib/tftpboot/boot/f14/x86_64
</source>

== The Bootable Kernel ==

Remember in the earlier LABEL f14_an-node01 example there was the APPEND initrd=boot/f14/x86_64/initrd.img ... line? The boot/f14/x86_64/initrd.img is the kernel, in the directory we just created!

The last piece of the puzzle then is; Where does the initrd.img file come from?

With most Linux distributions, there is an isolinux directory in the root of the installation DVD. The files needed to boot the DVD's installer (or "rescue mode") are in this directory. These are the same files we need to boot over PXE. If you have followed this tutorial up to this point, then you should be able to see the contents of this directory in your file browser. My PXE server uses the IP address 192.168.1.254, so I can go to
http://192.168.1.254/f14/x86_64/img/isolinux and see, among other files, initrd.img.

With the directory created and the ISO mounted, all that is left is to copy the contents of the isolinux files.

<source lang="bash">
cp -v /var/www/html/f14/x86_64/img/isolinux/* /var/lib/tftpboot/boot/f14/x86_64/
</source>
<source lang="text">
`/var/www/html/f14/x86_64/img/isolinux/boot.cat' -> `/var/lib/tftpboot/boot/f14/x86_64/boot.cat'
`/var/www/html/f14/x86_64/img/isolinux/boot.msg' -> `/var/lib/tftpboot/boot/f14/x86_64/boot.msg'
`/var/www/html/f14/x86_64/img/isolinux/grub.conf' -> `/var/lib/tftpboot/boot/f14/x86_64/grub.conf'
`/var/www/html/f14/x86_64/img/isolinux/initrd.img' -> `/var/lib/tftpboot/boot/f14/x86_64/initrd.img'
`/var/www/html/f14/x86_64/img/isolinux/isolinux.bin' -> `/var/lib/tftpboot/boot/f14/x86_64/isolinux.bin'
`/var/www/html/f14/x86_64/img/isolinux/isolinux.cfg' -> `/var/lib/tftpboot/boot/f14/x86_64/isolinux.cfg'
`/var/www/html/f14/x86_64/img/isolinux/memtest' -> `/var/lib/tftpboot/boot/f14/x86_64/memtest'
`/var/www/html/f14/x86_64/img/isolinux/splash.jpg' -> `/var/lib/tftpboot/boot/f14/x86_64/splash.jpg'
`/var/www/html/f14/x86_64/img/isolinux/splash.lss' -> `/var/lib/tftpboot/boot/f14/x86_64/splash.lss'
`/var/www/html/f14/x86_64/img/isolinux/syslinux-vesa-splash.jpg' -> `/var/lib/tftpboot/boot/f14/x86_64/syslinux-vesa-splash.jpg'
`/var/www/html/f14/x86_64/img/isolinux/TRANS.TBL' -> `/var/lib/tftpboot/boot/f14/x86_64/TRANS.TBL'
`/var/www/html/f14/x86_64/img/isolinux/vesamenu.c32' -> `/var/lib/tftpboot/boot/f14/x86_64/vesamenu.c32'
`/var/www/html/f14/x86_64/img/isolinux/vmlinuz' -> `/var/lib/tftpboot/boot/f14/x86_64/vmlinuz'
</source>

And we're done!

Now repeat the relevant steps for the rest of the operating systems you want to support via PXE.

= Credits =

References used:
* [http://linux-sxs.org/internet_serving/pxeboot.html Linux-SXS]
* [https://help.ubuntu.com/community/PXEInstallMultiDistro Ubuntu]
* [http://www.c3.hu/docs/oreilly/tcpip/tcpip/appd_03.htm O'Reilly]
* [http://www.linuxdoc.org/HOWTO/CDServer-HOWTO/addloops.html linuxdoc.org]

{{footer}}

Changing the ethX to Ethernet Device Mapping in EL5

2011-02-28T17:38:33Z

SRSullivan: /* The Problem */ Added note about using ethtool to ID/test ethX to Port mappings.

{{howto_header}}

'''Note''': This is for [[EL]]5 (CentOS 5.5, specifically). It also makes reference to [[Xen]], which you can ignore if you wish.

When you have two or more ethernet devices in one machine, the Red Hat/CentOS install may not install them in the order you want. This document will show you how to change the ethX to ethernet device mapping.

= Example =

I find it easier to follow instructions when I have an example to follow, so let me provide one here:

== Desired Mapping ==

Let's assume that you want this:

* eth0
** MAC: 00:aa:bb:cc:dd:ee
* eth1
** MAC: 00:11:22:33:44:55
* eth2
** MAC: 00:ff:ee:dd:88:99

== Initial Mapping ==

Now lets assume you got this:

* eth0
** MAC: 00:ff:ee:dd:88:99
* eth1
** MAC: 00:11:22:33:44:55
* eth2
** MAC: 00:aa:bb:cc:dd:ee

== The Problem ==

In the above example, eth1 is where we want it, so we leave it alone. The problem is that eth0 and eth2 are reversed.

If your not sure which ethX is currently mapped to which physical port, using ethtool -p ethX will make the light on the port blink. '''Note:''' This works for most, but not all hardware.

== The Fix ==

First, stop xend and network. This is important because if you change the MAC address to ethX mapping while the network is still up, the initd script will fail to bring down the network devices. The reason for stopping xend is because it is also integral to networking.

<source lang="bash">
/etc/init.d/xend stop
/etc/init.d/network stop
</source>

Now go to the directory that contains the network configuration files and then use cat to see the contents of the ifcfg-ethX network configuration files:

<source lang="bash">
cd /etc/sysconfig/network-scripts/
cat ifcfg-eth0
cat ifcfg-eth1
cat ifcfg-eth2
</source>

This will produce output something like this:

<source lang="text">
root@an_san01:/etc/sysconfig/network-scripts# cat ifcfg-eth0
# Intel Corporation 82540EM Gigabit Ethernet Controller
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
HWADDR=00:AA:BB:CC:DD:EE

root@an_san01:/etc/sysconfig/network-scripts# cat ifcfg-eth1
# Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
DEVICE=eth1
BOOTPROTO=dhcp
ONBOOT=yes
HWADDR=00:11:22:33:44:55

root@an_san01:/etc/sysconfig/network-scripts# cat ifcfg-eth2
# D-Link System Inc DGE-560T PCI Express Gigabit Ethernet Adapter
DEVICE=eth2
BOOTPROTO=dhcp
ONBOOT=yes
HWADDR=00:FF:EE:DD:88:99
</source>

The important lines are the HWADDR=... lines. Don't worry if the rest of the contents differ at this point. The above examples are intentionally generic and minimalist. For now, we are going to ignore them.

With networking stopped, all we need to do is change the HWADDR=... lines in ifcfg-eth0 to have the MAC address from ifcfg-eth2 and vice-versa.

So then, edit ifcfg-eth0 to look like this:
<source lang="bash">
vim ifcfg-eth0
</source>
<source lang="text">
# D-Link System Inc DGE-560T PCI Express Gigabit Ethernet Adapter
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
HWADDR=00:FF:EE:DD:88:99
</source>

And then edit ifcfg-eth2 to look like this:
<source lang="bash">
vim ifcfg-eth2
</source>
<source lang="text">
# Intel Corporation 82540EM Gigabit Ethernet Controller
DEVICE=eth2
BOOTPROTO=dhcp
ONBOOT=yes
HWADDR=00:AA:BB:CC:DD:EE
</source>

Note that I changed the comment at the top of the file to reflect the hardware description of the ethernet device. This is set by the installer and I like to keep it accurate. If you wish, you can simply delete it as it has no effect on the function of the network device.

Once the changes are complete, simply restart networking and xend:

<source lang="bash">
/etc/init.d/xend start
/etc/init.d/network start
</source>

You should be able to see that it is working by typing ifconfig and seeing something like this:

<source lang="bash">
ifconfig
</source>
<source lang="text">
eth0 Link encap:Ethernet HWaddr 00:FF:EE:DD:88:99
inet addr:192.168.1.71 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::92e6:baff:fe71:82d8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:555 errors:0 dropped:0 overruns:0 frame:0
TX packets:331 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:60616 (59.1 KiB) TX bytes:56370 (55.0 KiB)

eth1 Link encap:Ethernet HWaddr 00:11:22:33:44:55
inet addr:10.0.0.71 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::221:91ff:fe19:965a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:66 errors:0 dropped:0 overruns:0 frame:0
TX packets:58 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:11802 (11.5 KiB) TX bytes:10708 (10.4 KiB)

eth2 Link encap:Ethernet HWaddr 00:AA:BB:CC:DD:EE
inet addr:10.0.1.71 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::20e:cff:fe59:4578/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:65 errors:0 dropped:0 overruns:0 frame:0
TX packets:58 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:11670 (11.3 KiB) TX bytes:10734 (10.4 KiB)
</source>

As before, don't worry if the inet addr:... lines are different or missing. What is important is that the HWaddr ... entries correspond properly to the ethX devices. If they do, you are done! If they don't, double-check the ifcfg-ethX files have the proper HWADDR=... entries.

== Caveat! ==

'''''NOTE''''': If you are using a vlan, the device facing the vlan '''can not''' have the '''HWADDR=...''' value set! Set the others and leave this commented out. Otherwise, because of how the vlan loads, the OS will not see the physical device and will fail to bring up the interface at all.

{{footer}}

Talk:Setting Up a PXE Server on an RPM-based OS

2010-11-21T06:45:08Z

SRSullivan: reference to s/lib/share/ change.

For your tutorial, you move pxelinux.0 from /usr/share/syslinux/ but the package syslinux-tftpboot will provide it in /tftpboot for you.

Under CentOS 5, it's located in /usr/lib/syslinux/

( Research shows this change here [http://syslinux.zytor.com/wiki/index.php/Syslinux_3_Changelog#Changes_in_3.70] )

This is verifiable with a "yum provides */pxelinux.0".
Since this is a actual package, it should eliminate version drift from the original copy as the syslinux package updates.
--[[User:SRSullivan|SRSullivan]] 05:50, 21 November 2010 (UTC)

Talk:Setting Up a PXE Server on an RPM-based OS

2010-11-21T05:50:19Z

SRSullivan: Comments on simplifying setup by using an exsiting package.

For your tutorial, you move pxelinux.0 from /usr/share/syslinux/ but the package syslinux-tftpboot will provide it in /tftpboot for you.

Under CentOS 5, it's located in /usr/lib/syslinux/

This is verifiable with a "yum provides */pxelinux.0".
Since this is a actual package, it should eliminate version drift from the original copy as the syslinux package updates.
--[[User:SRSullivan|SRSullivan]] 05:50, 21 November 2010 (UTC)

Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

2010-10-18T20:25:23Z

SRSullivan: /* Removing Clustered Services From initd */

{{howto_header}}

'''Warning''': This is currently a dumping ground for notes. '''''DO NOT FOLLOW THIS DOCUMENT'S INSTRUCTIONS'''''. Seriously, it could blow up your computer or cause winter to come early.

----

This HowTo will walk you through setting up [[Xen]] [[VM]]s using [[DRBD]] and [[CLVM]] for high availability.

= Prerequisite =

This talk is an extension of the [[Two Node Fedora 13 Cluster]] HowTo. As such, you will be expected to have a freshly built two-node cluster with spare disk space on either node.

Please do not proceed until you have completed the first tutorial.

= Overview =

This tutorial will cover several topics; [[DRBD]], [[CLVM]], [[GFS2]], [[Xen]] [[dom0]] and [[domU]] [[VM]]s and [[rgmanager]]. Their relationship is thus:

* DRBD provides a mechanism to replicate data across both nodes in real time and guarantees a consistent view of that data from either node. Think of it like [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]], but across machines.
* CLVM sits on the DRBD partition and provides the underlying mechanism for allowing both nodes to access shared data in a clustered environment. It will host a shared filesystem by way of GFS2 as well as [[LV]]s that Xen's domU VMs will use as their disk space.
* GFS2 will be the clustered file system used on one of the DBRD-backed, CLVM-managed partitions. Files that need to be shared between nodes, like the Xen VM configuration files, will exist on this partition.
* Xen will be the hypervisor in use that will manage the various virtual machines. Each virtual machine will exist in an LVM LV.
** Xen's dom0 is the special "host" virtual machine. In this case, dom0 will be the OS installed in the first HowTo.
** Xen's domU virtual machines will be the "floating", highly available servers.
* Lastly, rgmanager will be the component of [[cman]] that will be configured to manage the automatic migration of the virtual machines when failures occur and when nodes recover.

= Setting Up Xen =

It may seem odd to start with [[Xen]] at this stage, but it is going to rather fundamentally alter each node's "host" operating system.

At this point, each node's host OS is a traditional operating system operating on the bare metal. When we install a dom0 kernel though, we tell Xen to boot a mini operating system first, and then to boot our "host" operating system. In effect, this converts the host node's operating system into just another virtual machine, albeit with a special view of the underlying hardware and Xen hypervisor.

This conversion is somewhat disruptive, so I like to get it out of the way right away. We will then do the rest of the setup before returning to Xen later on to create the floating virtual machines.

== A Note On The State Of Xen dom0 Support In Fedora ==

As of Fedora 8, support for Xen [[dom0]] has been removed, but support for the hypervisor and [[domU]] virtual machines remains. Red Hat's position is that KVM will be the supported platform going forward. That said, [http://fedoraproject.org/wiki/Features/XenPvopsDom0 this page] seems to indicate that PV Ops dom0 kernels will be supported in the future. Specifically, when dom0 support is merged into the mainline Linux kernel. When this will be is open to speculation, though "by Fedora 16" seems to be a reasonable educated guess.

What this means for us is that we need to use a non-standard dom0 kernel. Specifically, we will use a kernel created by [http://myoung.fedorapeople.org/ myoung] (Micheal Young) for Fedora 12. This kernel does not directly support DRBD, so be aware that we will need to build new DRBD kernel modules for his kernel and then rebuild the DRBD modules each time his kernel is updated.

== A Note On Rolling Your Own RPMs ==

If you want to roll the source RPMs for both the hypervisor and the [[dom0]] kernel, you will need to make both before you can boot into your dom0 for the first time. This is because the dom0 kernel needs the Xen microkernel provided by the xen hypervisor package to boot.

== Install The Hypervisor ==

We will use [[Xen]] 4.0.1, as provided by [http://pasik.reaktio.net/fedora/ Pasik]. We'll use the [http://pasik.reaktio.net/fedora/xen-4.0.1-0.2.fc13.src.rpm source RPM] to build our own RPM.

Regardless of whether you install from source RPMs or the pre-compiled ones, you will need to install the libvirt, qemu, SDL and PyXML packages from the standard repositories before you can proceed.

<source lang="bash">
yum -y install libvirt PyXML.x86_64 qemu.x86_64 SDL.x86_64
</source>

Please don't remove existing xen utilities and libraries prior to installing the newer version. If you do, core clustering components may be removed. Instead, be sure to use the rpm -Uvh switches to upgrade the existing packages that may be installed already.

=== Installing Prebuilt RPMs ===

These are locally stored copies of the Xen RPMs built for [[x86_64]] on Fedora 13. This requires some dependent packages be installed first.

'''Warning''': These are all recompiled for this website against Fedora 13, x86_64. If you feel more comfortable, please use [[RPM]]s from a source you are familiar with.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/xen-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-doc-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-hypervisor-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-libs-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-runtime-4.0.1-0.2.fc13.x86_64.rpm
rpm -Uvh xen*4.0.1-0.2.fc13.x86_64.rpm
</source>

Optionally, you may wish to install debug and/or devel RPMs as well.

<source lang="bash">
wget -c https://alteeve.com/files/xen-debuginfo-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/xen-devel-4.0.1-0.2.fc13.x86_64.rpm
rpm -Uvh xen-d*4.0.1-0.2.fc13.x86_64.rpm
</source>

=== Building RPMs From Source ===

To build the RPMs from the source, you need to make sure that you have the build environment installed.

'''Note''': If you are following these instruction after having installed a prior dom0 kernel, you will need to comment out exclude=kernel* in /etc/yum.conf so that the kernel-header package can be installed.

<source lang="bash">
vim /etc/yum.conf
</source>
<source lang="text">
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
#exclude=kernel*
</source>

Now install the development environment.

<source lang="bash">
yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
yum install transfig texi2html SDL-devel libX11-devel tetex-latex gtk2-devel libaio-devel dev86 iasl xz-devel e2fsprogs-devel glibc-devel.i686 xmlto asciidoc elfutils-libelf-devel
</source>

'''Note''': If you had to uncomment the exclude=kernel* line earlier, comment it back out now before you forget.

<source lang="bash">
vim /etc/yum.conf
</source>
<source lang="text">
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
exclude=kernel*
</source>

Now we can now build the RPMs from source.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/xen-4.0.1-0.2.fc13.src.rpm
rpm -ivh xen-4.0.1-0.2.fc13.src.rpm
cd /root/rpmbuild/SPECS/
rpmbuild -ba xen.spec
</source>

Once this is done, you will have seven RPMs built. Two of them are not really needed (xen-debuginfo-4.0.1-0.2.fc13.x86_64.rpm and xen-devel-4.0.1-0.2.fc13.x86_64.rpm) and I will leave it up to you may wish to install them or not. If you don't, modify the next command or move the debug and devel RPMs out of the way first.

<source lang="bash">
cd ~/rpmbuild/RPMS/x86_64/
rpm -Uvh xen*4.0.1-0.2.fc13.x86_64.rpm
</source>

== Installing The AN!Cluster dom0 Kernel ==

'''Notice'''!

The kernel provided here was recompiled on Fedora 13 and is a slightly modified version of [[#Installing Micheal Young's dom0 Kernel|Micheal Young's]] kernel available below. I was originally driven to recompile in an effort to solve a DRBD-related kernel oops. For now, unless you have the same DRBD kernel oops, I'd strongly recommend against using the AN!Cluster dom0 kernel until it has been tested much more thoroughly.

With that warning out of the way...

This kernel was compiled on a Fedora 13, x86_64. The DRBD RPMs available a little later where compiled against this dom0 kernel.

'''Note''': The --force is required because the current kernel is newer than 2.6.32 used here. Without this switch, the RPM would not install.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
</source>

If you would like to install the debug, devel and/or header RPMs for this kernel, they are available below.

'''Note''': The kernel-debuginfo-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm RPM is 213 [[MiB]].

<source lang="bash">
wget -c https://alteeve.com/files/an-cluster/kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-debuginfo-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-de*-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
</source>

=== Post AN!Cluster dom0 Install Configuration ===

The entry in grub's /boot/grub/menu.lst won't work. You will need to edit it so that it calls the existing installed operating system as a module.

'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your root= is likely different and your rd_MD_UUID= will definitely be different, even on the same machine across installs. Generally speaking, what follows the kernel /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ... entry made by the dom0 kernel can be copied after the module /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ... entry in the example below.

<source lang="bash">
vim /boot/grub/menu.lst
</source>
<source lang="bash">
title Xen 4.0.x, Linux 2.6.32.23-170.dom0_an1.fc13.x86_64
root (hd0,0)
kernel /xen.gz dom0_mem=1024M
module /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ...
module /initramfs-2.6.32.23-170.dom0_an1.fc13.x86_64.img

</source>

== Installing Micheal Young's dom0 Kernel ==

This uses a kernel built for Fedora 12, but it works on Fedora 13. This step involves either installing it over HTML or adding and enabling his repository and then installing it from there.

=== Installing Via myoung's Repository ===

This is almost always the preferred method. However, do note that when myoung updates his kernel, there will be a lag where the dom0 dependent RPMs provided here will no longer be compatible.

To add the repository, download the myoung.dom0.repo into the /etc/yum.repos.d/ directory.

<source lang="bash">
cd /etc/yum.repos.d/
wget -c http://myoung.fedorapeople.org/dom0/myoung.dom0.repo
</source>

To enable his repository, edit the repository file and change the two enabled=0 entries to enabled=1.

<source lang="bash">
vim /etc/yum.repos.d/myoung.dom0.repo
</source>
<source lang="text">
[myoung-dom0]
name=myoung's repository of Fedora based dom0 kernels - $basearch
baseurl=http://fedorapeople.org/~myoung/dom0/$basearch/
enabled=1
gpgcheck=0

[myoung-dom0-source]
name=myoung's repository of Fedora based dom0 kernels - Source
baseurl=http://fedorapeople.org/~myoung/dom0/src/
enabled=1
gpgcheck=0
</source>

Install the [[Xen]] [[dom0]] kernel (edit the version number with the updated version if it has changed).

<source lang="bash">
yum install kernel-2.6.32.21-170.xendom0.fc12.x86_64
</source>

=== Post Michael Young's dom0 Install Configuration ===

The entry in grub's /boot/grub/menu.lst won't work. You will need to edit it so that it calls the existing installed operating system as a module.

'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your root= is likely different and your rd_MD_UUID= will definitely be different, even on the same machine across installs. Generally speaking, what follows the kernel /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ... entry made by the dom0 kernel can be copied after the module /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ... entry in the example below.

<source lang="bash">
vim /boot/grub/menu.lst
</source>
<source lang="bash">
title Xen 4.0.x, Linux kernel 2.6.32.21-170.xendom0.fc12.x86_64
root (hd0,0)
kernel /xen.gz dom0_mem=1024M
module /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ...
module /initramfs-2.6.32.21-170.xendom0.fc12.x86_64.img
</source>

=== Disabling Automatic Kernel Updates ===

Seeing as we're using an older kernel, yum will want to replace it whenever there is an updated kernel* package available. Likewise if myoung updates his kernel. In the latter case, the updated kernel from Mr. Young would break compatibility with our DRBD module. So to be safe, we want to tell yum to never update the kernel.

To do this, we need to add exclude=kernel* to the /etc/yum.conf file.

<source lang="bash">
echo "exclude=kernel*" >> /etc/yum.conf
cat /etc/yum.conf
</source>
<source lang="text">
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=3
color=never

# This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
# It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m

# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d

exclude=kernel*
</source>

=== Make xend play nice with clustering ===

By default under Fedora 13, cman will start before xend. This is a problem because xend takes the network down as part of it's setup. This causes [[totem]] communication to fail which leads to fencing.

'''Note''': Move xenconsoled and xenstore to 09 and 10 start positions and then make xend depend on the before starting.

To avoid this, edit the initialization scripts for /etc/init.d/xend and it's dependents xenconsoled and xenstore to have a lower minimum start position. We need to maintain the start order of xenstore first, xenconsoled second and lastly xend. By default, their minimum start positions are 96, 97 and 98 respectively. We will change these to 10, 11 and 12, again, respectively.

Note that we are '''not''' altering the start position of xendomains! This is intentional as this daemon will start the [[domU]] VMs. This can not happen until all other cluster related daemons have started.

To change the start order we will change the line chkconfig: 2345 9x 01 lines to chkconfig: 2345 1x 01, where x is the given daemon's start number. Further, we'll make sure that xenstored begins first by add it to xenconsoled's Required-Start line. We'll then make sure that xenconsoled starts before xend by adding it to xend's Required-Start line.

To recap the changes;
* xenstored will start first.
** We'll change it's start position from 96 to 10.
** We will not add anything to it's Required-Start as it must be the first daemon to come up.
* xenconsoled will start second.
** We'll change it's start position from 97 to 11.
** We will add xenstored to it's Required-Start line.
* xend will start third.
** We'll change it's start position from 98 to 12.
** We will add xenconsoled to it's Required-Start line.

When done, the three initialization scripts should look like the examples below.

<source lang="bash">
vim /etc/init.d/xenstored
</source>
<source lang="bash">
#!/bin/bash
#
# xenstored Script to start and stop the Xen control daemon.
#
# Author: Daniel Berrange <berrange@redhat.com
#
# chkconfig: 2345 10 01
# description: Starts and stops the Xen xenstored daemon.
### BEGIN INIT INFO
# Provides: xenstored
# Required-Start: $syslog $remote_fs
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xenstored
# Description: Starts and stops the Xen xenstored daemon.
### END INIT INFO
</source>

<source lang="bash">
vim /etc/init.d/xenconsoled
</source>
<source lang="bash">
#!/bin/bash
#
# xenconsoled Script to start and stop the Xen xenconsoled daemon
#
# Author: Daniel P. Berrange <berrange@redhat.com>
#
# chkconfig: 2345 11 01
# description: Starts and stops the Xen control daemon.
### BEGIN INIT INFO
# Provides: xenconsoled
# Required-Start: $syslog $remote_fs xenstored
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xenconsoled
# Description: Starts and stops the Xen xenconsoled daemon.
### END INIT INFO
</source>

<source lang="bash">
vim /etc/init.d/xend
</source>
<source lang="bash">
#!/bin/bash
#
# xend Script to start and stop the Xen control daemon.
#
# Author: Keir Fraser <keir.fraser@cl.cam.ac.uk>
#
# chkconfig: 2345 12 98
# description: Starts and stops the Xen control daemon.
### BEGIN INIT INFO
# Provides: xend
# Required-Start: $syslog $remote_fs xenconsoled
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xend
# Description: Starts and stops the Xen control daemon.
### END INIT INFO
</source>

With xend set to start at a position lower than 98, we now have room for chkconfig to put other daemons after it in the start order, which will be needed a little later. First and foremost, we now need to tell cman to not start until after xend is up.

As above, we will now edit cman's /etc/init.d/cman script. This time though, we will not edit it's chkconfig line. Instead, we will simply add xend to the Required-Start line.

<source lang="bash">
vim /etc/init.d/cman
</source>
<source lang="bash">
#!/bin/bash
#
# cman - Cluster Manager init script
#
# chkconfig: - 21 79
# description: Starts and stops cman
#
#
### BEGIN INIT INFO
# Provides: cman
# Required-Start: $network $time xend
# Required-Stop: $network $time
# Default-Start:
# Default-Stop:
# Short-Description: Starts and stops cman
# Description: Starts and stops the Cluster Manager set of daemons
### END INIT INFO
</source>

Finally, remove and re-add the xend and cman daemons to re-order them in the start list:

<source lang="bash">
chkconfig xenstored off; chkconfig xenconsoled off; chkconfig xend off; chkconfig cman off;
chkconfig xenstored on; chkconfig xenconsoled on; chkconfig xend on; chkconfig cman on
</source>

Confirm that the order has changed so that xend is earlier in the boot sequence than cman. Assuming you've switched to run-level 3, run:

<source lang="bash">
ls -lah /etc/rc3.d/
</source>

Your start sequence should now look like:

<source lang="text">
lrwxrwxrwx. 1 root root 19 Sep 15 19:29 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 21 Sep 15 19:29 S27xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 14 Sep 15 19:29 S28xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Sep 15 19:29 S29cman -> ../init.d/cman
</source>

=== Booting Into The New dom0 ===

If everything went well, you should be able to boot the new dom0 operating system. If you watch the boot process closely, you will see that the boot process is different. You should now see the Xen hypervisor boot prior to handing off to the "host" operating system. This can be confirmed once the dom0 operating system has booted by checking that the file /proc/xen/capabilities exists. What it contains doesn't matter at this stage, only that it exists at all.

== Configure Networking ==

Networking in Xen, particularly in a cluster, can be confusing. If you are not familiar with networking in Xen, please review to following article before proceeding.

A note of a major change from previous layouts. In Xen 3.x, ethX would be copied to a virtual interface called vethX. Then the real ethX would be renamed to pethX and the virtual interface vethX would be renamed to ethX to take it's place. Finally, a bridge called xenbrX would be created and the real pethX and virtual ethX would be connected to it.

This has been changed somewhat it that now, by default, ethX is left alone and a simple bridge called virbrX would be created. We'll be changing this to be somewhat similar to the old style.

Specifically, the real ethX will be renamed to pethX. Then a bridge will be created called ethX, which plays the role of dom0's interface '''and''' bridges connections from VMs through pethX and out into the real world.

This is explained in more detail, and with diagrams, in the article below.

* [[Networking in Xen]]

=== Adding New NICs to Xen ===

By default, xend manages eth0 only. We need to add eth2. Personally, I don't like to put the storage network ethernet devices (eth1) under Xen's control as this potentially can cause DRBD problems on xend restart. Whether you add it or not I will leave to your preferences.

You can see which, if any, network devices are under Xen's control by running ifconfig and checking to see if there is a virbrX corresponding to a given ethX device.

<source lang="bash">
ifconfig
</source>
<source lang="text">
eth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet addr:192.168.1.74 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:224261 errors:0 dropped:0 overruns:0 frame:0
TX packets:55174 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:319384110 (304.5 MiB) TX bytes:27348739 (26.0 MiB)
Interrupt:225 Base address:0x8000

eth1 Link encap:Ethernet HWaddr 00:1B:21:72:9B:5A
inet addr:192.168.2.74 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:9b5a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:832 (832.0 b) TX bytes:6234 (6.0 KiB)
Memory:feae0000-feb00000

eth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet addr:192.168.3.74 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:818 (818.0 b) TX bytes:6081 (5.9 KiB)
Memory:fe9e0000-fea00000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

virbr0 Link encap:Ethernet HWaddr 02:23:C8:98:31:17
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:4013 (3.9 KiB)
</source>

In the above example, eth0 has a corresponding virbr0 bridge having it's own subnet. In non-clustered systems, this is fine. For our purposes though, it will not do.

=== Removing The qemu virbr0 Bridge ===

By default, [[QEMU]] creates a bridge called virbr0 designed to connect virtual machines to the first eth0 interface. Our system will not need this, so we will remove it. This bridge is configured in the /etc/libvirt/qemu/networks/default.xml file, so to remove this bridge, simply delete the contents of the file.

<source lang="bash">
cat /dev/null >/etc/libvirt/qemu/networks/default.xml
</source>

The next time you reboot, that bridge will be gone.

'''Madi''': Put in the command to delete the bridge before a reboot.

=== Create /etc/xen/scripts/an-network-script ===

This script will be used by Xen to turn the dom0 ethX interfaces into bridges. All traffic to the bridge, be it from dom0 or domU VMs, will be routeable out of the corresponding pethX device. As domU VMs come online, a hotplug script will create virtual interfaces between this new bridge and the domU's interface(s). Think of the vifX.Y devices as being the network cables you'd normally run between a server and a switch.

Before we proceed, please note three things;
# You don't need to use the file name an-network-script. I suggest this name mainly to keep in line with the rest of the 'AN!x' naming used here.
# If you install convirt or other hypervisor tools, they will likely create their own bridge script.
# Adding eth1 is optional, as we know ahead of time that eth1 will not be made available to any virtual machines as it is dedicated to [[DRBD]]. I'm adding it here because I like having things consistent; Do whichever makes more sense to you.

First, touch the file and then chmod it to be executable.

<source lang="bash">
touch /etc/xen/scripts/an-network-script
chmod 755 /etc/xen/scripts/an-network-script
</source>

Now edit it to contain the following:
<source lang="bash">
vim /etc/xen/scripts/an-network-script
</source>
<source lang="text">
#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=eth0
"$dir/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=eth2
</source>

Now tell Xen to execute that script by editing /etc/xen/xend-config.sxp file and changing the network-script argument to point to this new script (this is line 158 in the default xend-config.sxp script):

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>
<source lang="text">
#(network-script network-bridge)
#(network-script /bin/true)
(network-script an-network-script)
</source>

'''Warning''': The next step may trigger fencing of the nodes! As such, be sure that you're not running anything critical. If unsure, please stop cman or reboot the nodes.

<source lang="bash">
/etc/init.d/cman stop
</source>

Now restart xend.

<source lang="bash">
/etc/init.d/xend restart
</source>

If everything worked, you should now be able to run ifconfig and see that all the ethX devices have matching pethX, virtual and bridge devices.

<source lang="bash">
ifconfig
</source>
<source lang="text">
eth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet addr:192.168.1.74 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:78 errors:0 dropped:0 overruns:0 frame:0
TX packets:57 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9796 (9.5 KiB) TX bytes:12574 (12.2 KiB)

eth1 Link encap:Ethernet HWaddr 00:1B:21:72:9B:5A
inet addr:192.168.2.74 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:9b5a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:832 (832.0 b) TX bytes:6234 (6.0 KiB)
Memory:feae0000-feb00000

eth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet addr:192.168.3.74 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:32 errors:0 dropped:0 overruns:0 frame:0
TX packets:29 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5471 (5.3 KiB) TX bytes:5867 (5.7 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

peth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:224486 errors:0 dropped:0 overruns:0 frame:0
TX packets:55349 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:319406626 (304.6 MiB) TX bytes:27384681 (26.1 MiB)
Interrupt:225 Base address:0x8000

peth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:35 errors:0 dropped:0 overruns:0 frame:0
TX packets:70 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6827 (6.6 KiB) TX bytes:12470 (12.1 KiB)
Memory:fe9e0000-fea00000
</source>

'''Note''': The virbr0 may remain until you reboot your nodes.

If you see something like this, then you are ready to proceed! Now start your cluster back up.

<source lang="bash">
/etc/init.d/cman start
</source>

We're done for now. There is more to do in Xen, but this was all we needed to do in order to proceed with the next several steps. Onces we have the clustered storage online, we'll come back to Xen for the domU setup.

= Building the DRBD Array =

Building the DRBD array requires a few steps. First, raw space on either node must be prepared. Next, DRBD must be told that it is to create a resource using this newly configured raw space. Finally, the new array must be initialized.

== A Map of the Cluster's Storage ==

The layout of the storage in the cluster can quickly become difficult to follow. Below is an [[ASCII]] drawing which should help you see how DRBD will tie in to the rest of the cluster's storage. This map assumes a simple [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]] array underlying each node. If your node has a single hard drive, simply collapse the first two layers into one. Similarly, if your underlying storage is a more complex RAID array, simply expand the number of physical devices at the top level.

<source lang="text">
Node1 Node2
_____ _____ _____ _____
| sda | | sdb | | sda | | sdb |
|_____| |_____| |_____| |_____|
|_______| |_______|
_______ ____|___ _______ _______ ____|___ _______
__|__ __|__ __|__ __|__ __|__ __|__ __|__ __|__
| md0 | | md1 | | md2 | | md3 | | md3 | | md2 | | md1 | | md0 |
|_____| |_____| |_____| |_____| |_____| |_____| |_____| |_____|
| | | | | | | |
___|___ _|_ ____|____ |___________| ____|____ _|_ ___|___
| /boot | | / | | <swap> | | | <swap> | | / | | /boot |
|_______| |___| |_________| ______|______ |_________| |___| |_______|
| /dev/drbd0 |
|_____________|
|
____|______
| clvm PV |
|___________|
|
_____|_____
| drbd0_vg0 |
|___________|
|
_____|_____ ___...____
| | |
___|___ ___|___ ___|___
| lv_X | | lv_Y | | lv_N |
|_______| |_______| |_______|
</source>

== Install The DRBD Tools ==

DRBD has two components; The actual application and tools and the kernel module.

There are two options for installing the DRBD user-land tools at this point; AN!Cluster-built RPMs or using the ones shipped with Fedora. Regardless of which method you choose, you will need to either install the AN!Cluster DRBD kernel module RPMs or else rebuild the source RPMs referenced.

=== Install The AN!Cluster DRBD User-Land Tools ===

I am currently experimenting with ways to solve a DRBD triggered kernel oops in the Xen pvops 2.6.32 kernel. For this reason, I've recompiled the following user-land RPMs under the AN!Cluster variant dom0 kernel RPMs referenced earlier in this paper. If you used the AN! RPMs, then I suggest giving these RPMs a try. However, if you are using myoung's dom0, I recommend sticking to the Fedora-provided user-land DRBD tools.

<source lang="bash">
yum -y install bash-completion heartbeat pacemaker
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-utils-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-xen-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-bash-completion-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-udev-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-heartbeat-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-pacemaker-8.3.7-2.fc13.x86_64.rpm
rpm -ivh drbd*8.3.7-2.fc13.x86_64.rpm
</source>

=== Install The Stock Fedora DRBD User-Land Tools ===

You will need to install the following tools:

<source lang="bash">
yum install drbd.x86_64 drbd-xen.x86_64 drbd-utils.x86_64
</source>

=== Disable heartbeat ===

These packages require that the heartbeat packages be installed. This is for a different cluster platform which we are not using here, so we will disable it from starting with the system.

<source lang="bash">
chkconfig heartbeat off
</source>

== Install The DRBD Kernel Module ==

The kernel module '''must''' match the [[dom0]] kernel that is running. If you update the kernel and neglect to update the DRBD kernel module, the DRBD array '''will not start'''.

To help simplify things, links to pre-compiled DRBD kernel modules are provided. If the kernel version you have installed doesn't match your kernel, instructions on recompiling the DRBD kernel module from source RPM is provided as well.

=== Install Pre-Compiled DRBD Kernel Module RPMs ===

'''Warning''': The RPM provided here '''''will only work''''' with the kernel-2.6.32.21-168.xendom0_an1.fc13.x86_64.rpm kernel. If you are using Michael Young's dom0 kernel, please skip to [[#Building DRBD Kernel Module RPMs From Source|the next section]].

This RPM provides the DRBD kernel module. Note that these RPMs are compiled against the AN!Cluster variant of myoung's 2.6.32.21_168 dom0 kernel.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-km-2.6.32.23_170.dom0_an1.fc13.x86_64-8.3.7-12.fc13.x86_64.rpm
rpm -ivh drbd-km-2.6.32.23_170.dom0_an1.fc13.x86_64-8.3.7-12.fc13.x86_64.rpm
</source>

If you would like to install the debuginfo

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm
rpm -ivh drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm
</source>

=== Building DRBD Kernel Module RPMs From Source ===

If the above RPMs don't work or if the dom0 kernel you are using in any way differs, please follow the steps here to create a DRBD kernel module matched to your running dom0.

First, install the build environment.

<source lang="bash">
yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
</source>

Install the kernel headers and development library for the dom0 kernel:

'''Note''': The following commands use --force to get past the fact that the headers for the 2.6.33 are already installed, thus making RPM think that these are too old and will conflict. Please proceed with caution.

* If you are using Michael Young's kernel:

<source lang="bash">
cd ~
wget -c http://fedorapeople.org/~myoung/dom0/x86_64/kernel-headers-2.6.32.21-170.xendom0.fc12.x86_64.rpm
wget -c http://fedorapeople.org/~myoung/dom0/x86_64/kernel-devel-2.6.32.21-170.xendom0.fc12.x86_64.rpm
rpm -ivh --force kernel*2.6.32.21-170.xendom0.fc12.x86_64.rpm
</source>

* If you are using the AN!Cluster dom0 kernel:

<source lang="bash">
wget -c https://alteeve.com/files/an-cluster/kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm</source>

Now you need to download, prepare, build and install the source RPM.

<source lang="bash">
rpm -ivh http://fedora.mirror.iweb.ca/releases/13/Everything/source/SRPMS/drbd-8.3.7-2.fc13.src.rpm
cd /root/rpmbuild/SPECS/
rpmbuild -bp drbd.spec
cd /root/rpmbuild/BUILD/drbd-8.3.7/
./configure --enable-spec --with-km
cp /root/rpmbuild/BUILD/drbd-8.3.7/drbd-km.spec /root/rpmbuild/SPECS/
cd /root/rpmbuild/SPECS/
rpmbuild -ba drbd-km.spec
cd /root/rpmbuild/RPMS/x86_64
rpm -Uvh drbd-km-*
</source>

This may be needed if drbd-utils, the user-land DRBD tools, is listed as a requirement when trying to install the drbd-km* RPMs. This step will build all of the DRBD tools RPMs.

<source lang="bash">
yum install bash-completion heartbeat pacemaker
cd /root/rpmbuild/SPECS/
rpmbuild -ba drbd.spec
cd ~/rpmbuild/RPMS/x86_64/
rpm -Uvh drbd-*
chkconfig off heartbeat
</source>

You should be good to go now!

== Allocating Raw Space For DRBD On Each Node ==

If you followed the setup steps provided for in "[[Two Node Fedora 13 Cluster]]", you will have a set amount of unconfigured hard drive space. This is what we will use for the DRBD space on either node. If you've got a different setup, you will need to allocate some raw space before proceeding.

=== Create a Simple Partition ===

If you do not have two drives, please follow the next section's steps, but pay attention to the "'''note'''"s. In short, you will need to create one partition, leave the default type of the partition as 83, write the changes to disk and the proceed to the [[#DRBD Configuration Files|DRBD Configuration Files]] section.

=== Creating a RAID level 1 'md' Device ===

This assumes that you have two raw drives, /dev/sda and /dev/sdb. It further assumes that you've created three partitions which have been assigned to three existing /dev/mdX devices. With these assumptions, we will create /dev/sda4 and /dev/sdb4 and, using them, create a new /dev/md3 device that will host the DRBD partition.

If you have multiple drives and plan to use a different [[TLUG_Talk:_Storage_Technologies_and_Theory#RAID_Levels|RAID levels]], please adjust the follow commands accordingly.

==== Creating The New Partitions ====

'''Warning''': The next steps will have you directly accessing your server's hard drive configuration. Please do not proceed on a live server until you've had a chance to work through these steps on a test server. One mistake can '''''blow away all your data'''''.

Start the fdisk shell for the first hard drive; /dev/sda.

<source lang="bash">
fdisk /dev/sda
</source>
<source lang="text">
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help):
</source>

'''Note''': Depending on your configuration, you may not see the above warning or you may see a different warning. Note it, but it is likely nothing to worry about it.

View the current configuration with the '''p'''rint option

<source lang="bash">
p
</source>
<source lang="text">
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6fe1

Device Boot Start End Blocks Id System
/dev/sda1 1 5100 40960000 fd Linux raid autodetect
/dev/sda2 5100 5622 4194304 fd Linux raid autodetect
/dev/sda3 * 5622 5654 256000 fd Linux raid autodetect

Command (m for help):
</source>

Now we know for sure that the next free partition number is 4. We will now create the '''n'''ew partition.

<source lang="bash">
n
</source>
<source lang="text">
Command action
e extended
p primary partition (1-4)
</source>

We will make it a '''p'''rimary partition

<source lang="bash">
p
</source>
<source lang="text">
Selected partition 4
First cylinder (5654-60801, default 5654):
</source>

Then we simply hit <enter> to select the default starting block.

<source lang="bash">
<enter>
</source>
<source lang="text">
Using default value 5654
Last cylinder, +cylinders or +size{K,M,G} (5654-60801, default 60801):
</source>

Once again we will press <enter> to select the default ending block.

<source lang="bash">
<enter>
</source>
<source lang="text">
Using default value 60801

Command (m for help):
</source>

'''Note''': If you only have one drive and are not creating a [[RAID]] array, you do not to change the type of the partition so you can skip the next few steps. Continue at the step where you write the changes.

Now we need to change the type of partition that it is.

<source lang="bash">
t
</source>
<source lang="text">
Partition number (1-4):
</source>

We know that we are modifying partition number 4.

<source lang="bash">
4
</source>
<source lang="text">
Hex code (type L to list codes):
</source>

Now we need to set the [[hex]] code for the [[Filesystem_List#List_of_Linux_Partition_Types|partition type]] to set. We want to set fd, which defines Linux raid autodetect.

<source lang="bash">
fd
</source>
<source lang="text">
Changed system type of partition 4 to fd (Linux raid autodetect)
</source>

Now check that everything went as expected by once again '''p'''rinting the partition table.

<source lang="bash">
p
</source>
<source lang="text">
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6fe1

Device Boot Start End Blocks Id System
/dev/sda1 1 5100 40960000 fd Linux raid autodetect
/dev/sda2 5100 5622 4194304 fd Linux raid autodetect
/dev/sda3 * 5622 5654 256000 fd Linux raid autodetect
/dev/sda4 5654 60801 442972704+ fd Linux raid autodetect

Command (m for help):
</source>

'''Note''': If you only have one drive, your partitions will be 83 Linux or 82 Linux swap / Solaris, instead of fd Linux raid autodetect.

There it is. So finally, we need to '''w'''rite the changes to the disk.

<source lang="bash">
w
</source>
<source lang="text">
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
</source>

'''Note''': If you only have one drive, reboot now if you got the message above and then skip forward to the "[[#DRBD Configuration Files|DRBD Configuration Files]]" section.

If you see the above message, '''do not''' reboot yet. repeat these steps for the second drive, /dev/sdb, and then reboot.

==== Creating The New /dev/mdX Device ====

''If you only have one drive, skip this step.''

Now we need to use mdadm to create the new [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]] device. This will be used as the device that DRBD will directly access.

<source lang="bash">
mdadm --create /dev/md3 --homehost=localhost.localdomain --raid-devices=2 --level=1 /dev/sda4 /dev/sdb4
</source>
<source lang="text">
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
</source>

Seeing as /boot doesn't exist on this device, we can safely ignore this warning.

<source lang="bash">
y
</source>
<source lang="text">
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/md4 started.
</source>

You can now cat /proc/mdstat to verify that it indeed built. If you're interested, you could open a new terminal window and use watch cat /proc/mdstat and watch the array build.

<source lang="bash">
cat /proc/mdstat
</source>
<source lang="text">
md3 : active raid1 sdb4[1] sda4[0]
442971544 blocks super 1.2 [2/2] [UU]
[>....................] resync = 0.8% (3678976/442971544) finish=111.0min speed=65920K/sec

md2 : active raid1 sda2[0] sdb2[1]
4193272 blocks super 1.1 [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]
40958908 blocks super 1.1 [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk

md0 : active raid1 sda3[0] sdb3[1]
255988 blocks super 1.0 [2/2] [UU]

unused devices: <none>
</source>

Finally, we need to make sure that the new array will start when the system boots. To do this, we'll again use mdadm, but with different options that will have it output data in a format suitable for the /etc/mdadm.conf file. We'll redirect this output to that config file, thus updating it.

<source lang="bash">
mdadm --detail --scan | grep md3 >> /etc/mdadm.conf
cat /etc/mdadm.conf
</source>
<source lang="text">
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=b58df6d0:d925e7bb:c156168d:47c01718
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=ac2cf39c:77cd0314:fedb8407:9b945bb5
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=4e513936:4a966f4e:0dd8402e:6403d10d
ARRAY /dev/md3 metadata=1.2 name=localhost.localdomain:3 UUID=f0b6d0c1:490d47e7:91c7e63a:f8dacc21
</source>

You'll note that the last line, which we just added, is different from the previous lines. This isn't a concern, but you are welcome to re-write it to match the existing format if you wish.

Before you proceed, it is strongly advised that you reboot each node and then verify that the new array did in fact start with the system. You ''do not'' need to wait for the sync to finish before rebooting. It will pick up where you left off once rebooted.

'''Note''': You'll notice we did not format a file system on this raid array, this is intentional. DRBD use the raw device and does not need a file system on it.

== DRBD Configuration Files ==

DRBD uses a global configuration file, /etc/drbd.d/global_common.conf, and one or more resource files. The resource files need to be created in the /etc/drbd.d/ directory and must have the suffix .res. For this example, we will create a single resource called r0 which we will configure in /etc/drbd.d/r0.res.

=== /etc/drbd.d/global_common.conf ===

The stock /etc/drbd.d/global_common.conf is sane, so we won't bother altering it here.

Full details on all the drbd.conf configuration file directives and arguments can be found [http://www.drbd.org/users-guide/re-drbdconf.html here]. '''Note''': That link doesn't show this new configuration format. Please see [http://www.novell.com/documentation/sle_ha/book_sleha/?page=/documentation/sle_ha/book_sleha/data/sec_ha_drbd_configure.html Novell's] link.

=== /etc/drbd.d/r0.res ===

This is the important part. This defines the resource to use, and must reflect the IP addresses and storage devices that DRBD will use for this resource.

<source lang="bash">
vim /etc/drbd.d/r0.res
</source>
<source lang="bash">
# This is the name of the resource and it's settings. Generally, 'r0' is used
# as the name of the first resource. This is by convention only, though.
resource r0
{
# This tells DRBD where to make the new resource available at on each
# node. This is, again, by convention only.
device /dev/drbd0;

# The main argument here tells DRBD that we will have proper locking
# and fencing, and as such, to allow both nodes to set the resource to
# 'primary' simultaneously.
net
{
allow-two-primaries;
}

# This tells DRBD to automatically set both nodes to 'primary' when the
# nodes start.
startup
{
become-primary-on both;
}

# This tells DRBD to look for and store it's meta-data on the resource
# itself.
meta-disk internal;

# The name below must match the output from `uname -n` on each node.
on an-node01.alteeve.com
{
# This must be the IP address of the interface on the storage
# network (an-node01.sn, in this case).
address 192.168.2.71:7789;

# This is the underlying partition to use for this resource on
# this node.
disk /dev/md3;
}

# Repeat as above, but for the other node.
on an-node02.alteeve.com
{
address 192.168.2.72:7789;
disk /dev/md3;
}
}
</source>

This file must be copied to '''BOTH''' nodes and must match before you proceed.

== Starting The DRBD Resource ==

From the rest of this section, pay attention to whether you see
* '''Node1'''
* '''Node2'''
* '''Both'''

These indicate which node to run the following commands on. There is no functional difference between either node, so just randomly choose one to be '''Node1''' and the other will be '''Node2'''. Once you've chosen which is which, be consistent with which node you run the commands on. Of course, if a command block is proceeded by '''Both''', run the following code block on both nodes.

=== Loading the 'drbd' Module ===

'''Both'''

Normally, we'd load the drbd module by simply starting the /etc/init.d/drbd daemon. However, if we did that at this stage, we'd generate errors because there isn't an UpToDate disk in the array. To get around this, we'll manually load the drbd kernel module using modprobe.

<source lang="bash">
modprobe drbd
</source>

This won't return any output, but if you check, you should now see the special /proc/drbd file.

=== Monitoring Progress ===

'''Both'''

I find it very useful to monitor DRBD while running the rest of the setup. To do this, open a second terminal on each node and use watch to keep an eye on /proc/drbd. This way you will be able to monitor the progress of the array in near-real time.

'''Both'''
<source lang="bash">
watch cat /proc/drbd
</source>

At this stage, it should look like this:

<source lang="text">
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:Unconfigured
</source>

=== Initialize The Resource ===

'''Both'''

This step creates the DRBD meta-data on the new DRBD resource's backing devices. It is only needed when creating new DRBD partitions.

<source lang="bash">
drbdadm create-md r0
</source>
<source lang="text">
--== Thank you for participating in the global usage survey ==--
The server's response is:

you are the 9507th user to install this version
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
</source>

The /proc/drbd output should not have changed at this stage.

=== Starting the Resource ===

'''Both'''

This will attach the backing device, /dev/md3 in our case, and then start the new resource r0.

<source lang="bash">
drbdadm up r0
</source>

There will be no output at the command line. If you are watching /proc/drbd though, you should now see something like this:

<source lang="text">
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:442957988
</source>

That it is Secondary/Secondary and Inconsistent/Inconsistent is expected.

=== Setting the First Primary Node ===

'''Node1'''

As this is a totally new resource, DRBD doesn't know which side of the array is "more valid" than the other. In reality, neither is as there was no existing data of note on either node. This means that we now need to choose a node and tell DRBD to treat it as the "source" node. This step will also tell DRBD to make the "source" node primary. Once set, DRBD will begin sync'ing in the background.

<source lang="bash">
drbdadm -- --overwrite-data-of-peer primary r0
</source>

As before, there will be no output at the command line, but /proc/drbd will change to show the following:

<source lang="text">
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:69024 nr:0 dw:0 dr:69232 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:442888964
[>....................] sync'ed: 0.1% (432508/432576)M
finish: 307:33:42 speed: 320 (320) K/sec
</source>

If you're watching the secondary node, the /proc/drbd will show ro:Secondary/Primary ds:Inconsistent/UpToDate. This is, as you can guess, simply a reflection of it being the "over-written" node.

=== Setting the Second Node to Primary ===

'''Node2'''

The last step to complete the array is to tell the second node to also become primary.

<source lang="bash">
drbdadm primary r0
</source>

As with many drbdadm commands, nothing will be printed to the console. If you're watching the /proc/drbd though, you should see something like Primary/Primary ds:UpToDate/Inconsistent. The Inconsistent flag will remain until the sync is complete.

=== A Note On sync Speed ===

You will notice in the previous step that the sync speed seems awfully slow at 320 (320) K/sec.

'''This is not a problem!'''

As actual data is written to either side of the array, that data will be immediately copied to both nodes. As such, both nodes will always contain up to date copies of the real data. Given this, the syncer is intentionally set low so as to not put too much load on the underlying disks that could cause slow downs. If you still wish to increase the sync speed, you can do so with the following command.

'''Warning''': If you set the DRBD sync speed too high and saturate your disks' maximum write speed, clvmd daemon will likely fail to start in some cases, leading to fences. For this reason, keep your sync rate to about 2/3rds of the underlying disk maximum write speed and hold off on bumping up the sync speed until you know you will have a period of low activity on your cluster.

<source lang="bash">
drbdsetup /dev/drbd0 syncer -r 35M
</source>

The speed-up will not be instant. It will take a little while for the speed to pick up. Once the sync is finished, it is a good idea to revert to the default sync rate.

<source lang="text">
drbdadm syncer r0
</source>

= Setting Up CLVM =

The goal of DRBD in the cluster is to provide clustered [[LVM]], referred to as [[CLVM]] to the nodes. This is done by turning the DRBD partition into an CLVM physical volume.

So now we will create a [[PV]] on top of the new [[DRBD]] partition, /dev/drbd0, that we created in the previous step. Since this new LVM [[PV]] will exist on top of the shared DRBD partition, whatever get written to it's logical volumes will be immediately available on either node, regardless of which node actually initiated the write.

This capability is the underlying reason for creating this cluster; Neither machine is truly needed so if one machine dies, anything on top of the DRBD partition will still be available. When the failed machine returns, the surviving node will have a list of what blocks changed while the other node was gone and can use this list to quickly re-sync the other server.

== Making LVM Cluster-Aware ==

Normally, LVM is run on a single server. This means that at any time, the LVM can write data to the underlying drive and not need to worry if any other device might change anything. In clusters, this isn't the case. The other node could try to write to the shared storage, so then nodes need to enable "locking" to prevent the two nodes from trying to work on the same bit of data at the same time.

The process of enabling this locking is known as making LVM "cluster-aware".

LVM has tool called lvmconf that can be used to enable LVM locking. This is provided as part of the lvm2-cluster package.

<source lang="bash">
yum -y install lvm2-cluster.x86_64
</source>

Now to enable cluster awareness in LVM, run to following command.

<source lang="bash">
lvmconf --enable-cluster
</source>

By default, clvmd, the cluster lvm daemon, is stopped and not set to run on boot. Now that we've enabled LVM locking, we need to start it:

<source lang="bash">
/etc/init.d/clvmd status
</source>
<source lang="text">
clvmd is stopped
active volumes: (none)
</source>

As expected, it is stopped, so lets start it:

'''Note''': At this point cman is still set to not start a boot. Since we rebooted after creating the partitions that make up /dev/md3, cman will likely, and in my case was still off. clvmd will fail to start because the cluster manager (cman) is not started. --[[User:SRSullivan|SRSullivan]] 17:40, 18 October 2010 (UTC)

<source lang="bash">
/etc/init.d/clvmd start
</source>
<source lang="text">
Activating VGs: No volume groups found
[ OK ]
</source>

'''Note''': I've seen on a few occasions where starting clvmd will time out and, on occasion, fences will be issued. I've not sorted out why, but I have usually been able to resolve this by stopping clvmd and cman, then restarting cman and, finally, restarting clvmd. If I can sort out a way to reliably trigger this problem, I will submit a bug report.

== Filtering Out Devices ==

'''ToDo''': Find a less-aggressive filter.

With the stock /etc/lvm/lvm.conf configuration, all devices on the system will be checked for LVM volumes. This can cause a problem as LVM will give preference to the LVM data on the RAID device over the DRBD device. It sees a duplicate as both are, effectively, one and the same.

To work around this, we need to alter the filter = [] entry. At the time of writing, simply rejecting the underlying /dev/md3 device as a candidate wasn't enough. So for now, we will tell LVM to accept DRBD devices and reject all other devices. To do this, we'll insert "a|/dev/drbd*|" as the first array entry and change the existing entry to "r/.*/".

'''Note''': I would love feedback on a filter argument that successfully ignored just /dev/md3, if anyone can suggest one.

<source lang="bash">
vim /etc/lvm/lvm.conf
</source>
<source lang="text">
# By default we accept every block device:
#filter = [ "a/.*/" ]
filter = [ "a|/dev/drbd*|", "r/.*/" ]
</source>

Now delete the existing cache file so that LVM is forced to rescan the system.

<source lang="bash">
rm -f /etc/lvm/cache/.cache
</source>

The changes take effect immediately.

== Creating a new PV using the DRBD Partition ==

'''Node1'''

We can now proceed with setting up the new DRBD-based LVM physical volume. Once the PV is created, we can create a new volume group and start allocating space to logical volumes.

'''Note''': As we will be using our DRBD device, and as it is a shared block device, most of the following commands only need to be run on one node. Once the block device changes in any way, those changes will near-instantly appear on the other node. For this reason, unless explicitly stated to do so, only run the following commands on one node.

To setup the DRBD partition as an LVM PV, run pvcreate:

<source lang="bash">
pvcreate /dev/drbd0
</source>
<source lang="text">
Physical volume "/dev/drbd0" successfully created
</source>

'''Both'''

Now, on both nodes, check that the new physical volume is visible by using pvdisplay:

<source lang="bash">
pvdisplay
</source>
<source lang="text">
"/dev/drbd0" is a new physical volume of "422.44 GiB"
--- NEW Physical volume ---
PV Name /dev/drbd0
VG Name
PV Size 422.44 GiB
Allocatable NO
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID YHmdip-SuJN-KIEv-2tbK-BT9Q-wfOo-OuQuaW
</source>

If you see PV Name /dev/drbd0 (or your underlying partition) on both nodes, then your DRBD setup and LVM configuration changes are working perfectly!

== Creating a VG on the new PV ==

'''Node1'''

Now we need to create the volume group using the vgcreate command:

<source lang="bash">
vgcreate -c y drbd0_vg0 /dev/drbd0
</source>
<source lang="text">
Clustered volume group "drbd0_vg0" successfully created
</source>

'''Both'''

Now we'll check that the new VG is visible on both nodes using vgdisplay:

<source lang="bash">
vgdisplay
</source>
<source lang="text">
--- Volume group ---
VG Name drbd0_vg0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
Clustered yes
Shared no
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 422.43 GiB
PE Size 4.00 MiB
Total PE 108143
Alloc PE / Size 0 / 0
Free PE / Size 108143 / 422.43 GiB
VG UUID Bb8l9e-es2z-PhaF-Gg3o-2is2-DZ1S-V2RsBF
</source>

If the new VG is visible on both nodes, we are ready to create our first logical volume using the lvcreate tool.

== Creating the First LV on the new VG ==

'''Node1'''

Now we'll create a simple 20 GiB logical volumes. We will use it as a shared GFS2 store for shared files and to store our Xen domU config files later on.

<source lang="bash">
lvcreate -L 20G -n xen_shared drbd0_vg0
</source>
<source lang="text">
Logical volume "xen_shared" created
</source>

'''Both'''

As before, we will check that the new logical volume is visible from both nodes by using the lvdisplay command:

<source lang="bash">
lvdisplay
</source>
<source lang="text">
--- Logical volume ---
LV Name /dev/drbd0_vg0/xen_shared
VG Name drbd0_vg0
LV UUID AqQizc-KBpX-2scN-WFLb-jIeF-QDcM-PlQW84
LV Write Access read/write
LV Status available
# open 0
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
</source>

Again, if this is visible from both nodes, we're set! Repeat this process for all future LVs you will want to create. We will do this a little later to create LVs for Xen VMs.

= Creating A Shared GFS FileSystem =

GFS is a cluster-aware file system that can be simultaneously mounted on two or more nodes at once. We will use it as a place to store ISOs that we'll use to provision our virtual machines.

== Install The GFS2 Utilities ==

Start by installing the [[GFS2]] tools:

<source lang="bash">
yum -y install gfs2-utils.x86_64
</source>

== Format Our CLVM LV With The GFS2 File System ==

'''Node1'''

'''Note''': The following example is designed for the cluster used in the prerequisite HowTo.
* If you have more than 2 nodes, increase the -j 2 to the number of nodes you want to mount this file system on.
* If your cluster is named something other than an-cluster (as set in the cluster.conf file), change -t an-cluster:xen_shared to match you cluster's name. The xen_shared can be whatever you like, but it must be unique in the cluster. I tend to use a name that matches the LV name, but this is my own preference and is not required.

To format the partition run:
<source lang="bash">
mkfs.gfs2 -p lock_dlm -j 2 -t an-cluster:xen_shared /dev/drbd0_vg0/xen_shared
</source>
<source lang="text">
This will destroy any data on /dev/drbd0_vg0/xen_shared.
It appears to contain: symbolic link to `../dm-0'

Are you sure you want to proceed? [y/n]
</source>

Acknowledge the warning, if any, and then press y if you are ready to proceed.

<source lang="bash">
y
</source>
<source lang="text">
Device: /dev/drbd0_vg0/xen_shared
Blocksize: 4096
Device Size 20.00 GB (5242880 blocks)
Filesystem Size: 20.00 GB (5242878 blocks)
Journals: 2
Resource Groups: 80
Locking Protocol: "lock_dlm"
Lock Table: "an-cluster:xen_shared"
UUID: A1487063-2A3F-43B1-3A36-44936B0B4D1E
</source>

Once the format completes, you can mount /dev/drbd0_vg0/xen_shared as you would a normal file system.

'''Both''':

To complete the example, lets mount the GFS2 partition we made just now on /shared and then use df -h to verify.

<source lang="bash">
mkdir /xen_shared
mount /dev/drbd0_vg0/xen_shared /xen_shared
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.8G 34G 8% /
tmpfs 466M 29M 438M 7% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
/dev/dm-0 20G 259M 20G 2% /xen_shared
</source>

You may have noticed that it shows /dev/dm-0 instead of /dev/drbd0_vg0/xen_shared. If you look at the later, you will see that it is simply a [[symlink]] to the former.

<source lang="bash">
ls -lah /dev/drbd0_vg0/xen_shared
</source>
<source lang="text">
lrwxrwxrwx. 1 root root 7 Sep 9 13:24 /dev/drbd0_vg0/xen_shared -> ../dm-0
</source>

== Add An Entry To /etc/fstab ==

The last step is to add an entry for this new partition to each node's /etc/fstab file.

=== Reference The GFS2 Partition By Device Path ===

This is the more traditional method of referencing the GFS2 partition by using it's device path directly.

'''Warning''': An incorrect edit of the /etc/fstab file can leave your system unable to boot! Please review the line generated above to make sure it is accurate and compatible with your setup before proceeding.

<source lang="bash">
vim /etc/fstab
</source>
<source lang="text">
#
# /etc/fstab
# Created by anaconda on Tue Sep 7 06:29:51 2010
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=865690db-85d0-44c5-9f32-ffb6fdf47060 / ext3 defaults 1 1
UUID=8b9822b6-a92e-48c9-96b5-f8943142319e /boot ext3 defaults 1 2
UUID=94b03547-a7e3-45bb-b2d5-837498b370f4 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
xenfs /proc/xen xenfs defaults 0 0
/dev/drbd0_vg0/xen_shared /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

=== Reference The GFS2 Partition By UUID ===

It is sometimes preferable to create an fstab entry that locates the device path via it's [[UUID]]. To do this, you can run the following command which, though a bit cryptic, will print out an /etc/fstab compatible string.

'''Warning''': The same warnings apply here as above

<source lang="bash">
echo `gfs2_edit -p sb /dev/drbd0_vg0/xen_shared | grep sb_uuid | sed -e "s/.*sb_uuid *$.*$/UUID=\L\1\E \/xen_shared\t\tgfs2\trw,suid,dev,exec,nouser,async\t0 0/"`
</source>
<source lang="text">
UUID=a1487063-2a3f-43b1-3a36-44936b0b4d1e /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

You may have noticed that defaults isn't used. Rather, all but the auto option are manually set. This is because the system will drop to single-user mode at boot if it can't mount an auto partition at boot time (auto being implied by defaults). Given that our GFS2 partition sits on top of DRBD and the cluster, there is no way to make it available that early in the boot process.

Further, the gfs2 init script specifically excludes entries in /etc/fstab that have the 'noauto option set. For this reason, we can't simply specify that as we need the init script to see the partition so that it is mounted when GFS2 starts and unmounted when it stops.

Now add this string to /etc/fstab.

<source lang="bash">
vim /etc/fstab
</source>
<source lang="text">
#
# /etc/fstab
# Created by anaconda on Tue Sep 7 06:29:51 2010
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=865690db-85d0-44c5-9f32-ffb6fdf47060 / ext3 defaults 1 1
UUID=8b9822b6-a92e-48c9-96b5-f8943142319e /boot ext3 defaults 1 2
UUID=94b03547-a7e3-45bb-b2d5-837498b370f4 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
xenfs /proc/xen xenfs defaults 0 0
UUID=a1487063-2a3f-43b1-3a36-44936b0b4d1e /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

Please note; At the time of writing this HowTo, there is a bug in findfs and mount. According to [http://www.ietf.org/rfc/rfc4122.txt RFC 4122], programs should accept a [[UUID]] in either upper or lower case. However, this is not currently the case, so you '''must''' pass the UUID in lower-case. Please see bugs [https://bugzilla.redhat.com/show_bug.cgi?id=632373 632373] and [https://bugzilla.redhat.com/show_bug.cgi?id=632385 632385].

== Testing The gfs2 Initialization Script ==

To verify that the new entry is valid, check gfs2's status.

<source lang="bash">
/etc/init.d/gfs2 status
</source>
<source lang="text">
Configured GFS2 mountpoints:
/xen_shared
Active GFS2 mountpoints:
/xen_shared
</source>

Now test stopping and restarting to ensure that the GFS2 partition unmounts and mounts properly.

Stop gfs2.

<source lang="bash">
/etc/init.d/gfs2 stop
</source>
<source lang="text">
Unmounting GFS2 filesystem (/xen_shared): [ OK ]
</source>

Check with df -h to ensure that the mount is gone.

<source lang="bash">
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.0G 35G 6% /
tmpfs 466M 23M 444M 5% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
</source>

Start gfs2 again.

<source lang="bash">
/etc/init.d/gfs2 start
</source>
<source lang="text">
Mounting GFS2 filesystem (/xen_shared): [ OK ]
</source>

Again, check with df -h that it has been remounted.

<source lang="bash">
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.0G 35G 6% /
tmpfs 466M 29M 438M 7% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
/dev/dm-0 20G 259M 20G 2% /xen_shared
</source>

Done!

== Further Reading ==

* [[Grow a GFS2 Partition]]
* [[Hard drive has gone bad in DRBD]]

= Altering Daemon Start Order =

It is important that the various daemons in use by our cluster start in the right order. Most daemons will rely on services provided by another daemon to be running, and will not start or will not operate reliably otherwise.

We need to make sure that xend starts so that the network is stable. Then cman needs to start so that [[fencing]] and [[dlm]] are available. Next, drbd starts so that the clustered storage is available. Then clvmd must start so that the data on the DRBD resource is accessible. Now gfs2 needs to start so that the Xen domU configuration files can be found and finally xendomains must start to boot up the actual domU virtual machines.

To restate as a list, the start order must be:
* xend
* cman
* drbd
* clvmd
* gfs2
* xendomains

To make sure the start order is sane then, we'll edit each of the six daemon's init scripts and alter their Required-Start lines. To make the changes take effect, we will use chkconfig to remove and re-add them to the various start levels.

== Altering xend ==

This should already be done. If it isn't, please see "[[#Make xend play nice with clustering|Making xend play nice with clustering]]" above. If you are revisiting that section, you can skip the cman edit as we will need to make another change in the next step.

== Altering cman ==

This should already be done. If it isn't, please see "[[#Make xend play nice with clustering|Making xend play nice with clustering]]" above. If you are revisiting that section, you can skip the cman edit as we will need to make another change in the next step.

== Altering drbd ==

Now we will tell drbd to start after cman.

This requires the additional step of altering the chkconfig: - 70 08 line to instead read chkconfig: - 20 08. This isn't strictly needed, but will give more room for chkconfig to order the dependent daemons by allowing DRBD to be started as low as position 20, rather than waiting until position 70. This is somewhat more compatible with cman and clvmd which normally start at positions 21 and 24, respectively

<source lang="bash">
vim /etc/init.d/drbd
</source>
<source lang="text">
#!/bin/bash
#
# chkconfig: - 20 08
# description: Loads and unloads the drbd module
#
# Copright 2001-2008 LINBIT Information Technologies
# Philipp Reisner, Lars Ellenberg
#
### BEGIN INIT INFO
# Provides: drbd
# Required-Start: $local_fs $network $syslog cman
# Required-Stop: $local_fs $network $syslog
# Should-Start: sshd multipathd
# Should-Stop: sshd multipathd
# Default-Start:
# Default-Stop:
# Short-Description: Control drbd resources.
### END INIT INFO
</source>

== Altering clvmd ==

Now we will now tell clvmd to start after drbd.

'''Note''': There is currently a minor bug with lvm2-cluster version 2.02.73-2 in that /etc/init.d/clvmd is set by default to mode 0555. This is easily corrected by running the following command. Please check bug [https://bugzilla.redhat.com/show_bug.cgi?id=636066 636066] to see if this has been resolved.

<source lang="bash">
chmod u+w /etc/init.d/clvmd
</source>

Once you've got write access, edit the file.

<source lang="bash">
vim /etc/init.d/clvmd
</source>
<source lang="text">
#!/bin/bash
#
# chkconfig: - 24 76
# description: Starts and stops clvmd
#
# For Red-Hat-based distributions such as Fedora, RHEL, CentOS.
#
### BEGIN INIT INFO
# Provides: clvmd
# Required-Start: $local_fs drbd
# Required-Stop: $local_fs
# Default-Start:
# Default-Stop: 0 1 6
# Short-Description: Clustered LVM Daemon
### END INIT INFO
</source>

== Altering gfs2 ==

Now we will now tell gfs2 to start after clvmd. You will notice that cman is already listed under Required-Start and Required-Stop. It's true that cman must be started, but we've created a chain here so we can safely replace it with clvmd in the start line.

<source lang="bash">
vim /etc/init.d/gfs2
</source>
<source lang="text">
#!/bin/bash
#
# gfs2 mount/unmount helper
#
# chkconfig: - 26 74
# description: mount/unmount gfs2 filesystems configured in /etc/fstab

### BEGIN INIT INFO
# Provides: gfs2
# Required-Start: $network clvmd
# Required-Stop: $network
# Default-Start:
# Default-Stop:
# Short-Description: mount/unmount gfs2 filesystems configured in /etc/fstab
# Description: mount/unmount gfs2 filesystems configured in /etc/fstab
### END INIT INFO
</source>

== Altering xendomains ==

Finally, we will alter xendomains so that it starts last, after gfs2.

<source lang="bash">
vim /etc/init.d/xendomains
</source>
<source lang="text">
#!/bin/bash
#
# /etc/init.d/xendomains
# Start / stop domains automatically when domain 0 boots / shuts down.
#
# chkconfig: 345 99 00
# description: Start / stop Xen domains.
#
# This script offers fairly basic functionality. It should work on Redhat
# but also on LSB-compliant SuSE releases and on Debian with the LSB package
# installed. (LSB is the Linux Standard Base)
#
# Based on the example in the "Designing High Quality Integrated Linux
# Applications HOWTO" by Avi Alkalay
# <http://www.tldp.org/HOWTO/HighQuality-Apps-HOWTO/>
#
### BEGIN INIT INFO
# Provides: xendomains
# Required-Start: $syslog $remote_fs xend gfs2
# Should-Start:
# Required-Stop: $syslog $remote_fs xend
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop secondary xen domains
# Description: Start / stop domains automatically when domain 0
# boots / shuts down.
### END INIT INFO
</source>

== Applying The Changes ==

Change the start order by removing and re-adding all cluster-related daemons using chkconfig.

<source lang="bash">
chkconfig xenstored off; chkconfig xenconsoled off; chkconfig xend off; chkconfig cman off; chkconfig drbd off; chkconfig clvmd off; chkconfig gfs2 off; chkconfig xendomains off
chkconfig xendomains on; chkconfig gfs2 on; chkconfig clvmd on; chkconfig drbd on; chkconfig cman on; chkconfig xend on; chkconfig xenconsoled on; chkconfig xenstored on
</source>

Now verify that the start order is as we want it.

<source lang="bash">
ls -lah /etc/rc3.d/
</source>
<source lang="text">
lrwxrwxrwx. 1 root root 19 Sep 20 13:37 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 21 Sep 20 13:37 S27xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S28xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S29cman -> ../init.d/cman
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S70drbd -> ../init.d/drbd
lrwxrwxrwx. 1 root root 15 Sep 20 13:37 S71clvmd -> ../init.d/clvmd
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S72gfs2 -> ../init.d/gfs2
lrwxrwxrwx. 1 root root 20 Sep 20 13:37 S99xendomains -> ../init.d/xendomains
</source>

= Setting Up Xen =

'''''WARNING''''': Everything below here is pretty '''seriously screwed up'''.

'''Note''': This is not meant to be an extensive tutorial on Xen itself. It covers enough to get domU VMs provisioned in a manner that will take advantage of the cluster. As such, there is minimal explanation of configuration file options. If you need further help, please drop by the ##xen (yes, two ##) [[IRC]] channel on freenode.org.

== Install The Hypervisor Tools ==

These tools are very useful in provisioning and managing domU VMs.

<source lang="bash">
yum -y install virt-install virt-viewer
</source>

== Install The HVM/KVM Tools ==

For hvm (Hardware Virtual Machines), which is required for [http://www.virtuatopia.com/index.php/Virtualizing_Windows_Server_2008_with_Xen paravirtualized Microsoft] VMs, you must install the following packages as well.

<source lang="bash">
yum -y install qemu-kvm.x86_64 qemu-kvm-tools.x86_64
</source>

== Ensure That Virtualization Is Enabled ==

Many motherboards disable hvm by default in their [[BIOS]]. Assuming that you've got a dom0 kernel running at this stage, you can check if this is the case by checking the xm info output.

<source lang="bash">
xm info
</source>
<source lang="text">
host : an-node04.alteeve.com
release : 2.6.32.23-170.dom0_an1.fc13.x86_64
version : #1 SMP Sun Oct 10 20:39:19 EDT 2010
machine : x86_64
nr_cpus : 4
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 2209
hw_caps : 178bf3ff:efd3fbff:00000000:00001310:00802001:00000000:000037ff:00000000
virt_caps : hvm
total_memory : 4063
free_memory : 2987
node_to_cpu : node0:0-3
node_to_memory : node0:2987
node_to_dma32_mem : node0:2928
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline : dom0_mem=1024M
cc_compiler : gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC)
cc_compile_by : root
cc_compile_domain :
cc_compile_date : Mon Oct 11 01:10:38 EDT 2010
xend_config_format : 4
</source>

Look at the virt_caps and xen_caps lines. Notice the hvm entries? This shows that HVM, also known as "secure virtualization", has been enabled. If you do not see this, please check your mainboard manual for information on enabling this on your system.

'''Note''': The next paragraph applies only when running a vanilla kernel.

If you are running a vanilla kernel, you can check to see if your CPU has support for HVM guests but checking /proc/cpuinfo. What you're looking for depends on your CPU manufacturer. If you have an Intel CPU, you need to look for the vmx flag. Likewise, with AMD CPUs, you need to look for the svm flag.

For a more complete, if somewhat dated paper on this topic, please [http://fedoraproject.org/wiki/FedoraXenQuickstartFC6#Fully-virtualized_guests_.28HVM.2FIntel-VT.2FAMD-V.29 Fedora 6 Xen Quickstart Guide, System Requirements].

== Enabling Migration ==

By default, xend will not allow domU VMs from being migrated onto or off of a given dom0 host. Given that we've got a cluster though, we very much want this behaviour, so now we will enable it. This is done by making edits to /etc/xen/xend-config.sxp. Below is a concise list of options that must be set. Some exist already in the file and need to be commented out or altered.

'''Warning''': The values below are '''very''' permissive. Please review each option and improve the security to fit your network before going into production!

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>
<source lang="text">
(xend-http-server yes)
(xend-unix-server yes)
(xend-tcp-xmlrpc-server yes)
(xend-relocation-server yes)
(xend-udev-event-server yes)
(xend-port 8000)
(xend-relocation-port 8002)
(xend-address '')
(xend-relocation-address '')
(xend-relocation-hosts-allow '')
</source>

Once done, restart xend. It is usually safest to stop the cluster before hand to avoid accidental fencing caused by the underlying network being reconfigured.

<source lang="bash">
/etc/init.d/gfs2 stop
/etc/init.d/clvmd stop
/etc/init.d/cman stop
/etc/init.d/xend restart
/etc/init.d/cman start
/etc/init.d/clvmd start
/etc/init.d/gfs2 start
</source>

== Virtual Machine Naming Convention ==

'''Note''': This section acts as a recommendation only. Feel free to alter this to fit your style and needs.

Personally, I like to name my VMs similar to c5_shorewall_01. To elaborate, I like to use the format:
* os_role_seq (<Operating System ID>_<Role of the VM>_<Sequence Integer>).

There are no (known) restrictions on virtual machine names, so feel free to use names that made sense for you. I do strongly recommend that you match the name of your domU VM to the name of it's host LVM logical volume.

== Provisioning domU VMs ==

There are two ways to provision new VMs that we will cover (there are many others); Using virt-install and using xm create -f /path/to/domain.cfg where domain.cfg is a hand-crafted python script.

=== Provisioning with virt-install ===

This uses a long command line argument that provisions the VM and loads it into libvirt. Where possible, this is probably the best way to provision a domU. However, there are occasions where it may not work.

Here is an example where a domU is provisioned for a [[Fedora]] 13, x86_64 VM with a dedicated logical volume on the clustered LVM. The command to create the LV precedes the command to provision the VM. Please review and adjust values as you need. Consult man virt-install for a more complete list of available options and their uses.

Following the provision command are two examples of how to backup the configuration to a flat file. The first directly dumps the configuration into the libvirt format. The second backs up the configuration to an [[XML]] file and then shows and example of how that file can be converted into a standard python script.

<source lang="bash">
# Fedora 13 x86_64 RPM builder VM
lvcreate -L 40G -n f13_builder_01 drbd0_vg0
virt-install --connect xen \
--name f13_builder_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--location http://192.168.1.10/f13/x86_64/img/ \
--os-type linux \
--os-variant fedora13 \
--disk path=/dev/drbd0_vg0/f13_builder_01 \
--network bridge=eth0,mac=00:16:3e:00:10:01 \
--vnc \
--paravirt
# Backup the domU configuration to the libvirt native format.
xm list -l f13_builder_01 > /xen_shared/dom_config/f13_builder.cfg
# Backup the config to an XML file.
virsh dumpxml f13_builder_01 > /xen_shared/domU_config/f13_builder_01.xml
# Convert it to a "traditional" python script. Be sure the edit the resulting .cfg file to remove the 'vifname=' sections.
virsh -c xen:/// domxml-to-native xen-xm f13_builder_01.xml > f13_builder_01.cfg
</source>

The next two examples show the provisioning of a [[CentOS]] v5.5 and [[RHEL]] v6.0, beta 2 machines. The dump and backup methods above should be easily adapted to work with these VMs.

<source lang="bash">
# A CentOS test server
lvcreate -L 40G -n c5_test_01 drbd0_vg0
virt-install --connect xen \
--name c5_test_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1-3 \
--location http://192.168.1.10/c5/x86_64/img/ \
--os-type linux \
--os-variant rhel5.4 \
--disk path=/dev/drbd0_vg0/c5_test_01 \
--network bridge=eth0,mac=00:16:3e:00:10:02 \
--vnc \
--paravirt
</source>

<source lang="bash">
# Red Hat Enterprise Linux 6 beta 2 test server
lvcreate -L 40G -n rh6b2_test_01 drbd0_vg0
virt-install --connect xen \
--name rh6b2_test_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--location http://192.168.1.10/rhel6/x86_64/img/ \
--os-type linux \
--os-variant rhel6 \
--disk path=/dev/drbd0_vg0/rh6b2_test_01 \
--network bridge=eth0,mac=00:16:3e:00:10:03 \
--vnc \
--paravirt
</source>

=== Provisioning With 'xm create' ===

At the time of writing this, I could not sort out the magical incantation for provisioning a [[Windows]] VM using virt-install. Instead, I used the "old" style of crafting a configuration file using a python script. This is useful to know as many templates exist on the web for various VMs. Following the steps below, you should be able to fairly easily adapt them.

In this example, we will provision a VM using [[HVM]] from an [[ISO]] image of the Windows 2008 Server installation DBD. In this case, there is a problem with virt-install finding the qemu-dm file.

This configuration file will be saved in the /xen_shared/domU_config directory, which exists on the shared [[GFS2]] partition we created earlier.

<source lang="bash">
mkdir /xen_shared/domU_config
vim /xen_shared/domU_config/win2008_sql_01.cfg
</source>
<source lang="python">
# This is the Windows 2008 Enterprise Server x86_64 hosting MS SQL Server 2008 Enterprise
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
memory = 1024

# Should be at least 2KB per MB of domain memory, plus a few MB per vcpu.
shadow_memory = 8
name = "win2008_sql_01"
#vif = [ 'type=ioemu, bridge=xenbr0' ]
vif = [ 'type=ioemu, bridge=eth0,mac=00:16:3e:00:30:03' ]
acpi = 1
apic = 1
# Remove the 'file:...' entry (or change it to another ISO) after the install is complete.
disk = [ 'phy:/dev/drbd0_vg0/win2008_sql_01,hda,w', 'file:/xen_shared/iso/MS-Win2008-Ent-x86_64-SP2.iso,hdc:cdrom,r' ]

device_model = '/usr/lib/xen/bin/qemu-dm'

#-----------------------------------------------------------------------------
# boot on floppy (a), hard disk (c) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
boot="dc"
sdl=0
vnc=1
vncconsole=1
vncpasswd=''

serial='pty'
usbdevice='tablet'
</source>

Now provision the VM using xm create.

<source lang="bash">
xm create -f /xen_shared/domU_config/win2008_sql_01.cfg
</source>

At this point, the VM is not loaded into libvirt. This is a problem as, on boot, the node will not know that the VM exists. As a consequence, some tools like virt-manager will not see the VM until it is manually started with xm create -f /xen_shared/domU_config/domain.cfg. Further, and perhaps more troubling, changes made to the VM's config made outside to config file will be lost when you restart from the config file.

To fix this, we'll use a few hacks to load the config into libvirt. This needs to be done while the domU is loaded and it must be done on the node currently hosting the VM.

First, make sure that the domU's configuration is visible. Then, dump the config and filter it through grep and sed to pull out the [[UUID]]. Once you know that you get the proper UUID, create the directory under /var/lib/xend/domains/ and then create the config.sxp file with the domU's configuration.

'''NOTE''': Be sure to change win2008_01 in the examples below to match the name of the domU that you want to setup.

<source lang="bash">
xm list -l win2008_01
xm list -l win2008_01 | grep '^ (uuid' | sed -e "s/ (uuid $.*$)/\1/"
mkdir /var/lib/xend/domains/`xm list -l win2008_01 | grep '^ (uuid' | sed -e "s/ (uuid $.*$)/\1/"`
cd /var/lib/xend/domains/`xm list -l win2008_01 | grep '^ (uuid' | sed -e "s/ (uuid $.*$)/\1/"`
xm list -l win2008_01 > config.sxp
</source>

There are several uuid strings in the output from xm list -l domain. Thankfully, the one we want is the only one indented by four spaces. This is how we can be fairly confident that the UUID returned by sed above is, in fact,

= Making VMs Highly Available =

Now this is the point, isn't it?

In this final step, we're going to move the startup of drbd, clvmd, gfs2 and xendomains out of init.d and move them into our cluster. The reason for this is that the cluster is much wiser about how to handle clustered services. We do not want a node that is not in the cluster, that is, a node without quorum, to try and connect to shared resources. The cluster manager handles this.

'''Note''': This how-to is trying to keep things simple, so we will use rgmanager which is built in to cman. There is a compelling argument to use [[Pacemaker]] instead. That is a bit beyond the scope of the How-To though.

== Removing Clustered Services From initd ==

As stated, we will now remove the cluster-related services from starting with the os.

'''WARNING''': Incomplete, do not proceed with these steps! --[[User:SRSullivan|SRSullivan]] 20:25, 18 October 2010 (UTC)

<source lang="bash">
chkconfig xendomains off; chkconfig gfs2 off; chkconfig clvmd off; chkconfig drbd off
</source>


<source lang="bash">
</source>
<source lang="text">
</source>

{{footer}}

Talk:Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

2010-10-18T19:21:39Z

SRSullivan: Descrpenceis in init order. Documentation for testing.

We'll remove the yum -y before the document goes live. For now, I want it there to save time while (repeatedly) rebuilding the clusters as it saves a lot of time.

--[[User:Digimer|Digimer]] 18:58, 13 October 2010 (UTC)

== Dealing with Raid 5/6 ==

Under raid 5/6 if you follow these instruction to closely you'll end up with bad config data in mdadm.conf

<source lang="bash">
cat /proc/mdstat
</source>
<source lang="text">
Personalities : [raid1] [raid6] [raid5] [raid4]
md3 : active raid5 sdd5[4] sdc5[2] sdb5[1] sda5[0]
1395170304 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
[=====>...............] recovery = 26.5% (123522028/465056768) finish=126.7min speed=44896K/sec

md2 : active raid5 sda1[0] sdd1[4] sdc1[2] sdb1[1]
62909952 blocks super 1.1 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md1 : active raid5 sda2[0] sdd2[4] sdc2[2] sdb2[1]
6286848 blocks super 1.1 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]

md0 : active raid1 sda3[0] sdd3[3] sdc3[2] sdb3[1]
255988 blocks super 1.0 [4/4] [UUUU]

unused devices: <none>
</source>

You will notice the [UUU_]. The default, and this is not in the documentation for mdadm, is to build the array with a degraded spare and then resync it. This is faster, but produces false data for the next step if it is done before the array completes it resync.

Ref: [http://serverfault.com/questions/43575/how-to-create-a-software-raid5-array-without-a-spare/43581#43581]

<source lang="bash">
mdadm --detail --scan | grep md3 >> /etc/mdadm.conf
cat /etc/mdadm.conf
</source>

Will produce:
<source lang="text">ARRAY /dev/md3 metadata=1.2 spares=1 name=localhost.localdomain:3 UUID=b4fc7771:bef0b6a5:c4c40550:ab3bff42
</source>

Which should actually be the following (with out the spares=1)
<source lang="text">
ARRAY /dev/md3 metadata=1.2 name=localhost.localdomain:3 UUID=b4fc7771:bef0b6a5:c4c40550:ab3bff42
</source>
--[[User:SRSullivan|SRSullivan]] 20:05, 15 October 2010 (UTC)

== Tracking when to restart cman after reboots ==

During the creation of devices we reboot the machine, previous instructions had us turn of cman from starting on boot. For the final release document we should carefully note when cman will be down and when it should be re-enabled either manually or permanently.
--[[User:SRSullivan|SRSullivan]] 17:44, 18 October 2010 (UTC)

== chkconfig ordering ==
chkconfig version 1.3.45

Reverse (Madison) Order,

<source lang="bash">
chkconfig xendomains on; chkconfig gfs2 on; chkconfig clvmd on; chkconfig drbd on; chkconfig cman on; chkconfig xend on; chkconfig xenconsoled on; chkconfig xenstored on
</source>
<source lang="text">
lrwxrwxrwx. 1 root root 21 Oct 18 15:16 S26xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 19 Oct 18 15:16 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 14 Oct 18 15:16 S27xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Oct 18 15:16 S28cman -> ../init.d/cman
lrwxrwxrwx. 1 root root 14 Oct 18 15:16 S56drbd -> ../init.d/drbd
lrwxrwxrwx. 1 root root 15 Oct 18 15:16 S57clvmd -> ../init.d/clvmd
lrwxrwxrwx. 1 root root 14 Oct 18 15:16 S58gfs2 -> ../init.d/gfs2
lrwxrwxrwx. 1 root root 20 Oct 18 15:16 S99xendomains -> ../init.d/xendomains
</source>

Forward (Scott's) Order,
<source lang="bash">
chkconfig xenstored on; chkconfig xenconsoled on; chkconfig xend on; chkconfig cman on; chkconfig drbd on; chkconfig clvmd on; chkconfig gfs2 on; chkconfig xendomains on
</source>

<source lang="text">
lrwxrwxrwx. 1 root root 19 Oct 18 15:20 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 21 Oct 18 15:20 S27xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 14 Oct 18 15:20 S28xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Oct 18 15:20 S29cman -> ../init.d/cman
lrwxrwxrwx. 1 root root 14 Oct 18 15:20 S56drbd -> ../init.d/drbd
lrwxrwxrwx. 1 root root 15 Oct 18 15:20 S57clvmd -> ../init.d/clvmd
lrwxrwxrwx. 1 root root 14 Oct 18 15:20 S58gfs2 -> ../init.d/gfs2
lrwxrwxrwx. 1 root root 20 Oct 18 15:20 S99xendomains -> ../init.d/xendomains
</source>

Talk:Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

2010-10-18T17:44:34Z

SRSullivan: cman restarting notes

Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

2010-10-18T17:40:54Z

SRSullivan: /* Making LVM Cluster-Aware */ Oversight notice, cman still off since reboot just before creating /dev/md3.

{{howto_header}}

'''Warning''': This is currently a dumping ground for notes. '''''DO NOT FOLLOW THIS DOCUMENT'S INSTRUCTIONS'''''. Seriously, it could blow up your computer or cause winter to come early.

----

This HowTo will walk you through setting up [[Xen]] [[VM]]s using [[DRBD]] and [[CLVM]] for high availability.

= Prerequisite =

This talk is an extension of the [[Two Node Fedora 13 Cluster]] HowTo. As such, you will be expected to have a freshly built two-node cluster with spare disk space on either node.

Please do not proceed until you have completed the first tutorial.

= Overview =

This tutorial will cover several topics; [[DRBD]], [[CLVM]], [[GFS2]], [[Xen]] [[dom0]] and [[domU]] [[VM]]s and [[rgmanager]]. Their relationship is thus:

* DRBD provides a mechanism to replicate data across both nodes in real time and guarantees a consistent view of that data from either node. Think of it like [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]], but across machines.
* CLVM sits on the DRBD partition and provides the underlying mechanism for allowing both nodes to access shared data in a clustered environment. It will host a shared filesystem by way of GFS2 as well as [[LV]]s that Xen's domU VMs will use as their disk space.
* GFS2 will be the clustered file system used on one of the DBRD-backed, CLVM-managed partitions. Files that need to be shared between nodes, like the Xen VM configuration files, will exist on this partition.
* Xen will be the hypervisor in use that will manage the various virtual machines. Each virtual machine will exist in an LVM LV.
** Xen's dom0 is the special "host" virtual machine. In this case, dom0 will be the OS installed in the first HowTo.
** Xen's domU virtual machines will be the "floating", highly available servers.
* Lastly, rgmanager will be the component of [[cman]] that will be configured to manage the automatic migration of the virtual machines when failures occur and when nodes recover.

= Setting Up Xen =

It may seem odd to start with [[Xen]] at this stage, but it is going to rather fundamentally alter each node's "host" operating system.

At this point, each node's host OS is a traditional operating system operating on the bare metal. When we install a dom0 kernel though, we tell Xen to boot a mini operating system first, and then to boot our "host" operating system. In effect, this converts the host node's operating system into just another virtual machine, albeit with a special view of the underlying hardware and Xen hypervisor.

This conversion is somewhat disruptive, so I like to get it out of the way right away. We will then do the rest of the setup before returning to Xen later on to create the floating virtual machines.

== A Note On The State Of Xen dom0 Support In Fedora ==

As of Fedora 8, support for Xen [[dom0]] has been removed, but support for the hypervisor and [[domU]] virtual machines remains. Red Hat's position is that KVM will be the supported platform going forward. That said, [http://fedoraproject.org/wiki/Features/XenPvopsDom0 this page] seems to indicate that PV Ops dom0 kernels will be supported in the future. Specifically, when dom0 support is merged into the mainline Linux kernel. When this will be is open to speculation, though "by Fedora 16" seems to be a reasonable educated guess.

What this means for us is that we need to use a non-standard dom0 kernel. Specifically, we will use a kernel created by [http://myoung.fedorapeople.org/ myoung] (Micheal Young) for Fedora 12. This kernel does not directly support DRBD, so be aware that we will need to build new DRBD kernel modules for his kernel and then rebuild the DRBD modules each time his kernel is updated.

== A Note On Rolling Your Own RPMs ==

If you want to roll the source RPMs for both the hypervisor and the [[dom0]] kernel, you will need to make both before you can boot into your dom0 for the first time. This is because the dom0 kernel needs the Xen microkernel provided by the xen hypervisor package to boot.

== Install The Hypervisor ==

We will use [[Xen]] 4.0.1, as provided by [http://pasik.reaktio.net/fedora/ Pasik]. We'll use the [http://pasik.reaktio.net/fedora/xen-4.0.1-0.2.fc13.src.rpm source RPM] to build our own RPM.

Regardless of whether you install from source RPMs or the pre-compiled ones, you will need to install the libvirt, qemu, SDL and PyXML packages from the standard repositories before you can proceed.

<source lang="bash">
yum -y install libvirt PyXML.x86_64 qemu.x86_64 SDL.x86_64
</source>

Please don't remove existing xen utilities and libraries prior to installing the newer version. If you do, core clustering components may be removed. Instead, be sure to use the rpm -Uvh switches to upgrade the existing packages that may be installed already.

=== Installing Prebuilt RPMs ===

These are locally stored copies of the Xen RPMs built for [[x86_64]] on Fedora 13. This requires some dependent packages be installed first.

'''Warning''': These are all recompiled for this website against Fedora 13, x86_64. If you feel more comfortable, please use [[RPM]]s from a source you are familiar with.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/xen-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-doc-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-hypervisor-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-libs-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-runtime-4.0.1-0.2.fc13.x86_64.rpm
rpm -Uvh xen*4.0.1-0.2.fc13.x86_64.rpm
</source>

Optionally, you may wish to install debug and/or devel RPMs as well.

<source lang="bash">
wget -c https://alteeve.com/files/xen-debuginfo-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/xen-devel-4.0.1-0.2.fc13.x86_64.rpm
rpm -Uvh xen-d*4.0.1-0.2.fc13.x86_64.rpm
</source>

=== Building RPMs From Source ===

To build the RPMs from the source, you need to make sure that you have the build environment installed.

'''Note''': If you are following these instruction after having installed a prior dom0 kernel, you will need to comment out exclude=kernel* in /etc/yum.conf so that the kernel-header package can be installed.

<source lang="bash">
vim /etc/yum.conf
</source>
<source lang="text">
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
#exclude=kernel*
</source>

Now install the development environment.

<source lang="bash">
yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
yum install transfig texi2html SDL-devel libX11-devel tetex-latex gtk2-devel libaio-devel dev86 iasl xz-devel e2fsprogs-devel glibc-devel.i686 xmlto asciidoc elfutils-libelf-devel
</source>

'''Note''': If you had to uncomment the exclude=kernel* line earlier, comment it back out now before you forget.

<source lang="bash">
vim /etc/yum.conf
</source>
<source lang="text">
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
exclude=kernel*
</source>

Now we can now build the RPMs from source.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/xen-4.0.1-0.2.fc13.src.rpm
rpm -ivh xen-4.0.1-0.2.fc13.src.rpm
cd /root/rpmbuild/SPECS/
rpmbuild -ba xen.spec
</source>

Once this is done, you will have seven RPMs built. Two of them are not really needed (xen-debuginfo-4.0.1-0.2.fc13.x86_64.rpm and xen-devel-4.0.1-0.2.fc13.x86_64.rpm) and I will leave it up to you may wish to install them or not. If you don't, modify the next command or move the debug and devel RPMs out of the way first.

<source lang="bash">
cd ~/rpmbuild/RPMS/x86_64/
rpm -Uvh xen*4.0.1-0.2.fc13.x86_64.rpm
</source>

== Installing The AN!Cluster dom0 Kernel ==

'''Notice'''!

The kernel provided here was recompiled on Fedora 13 and is a slightly modified version of [[#Installing Micheal Young's dom0 Kernel|Micheal Young's]] kernel available below. I was originally driven to recompile in an effort to solve a DRBD-related kernel oops. For now, unless you have the same DRBD kernel oops, I'd strongly recommend against using the AN!Cluster dom0 kernel until it has been tested much more thoroughly.

With that warning out of the way...

This kernel was compiled on a Fedora 13, x86_64. The DRBD RPMs available a little later where compiled against this dom0 kernel.

'''Note''': The --force is required because the current kernel is newer than 2.6.32 used here. Without this switch, the RPM would not install.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
</source>

If you would like to install the debug, devel and/or header RPMs for this kernel, they are available below.

'''Note''': The kernel-debuginfo-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm RPM is 213 [[MiB]].

<source lang="bash">
wget -c https://alteeve.com/files/an-cluster/kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-debuginfo-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-de*-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
</source>

=== Post AN!Cluster dom0 Install Configuration ===

The entry in grub's /boot/grub/menu.lst won't work. You will need to edit it so that it calls the existing installed operating system as a module.

'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your root= is likely different and your rd_MD_UUID= will definitely be different, even on the same machine across installs. Generally speaking, what follows the kernel /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ... entry made by the dom0 kernel can be copied after the module /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ... entry in the example below.

<source lang="bash">
vim /boot/grub/menu.lst
</source>
<source lang="bash">
title Xen 4.0.x, Linux 2.6.32.23-170.dom0_an1.fc13.x86_64
root (hd0,0)
kernel /xen.gz dom0_mem=1024M
module /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ...
module /initramfs-2.6.32.23-170.dom0_an1.fc13.x86_64.img

</source>

== Installing Micheal Young's dom0 Kernel ==

This uses a kernel built for Fedora 12, but it works on Fedora 13. This step involves either installing it over HTML or adding and enabling his repository and then installing it from there.

=== Installing Via myoung's Repository ===

This is almost always the preferred method. However, do note that when myoung updates his kernel, there will be a lag where the dom0 dependent RPMs provided here will no longer be compatible.

To add the repository, download the myoung.dom0.repo into the /etc/yum.repos.d/ directory.

<source lang="bash">
cd /etc/yum.repos.d/
wget -c http://myoung.fedorapeople.org/dom0/myoung.dom0.repo
</source>

To enable his repository, edit the repository file and change the two enabled=0 entries to enabled=1.

<source lang="bash">
vim /etc/yum.repos.d/myoung.dom0.repo
</source>
<source lang="text">
[myoung-dom0]
name=myoung's repository of Fedora based dom0 kernels - $basearch
baseurl=http://fedorapeople.org/~myoung/dom0/$basearch/
enabled=1
gpgcheck=0

[myoung-dom0-source]
name=myoung's repository of Fedora based dom0 kernels - Source
baseurl=http://fedorapeople.org/~myoung/dom0/src/
enabled=1
gpgcheck=0
</source>

Install the [[Xen]] [[dom0]] kernel (edit the version number with the updated version if it has changed).

<source lang="bash">
yum install kernel-2.6.32.21-170.xendom0.fc12.x86_64
</source>

=== Post Michael Young's dom0 Install Configuration ===

The entry in grub's /boot/grub/menu.lst won't work. You will need to edit it so that it calls the existing installed operating system as a module.

'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your root= is likely different and your rd_MD_UUID= will definitely be different, even on the same machine across installs. Generally speaking, what follows the kernel /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ... entry made by the dom0 kernel can be copied after the module /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ... entry in the example below.

<source lang="bash">
vim /boot/grub/menu.lst
</source>
<source lang="bash">
title Xen 4.0.x, Linux kernel 2.6.32.21-170.xendom0.fc12.x86_64
root (hd0,0)
kernel /xen.gz dom0_mem=1024M
module /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ...
module /initramfs-2.6.32.21-170.xendom0.fc12.x86_64.img
</source>

=== Disabling Automatic Kernel Updates ===

Seeing as we're using an older kernel, yum will want to replace it whenever there is an updated kernel* package available. Likewise if myoung updates his kernel. In the latter case, the updated kernel from Mr. Young would break compatibility with our DRBD module. So to be safe, we want to tell yum to never update the kernel.

To do this, we need to add exclude=kernel* to the /etc/yum.conf file.

<source lang="bash">
echo "exclude=kernel*" >> /etc/yum.conf
cat /etc/yum.conf
</source>
<source lang="text">
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=3
color=never

# This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
# It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m

# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d

exclude=kernel*
</source>

=== Make xend play nice with clustering ===

By default under Fedora 13, cman will start before xend. This is a problem because xend takes the network down as part of it's setup. This causes [[totem]] communication to fail which leads to fencing.

'''Note''': Move xenconsoled and xenstore to 09 and 10 start positions and then make xend depend on the before starting.

To avoid this, edit the initialization scripts for /etc/init.d/xend and it's dependents xenconsoled and xenstore to have a lower minimum start position. We need to maintain the start order of xenstore first, xenconsoled second and lastly xend. By default, their minimum start positions are 96, 97 and 98 respectively. We will change these to 10, 11 and 12, again, respectively.

Note that we are '''not''' altering the start position of xendomains! This is intentional as this daemon will start the [[domU]] VMs. This can not happen until all other cluster related daemons have started.

To change the start order we will change the line chkconfig: 2345 9x 01 lines to chkconfig: 2345 1x 01, where x is the given daemon's start number. Further, we'll make sure that xenstored begins first by add it to xenconsoled's Required-Start line. We'll then make sure that xenconsoled starts before xend by adding it to xend's Required-Start line.

To recap the changes;
* xenstored will start first.
** We'll change it's start position from 96 to 10.
** We will not add anything to it's Required-Start as it must be the first daemon to come up.
* xenconsoled will start second.
** We'll change it's start position from 97 to 11.
** We will add xenstored to it's Required-Start line.
* xend will start third.
** We'll change it's start position from 98 to 12.
** We will add xenconsoled to it's Required-Start line.

When done, the three initialization scripts should look like the examples below.

<source lang="bash">
vim /etc/init.d/xenstored
</source>
<source lang="bash">
#!/bin/bash
#
# xenstored Script to start and stop the Xen control daemon.
#
# Author: Daniel Berrange <berrange@redhat.com
#
# chkconfig: 2345 10 01
# description: Starts and stops the Xen xenstored daemon.
### BEGIN INIT INFO
# Provides: xenstored
# Required-Start: $syslog $remote_fs
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xenstored
# Description: Starts and stops the Xen xenstored daemon.
### END INIT INFO
</source>

<source lang="bash">
vim /etc/init.d/xenconsoled
</source>
<source lang="bash">
#!/bin/bash
#
# xenconsoled Script to start and stop the Xen xenconsoled daemon
#
# Author: Daniel P. Berrange <berrange@redhat.com>
#
# chkconfig: 2345 11 01
# description: Starts and stops the Xen control daemon.
### BEGIN INIT INFO
# Provides: xenconsoled
# Required-Start: $syslog $remote_fs xenstored
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xenconsoled
# Description: Starts and stops the Xen xenconsoled daemon.
### END INIT INFO
</source>

<source lang="bash">
vim /etc/init.d/xend
</source>
<source lang="bash">
#!/bin/bash
#
# xend Script to start and stop the Xen control daemon.
#
# Author: Keir Fraser <keir.fraser@cl.cam.ac.uk>
#
# chkconfig: 2345 12 98
# description: Starts and stops the Xen control daemon.
### BEGIN INIT INFO
# Provides: xend
# Required-Start: $syslog $remote_fs xenconsoled
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xend
# Description: Starts and stops the Xen control daemon.
### END INIT INFO
</source>

With xend set to start at a position lower than 98, we now have room for chkconfig to put other daemons after it in the start order, which will be needed a little later. First and foremost, we now need to tell cman to not start until after xend is up.

As above, we will now edit cman's /etc/init.d/cman script. This time though, we will not edit it's chkconfig line. Instead, we will simply add xend to the Required-Start line.

<source lang="bash">
vim /etc/init.d/cman
</source>
<source lang="bash">
#!/bin/bash
#
# cman - Cluster Manager init script
#
# chkconfig: - 21 79
# description: Starts and stops cman
#
#
### BEGIN INIT INFO
# Provides: cman
# Required-Start: $network $time xend
# Required-Stop: $network $time
# Default-Start:
# Default-Stop:
# Short-Description: Starts and stops cman
# Description: Starts and stops the Cluster Manager set of daemons
### END INIT INFO
</source>

Finally, remove and re-add the xend and cman daemons to re-order them in the start list:

<source lang="bash">
chkconfig xenstored off; chkconfig xenconsoled off; chkconfig xend off; chkconfig cman off;
chkconfig xenstored on; chkconfig xenconsoled on; chkconfig xend on; chkconfig cman on
</source>

Confirm that the order has changed so that xend is earlier in the boot sequence than cman. Assuming you've switched to run-level 3, run:

<source lang="bash">
ls -lah /etc/rc3.d/
</source>

Your start sequence should now look like:

<source lang="text">
lrwxrwxrwx. 1 root root 19 Sep 15 19:29 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 21 Sep 15 19:29 S27xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 14 Sep 15 19:29 S28xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Sep 15 19:29 S29cman -> ../init.d/cman
</source>

=== Booting Into The New dom0 ===

If everything went well, you should be able to boot the new dom0 operating system. If you watch the boot process closely, you will see that the boot process is different. You should now see the Xen hypervisor boot prior to handing off to the "host" operating system. This can be confirmed once the dom0 operating system has booted by checking that the file /proc/xen/capabilities exists. What it contains doesn't matter at this stage, only that it exists at all.

== Configure Networking ==

Networking in Xen, particularly in a cluster, can be confusing. If you are not familiar with networking in Xen, please review to following article before proceeding.

A note of a major change from previous layouts. In Xen 3.x, ethX would be copied to a virtual interface called vethX. Then the real ethX would be renamed to pethX and the virtual interface vethX would be renamed to ethX to take it's place. Finally, a bridge called xenbrX would be created and the real pethX and virtual ethX would be connected to it.

This has been changed somewhat it that now, by default, ethX is left alone and a simple bridge called virbrX would be created. We'll be changing this to be somewhat similar to the old style.

Specifically, the real ethX will be renamed to pethX. Then a bridge will be created called ethX, which plays the role of dom0's interface '''and''' bridges connections from VMs through pethX and out into the real world.

This is explained in more detail, and with diagrams, in the article below.

* [[Networking in Xen]]

=== Adding New NICs to Xen ===

By default, xend manages eth0 only. We need to add eth2. Personally, I don't like to put the storage network ethernet devices (eth1) under Xen's control as this potentially can cause DRBD problems on xend restart. Whether you add it or not I will leave to your preferences.

You can see which, if any, network devices are under Xen's control by running ifconfig and checking to see if there is a virbrX corresponding to a given ethX device.

<source lang="bash">
ifconfig
</source>
<source lang="text">
eth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet addr:192.168.1.74 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:224261 errors:0 dropped:0 overruns:0 frame:0
TX packets:55174 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:319384110 (304.5 MiB) TX bytes:27348739 (26.0 MiB)
Interrupt:225 Base address:0x8000

eth1 Link encap:Ethernet HWaddr 00:1B:21:72:9B:5A
inet addr:192.168.2.74 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:9b5a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:832 (832.0 b) TX bytes:6234 (6.0 KiB)
Memory:feae0000-feb00000

eth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet addr:192.168.3.74 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:818 (818.0 b) TX bytes:6081 (5.9 KiB)
Memory:fe9e0000-fea00000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

virbr0 Link encap:Ethernet HWaddr 02:23:C8:98:31:17
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:4013 (3.9 KiB)
</source>

In the above example, eth0 has a corresponding virbr0 bridge having it's own subnet. In non-clustered systems, this is fine. For our purposes though, it will not do.

=== Removing The qemu virbr0 Bridge ===

By default, [[QEMU]] creates a bridge called virbr0 designed to connect virtual machines to the first eth0 interface. Our system will not need this, so we will remove it. This bridge is configured in the /etc/libvirt/qemu/networks/default.xml file, so to remove this bridge, simply delete the contents of the file.

<source lang="bash">
cat /dev/null >/etc/libvirt/qemu/networks/default.xml
</source>

The next time you reboot, that bridge will be gone.

'''Madi''': Put in the command to delete the bridge before a reboot.

=== Create /etc/xen/scripts/an-network-script ===

This script will be used by Xen to turn the dom0 ethX interfaces into bridges. All traffic to the bridge, be it from dom0 or domU VMs, will be routeable out of the corresponding pethX device. As domU VMs come online, a hotplug script will create virtual interfaces between this new bridge and the domU's interface(s). Think of the vifX.Y devices as being the network cables you'd normally run between a server and a switch.

Before we proceed, please note three things;
# You don't need to use the file name an-network-script. I suggest this name mainly to keep in line with the rest of the 'AN!x' naming used here.
# If you install convirt or other hypervisor tools, they will likely create their own bridge script.
# Adding eth1 is optional, as we know ahead of time that eth1 will not be made available to any virtual machines as it is dedicated to [[DRBD]]. I'm adding it here because I like having things consistent; Do whichever makes more sense to you.

First, touch the file and then chmod it to be executable.

<source lang="bash">
touch /etc/xen/scripts/an-network-script
chmod 755 /etc/xen/scripts/an-network-script
</source>

Now edit it to contain the following:
<source lang="bash">
vim /etc/xen/scripts/an-network-script
</source>
<source lang="text">
#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=eth0
"$dir/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=eth2
</source>

Now tell Xen to execute that script by editing /etc/xen/xend-config.sxp file and changing the network-script argument to point to this new script (this is line 158 in the default xend-config.sxp script):

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>
<source lang="text">
#(network-script network-bridge)
#(network-script /bin/true)
(network-script an-network-script)
</source>

'''Warning''': The next step may trigger fencing of the nodes! As such, be sure that you're not running anything critical. If unsure, please stop cman or reboot the nodes.

<source lang="bash">
/etc/init.d/cman stop
</source>

Now restart xend.

<source lang="bash">
/etc/init.d/xend restart
</source>

If everything worked, you should now be able to run ifconfig and see that all the ethX devices have matching pethX, virtual and bridge devices.

<source lang="bash">
ifconfig
</source>
<source lang="text">
eth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet addr:192.168.1.74 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:78 errors:0 dropped:0 overruns:0 frame:0
TX packets:57 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9796 (9.5 KiB) TX bytes:12574 (12.2 KiB)

eth1 Link encap:Ethernet HWaddr 00:1B:21:72:9B:5A
inet addr:192.168.2.74 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:9b5a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:832 (832.0 b) TX bytes:6234 (6.0 KiB)
Memory:feae0000-feb00000

eth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet addr:192.168.3.74 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:32 errors:0 dropped:0 overruns:0 frame:0
TX packets:29 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5471 (5.3 KiB) TX bytes:5867 (5.7 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

peth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:224486 errors:0 dropped:0 overruns:0 frame:0
TX packets:55349 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:319406626 (304.6 MiB) TX bytes:27384681 (26.1 MiB)
Interrupt:225 Base address:0x8000

peth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:35 errors:0 dropped:0 overruns:0 frame:0
TX packets:70 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6827 (6.6 KiB) TX bytes:12470 (12.1 KiB)
Memory:fe9e0000-fea00000
</source>

'''Note''': The virbr0 may remain until you reboot your nodes.

If you see something like this, then you are ready to proceed! Now start your cluster back up.

<source lang="bash">
/etc/init.d/cman start
</source>

We're done for now. There is more to do in Xen, but this was all we needed to do in order to proceed with the next several steps. Onces we have the clustered storage online, we'll come back to Xen for the domU setup.

= Building the DRBD Array =

Building the DRBD array requires a few steps. First, raw space on either node must be prepared. Next, DRBD must be told that it is to create a resource using this newly configured raw space. Finally, the new array must be initialized.

== A Map of the Cluster's Storage ==

The layout of the storage in the cluster can quickly become difficult to follow. Below is an [[ASCII]] drawing which should help you see how DRBD will tie in to the rest of the cluster's storage. This map assumes a simple [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]] array underlying each node. If your node has a single hard drive, simply collapse the first two layers into one. Similarly, if your underlying storage is a more complex RAID array, simply expand the number of physical devices at the top level.

<source lang="text">
Node1 Node2
_____ _____ _____ _____
| sda | | sdb | | sda | | sdb |
|_____| |_____| |_____| |_____|
|_______| |_______|
_______ ____|___ _______ _______ ____|___ _______
__|__ __|__ __|__ __|__ __|__ __|__ __|__ __|__
| md0 | | md1 | | md2 | | md3 | | md3 | | md2 | | md1 | | md0 |
|_____| |_____| |_____| |_____| |_____| |_____| |_____| |_____|
| | | | | | | |
___|___ _|_ ____|____ |___________| ____|____ _|_ ___|___
| /boot | | / | | <swap> | | | <swap> | | / | | /boot |
|_______| |___| |_________| ______|______ |_________| |___| |_______|
| /dev/drbd0 |
|_____________|
|
____|______
| clvm PV |
|___________|
|
_____|_____
| drbd0_vg0 |
|___________|
|
_____|_____ ___...____
| | |
___|___ ___|___ ___|___
| lv_X | | lv_Y | | lv_N |
|_______| |_______| |_______|
</source>

== Install The DRBD Tools ==

DRBD has two components; The actual application and tools and the kernel module.

There are two options for installing the DRBD user-land tools at this point; AN!Cluster-built RPMs or using the ones shipped with Fedora. Regardless of which method you choose, you will need to either install the AN!Cluster DRBD kernel module RPMs or else rebuild the source RPMs referenced.

=== Install The AN!Cluster DRBD User-Land Tools ===

I am currently experimenting with ways to solve a DRBD triggered kernel oops in the Xen pvops 2.6.32 kernel. For this reason, I've recompiled the following user-land RPMs under the AN!Cluster variant dom0 kernel RPMs referenced earlier in this paper. If you used the AN! RPMs, then I suggest giving these RPMs a try. However, if you are using myoung's dom0, I recommend sticking to the Fedora-provided user-land DRBD tools.

<source lang="bash">
yum -y install bash-completion heartbeat pacemaker
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-utils-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-xen-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-bash-completion-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-udev-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-heartbeat-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-pacemaker-8.3.7-2.fc13.x86_64.rpm
rpm -ivh drbd*8.3.7-2.fc13.x86_64.rpm
</source>

=== Install The Stock Fedora DRBD User-Land Tools ===

You will need to install the following tools:

<source lang="bash">
yum install drbd.x86_64 drbd-xen.x86_64 drbd-utils.x86_64
</source>

=== Disable heartbeat ===

These packages require that the heartbeat packages be installed. This is for a different cluster platform which we are not using here, so we will disable it from starting with the system.

<source lang="bash">
chkconfig heartbeat off
</source>

== Install The DRBD Kernel Module ==

The kernel module '''must''' match the [[dom0]] kernel that is running. If you update the kernel and neglect to update the DRBD kernel module, the DRBD array '''will not start'''.

To help simplify things, links to pre-compiled DRBD kernel modules are provided. If the kernel version you have installed doesn't match your kernel, instructions on recompiling the DRBD kernel module from source RPM is provided as well.

=== Install Pre-Compiled DRBD Kernel Module RPMs ===

'''Warning''': The RPM provided here '''''will only work''''' with the kernel-2.6.32.21-168.xendom0_an1.fc13.x86_64.rpm kernel. If you are using Michael Young's dom0 kernel, please skip to [[#Building DRBD Kernel Module RPMs From Source|the next section]].

This RPM provides the DRBD kernel module. Note that these RPMs are compiled against the AN!Cluster variant of myoung's 2.6.32.21_168 dom0 kernel.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-km-2.6.32.23_170.dom0_an1.fc13.x86_64-8.3.7-12.fc13.x86_64.rpm
rpm -ivh drbd-km-2.6.32.23_170.dom0_an1.fc13.x86_64-8.3.7-12.fc13.x86_64.rpm
</source>

If you would like to install the debuginfo

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm
rpm -ivh drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm
</source>

=== Building DRBD Kernel Module RPMs From Source ===

If the above RPMs don't work or if the dom0 kernel you are using in any way differs, please follow the steps here to create a DRBD kernel module matched to your running dom0.

First, install the build environment.

<source lang="bash">
yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
</source>

Install the kernel headers and development library for the dom0 kernel:

'''Note''': The following commands use --force to get past the fact that the headers for the 2.6.33 are already installed, thus making RPM think that these are too old and will conflict. Please proceed with caution.

* If you are using Michael Young's kernel:

<source lang="bash">
cd ~
wget -c http://fedorapeople.org/~myoung/dom0/x86_64/kernel-headers-2.6.32.21-170.xendom0.fc12.x86_64.rpm
wget -c http://fedorapeople.org/~myoung/dom0/x86_64/kernel-devel-2.6.32.21-170.xendom0.fc12.x86_64.rpm
rpm -ivh --force kernel*2.6.32.21-170.xendom0.fc12.x86_64.rpm
</source>

* If you are using the AN!Cluster dom0 kernel:

<source lang="bash">
wget -c https://alteeve.com/files/an-cluster/kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm</source>

Now you need to download, prepare, build and install the source RPM.

<source lang="bash">
rpm -ivh http://fedora.mirror.iweb.ca/releases/13/Everything/source/SRPMS/drbd-8.3.7-2.fc13.src.rpm
cd /root/rpmbuild/SPECS/
rpmbuild -bp drbd.spec
cd /root/rpmbuild/BUILD/drbd-8.3.7/
./configure --enable-spec --with-km
cp /root/rpmbuild/BUILD/drbd-8.3.7/drbd-km.spec /root/rpmbuild/SPECS/
cd /root/rpmbuild/SPECS/
rpmbuild -ba drbd-km.spec
cd /root/rpmbuild/RPMS/x86_64
rpm -Uvh drbd-km-*
</source>

This may be needed if drbd-utils, the user-land DRBD tools, is listed as a requirement when trying to install the drbd-km* RPMs. This step will build all of the DRBD tools RPMs.

<source lang="bash">
yum install bash-completion heartbeat pacemaker
cd /root/rpmbuild/SPECS/
rpmbuild -ba drbd.spec
cd ~/rpmbuild/RPMS/x86_64/
rpm -Uvh drbd-*
chkconfig off heartbeat
</source>

You should be good to go now!

== Allocating Raw Space For DRBD On Each Node ==

If you followed the setup steps provided for in "[[Two Node Fedora 13 Cluster]]", you will have a set amount of unconfigured hard drive space. This is what we will use for the DRBD space on either node. If you've got a different setup, you will need to allocate some raw space before proceeding.

=== Create a Simple Partition ===

If you do not have two drives, please follow the next section's steps, but pay attention to the "'''note'''"s. In short, you will need to create one partition, leave the default type of the partition as 83, write the changes to disk and the proceed to the [[#DRBD Configuration Files|DRBD Configuration Files]] section.

=== Creating a RAID level 1 'md' Device ===

This assumes that you have two raw drives, /dev/sda and /dev/sdb. It further assumes that you've created three partitions which have been assigned to three existing /dev/mdX devices. With these assumptions, we will create /dev/sda4 and /dev/sdb4 and, using them, create a new /dev/md3 device that will host the DRBD partition.

If you have multiple drives and plan to use a different [[TLUG_Talk:_Storage_Technologies_and_Theory#RAID_Levels|RAID levels]], please adjust the follow commands accordingly.

==== Creating The New Partitions ====

'''Warning''': The next steps will have you directly accessing your server's hard drive configuration. Please do not proceed on a live server until you've had a chance to work through these steps on a test server. One mistake can '''''blow away all your data'''''.

Start the fdisk shell for the first hard drive; /dev/sda.

<source lang="bash">
fdisk /dev/sda
</source>
<source lang="text">
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help):
</source>

'''Note''': Depending on your configuration, you may not see the above warning or you may see a different warning. Note it, but it is likely nothing to worry about it.

View the current configuration with the '''p'''rint option

<source lang="bash">
p
</source>
<source lang="text">
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6fe1

Device Boot Start End Blocks Id System
/dev/sda1 1 5100 40960000 fd Linux raid autodetect
/dev/sda2 5100 5622 4194304 fd Linux raid autodetect
/dev/sda3 * 5622 5654 256000 fd Linux raid autodetect

Command (m for help):
</source>

Now we know for sure that the next free partition number is 4. We will now create the '''n'''ew partition.

<source lang="bash">
n
</source>
<source lang="text">
Command action
e extended
p primary partition (1-4)
</source>

We will make it a '''p'''rimary partition

<source lang="bash">
p
</source>
<source lang="text">
Selected partition 4
First cylinder (5654-60801, default 5654):
</source>

Then we simply hit <enter> to select the default starting block.

<source lang="bash">
<enter>
</source>
<source lang="text">
Using default value 5654
Last cylinder, +cylinders or +size{K,M,G} (5654-60801, default 60801):
</source>

Once again we will press <enter> to select the default ending block.

<source lang="bash">
<enter>
</source>
<source lang="text">
Using default value 60801

Command (m for help):
</source>

'''Note''': If you only have one drive and are not creating a [[RAID]] array, you do not to change the type of the partition so you can skip the next few steps. Continue at the step where you write the changes.

Now we need to change the type of partition that it is.

<source lang="bash">
t
</source>
<source lang="text">
Partition number (1-4):
</source>

We know that we are modifying partition number 4.

<source lang="bash">
4
</source>
<source lang="text">
Hex code (type L to list codes):
</source>

Now we need to set the [[hex]] code for the [[Filesystem_List#List_of_Linux_Partition_Types|partition type]] to set. We want to set fd, which defines Linux raid autodetect.

<source lang="bash">
fd
</source>
<source lang="text">
Changed system type of partition 4 to fd (Linux raid autodetect)
</source>

Now check that everything went as expected by once again '''p'''rinting the partition table.

<source lang="bash">
p
</source>
<source lang="text">
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6fe1

Device Boot Start End Blocks Id System
/dev/sda1 1 5100 40960000 fd Linux raid autodetect
/dev/sda2 5100 5622 4194304 fd Linux raid autodetect
/dev/sda3 * 5622 5654 256000 fd Linux raid autodetect
/dev/sda4 5654 60801 442972704+ fd Linux raid autodetect

Command (m for help):
</source>

'''Note''': If you only have one drive, your partitions will be 83 Linux or 82 Linux swap / Solaris, instead of fd Linux raid autodetect.

There it is. So finally, we need to '''w'''rite the changes to the disk.

<source lang="bash">
w
</source>
<source lang="text">
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
</source>

'''Note''': If you only have one drive, reboot now if you got the message above and then skip forward to the "[[#DRBD Configuration Files|DRBD Configuration Files]]" section.

If you see the above message, '''do not''' reboot yet. repeat these steps for the second drive, /dev/sdb, and then reboot.

==== Creating The New /dev/mdX Device ====

''If you only have one drive, skip this step.''

Now we need to use mdadm to create the new [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]] device. This will be used as the device that DRBD will directly access.

<source lang="bash">
mdadm --create /dev/md3 --homehost=localhost.localdomain --raid-devices=2 --level=1 /dev/sda4 /dev/sdb4
</source>
<source lang="text">
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
</source>

Seeing as /boot doesn't exist on this device, we can safely ignore this warning.

<source lang="bash">
y
</source>
<source lang="text">
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/md4 started.
</source>

You can now cat /proc/mdstat to verify that it indeed built. If you're interested, you could open a new terminal window and use watch cat /proc/mdstat and watch the array build.

<source lang="bash">
cat /proc/mdstat
</source>
<source lang="text">
md3 : active raid1 sdb4[1] sda4[0]
442971544 blocks super 1.2 [2/2] [UU]
[>....................] resync = 0.8% (3678976/442971544) finish=111.0min speed=65920K/sec

md2 : active raid1 sda2[0] sdb2[1]
4193272 blocks super 1.1 [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]
40958908 blocks super 1.1 [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk

md0 : active raid1 sda3[0] sdb3[1]
255988 blocks super 1.0 [2/2] [UU]

unused devices: <none>
</source>

Finally, we need to make sure that the new array will start when the system boots. To do this, we'll again use mdadm, but with different options that will have it output data in a format suitable for the /etc/mdadm.conf file. We'll redirect this output to that config file, thus updating it.

<source lang="bash">
mdadm --detail --scan | grep md3 >> /etc/mdadm.conf
cat /etc/mdadm.conf
</source>
<source lang="text">
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=b58df6d0:d925e7bb:c156168d:47c01718
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=ac2cf39c:77cd0314:fedb8407:9b945bb5
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=4e513936:4a966f4e:0dd8402e:6403d10d
ARRAY /dev/md3 metadata=1.2 name=localhost.localdomain:3 UUID=f0b6d0c1:490d47e7:91c7e63a:f8dacc21
</source>

You'll note that the last line, which we just added, is different from the previous lines. This isn't a concern, but you are welcome to re-write it to match the existing format if you wish.

Before you proceed, it is strongly advised that you reboot each node and then verify that the new array did in fact start with the system. You ''do not'' need to wait for the sync to finish before rebooting. It will pick up where you left off once rebooted.

'''Note''': You'll notice we did not format a file system on this raid array, this is intentional. DRBD use the raw device and does not need a file system on it.

== DRBD Configuration Files ==

DRBD uses a global configuration file, /etc/drbd.d/global_common.conf, and one or more resource files. The resource files need to be created in the /etc/drbd.d/ directory and must have the suffix .res. For this example, we will create a single resource called r0 which we will configure in /etc/drbd.d/r0.res.

=== /etc/drbd.d/global_common.conf ===

The stock /etc/drbd.d/global_common.conf is sane, so we won't bother altering it here.

Full details on all the drbd.conf configuration file directives and arguments can be found [http://www.drbd.org/users-guide/re-drbdconf.html here]. '''Note''': That link doesn't show this new configuration format. Please see [http://www.novell.com/documentation/sle_ha/book_sleha/?page=/documentation/sle_ha/book_sleha/data/sec_ha_drbd_configure.html Novell's] link.

=== /etc/drbd.d/r0.res ===

This is the important part. This defines the resource to use, and must reflect the IP addresses and storage devices that DRBD will use for this resource.

<source lang="bash">
vim /etc/drbd.d/r0.res
</source>
<source lang="bash">
# This is the name of the resource and it's settings. Generally, 'r0' is used
# as the name of the first resource. This is by convention only, though.
resource r0
{
# This tells DRBD where to make the new resource available at on each
# node. This is, again, by convention only.
device /dev/drbd0;

# The main argument here tells DRBD that we will have proper locking
# and fencing, and as such, to allow both nodes to set the resource to
# 'primary' simultaneously.
net
{
allow-two-primaries;
}

# This tells DRBD to automatically set both nodes to 'primary' when the
# nodes start.
startup
{
become-primary-on both;
}

# This tells DRBD to look for and store it's meta-data on the resource
# itself.
meta-disk internal;

# The name below must match the output from `uname -n` on each node.
on an-node01.alteeve.com
{
# This must be the IP address of the interface on the storage
# network (an-node01.sn, in this case).
address 192.168.2.71:7789;

# This is the underlying partition to use for this resource on
# this node.
disk /dev/md3;
}

# Repeat as above, but for the other node.
on an-node02.alteeve.com
{
address 192.168.2.72:7789;
disk /dev/md3;
}
}
</source>

This file must be copied to '''BOTH''' nodes and must match before you proceed.

== Starting The DRBD Resource ==

From the rest of this section, pay attention to whether you see
* '''Node1'''
* '''Node2'''
* '''Both'''

These indicate which node to run the following commands on. There is no functional difference between either node, so just randomly choose one to be '''Node1''' and the other will be '''Node2'''. Once you've chosen which is which, be consistent with which node you run the commands on. Of course, if a command block is proceeded by '''Both''', run the following code block on both nodes.

=== Loading the 'drbd' Module ===

'''Both'''

Normally, we'd load the drbd module by simply starting the /etc/init.d/drbd daemon. However, if we did that at this stage, we'd generate errors because there isn't an UpToDate disk in the array. To get around this, we'll manually load the drbd kernel module using modprobe.

<source lang="bash">
modprobe drbd
</source>

This won't return any output, but if you check, you should now see the special /proc/drbd file.

=== Monitoring Progress ===

'''Both'''

I find it very useful to monitor DRBD while running the rest of the setup. To do this, open a second terminal on each node and use watch to keep an eye on /proc/drbd. This way you will be able to monitor the progress of the array in near-real time.

'''Both'''
<source lang="bash">
watch cat /proc/drbd
</source>

At this stage, it should look like this:

<source lang="text">
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:Unconfigured
</source>

=== Initialize The Resource ===

'''Both'''

This step creates the DRBD meta-data on the new DRBD resource's backing devices. It is only needed when creating new DRBD partitions.

<source lang="bash">
drbdadm create-md r0
</source>
<source lang="text">
--== Thank you for participating in the global usage survey ==--
The server's response is:

you are the 9507th user to install this version
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
</source>

The /proc/drbd output should not have changed at this stage.

=== Starting the Resource ===

'''Both'''

This will attach the backing device, /dev/md3 in our case, and then start the new resource r0.

<source lang="bash">
drbdadm up r0
</source>

There will be no output at the command line. If you are watching /proc/drbd though, you should now see something like this:

<source lang="text">
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:442957988
</source>

That it is Secondary/Secondary and Inconsistent/Inconsistent is expected.

=== Setting the First Primary Node ===

'''Node1'''

As this is a totally new resource, DRBD doesn't know which side of the array is "more valid" than the other. In reality, neither is as there was no existing data of note on either node. This means that we now need to choose a node and tell DRBD to treat it as the "source" node. This step will also tell DRBD to make the "source" node primary. Once set, DRBD will begin sync'ing in the background.

<source lang="bash">
drbdadm -- --overwrite-data-of-peer primary r0
</source>

As before, there will be no output at the command line, but /proc/drbd will change to show the following:

<source lang="text">
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:69024 nr:0 dw:0 dr:69232 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:442888964
[>....................] sync'ed: 0.1% (432508/432576)M
finish: 307:33:42 speed: 320 (320) K/sec
</source>

If you're watching the secondary node, the /proc/drbd will show ro:Secondary/Primary ds:Inconsistent/UpToDate. This is, as you can guess, simply a reflection of it being the "over-written" node.

=== Setting the Second Node to Primary ===

'''Node2'''

The last step to complete the array is to tell the second node to also become primary.

<source lang="bash">
drbdadm primary r0
</source>

As with many drbdadm commands, nothing will be printed to the console. If you're watching the /proc/drbd though, you should see something like Primary/Primary ds:UpToDate/Inconsistent. The Inconsistent flag will remain until the sync is complete.

=== A Note On sync Speed ===

You will notice in the previous step that the sync speed seems awfully slow at 320 (320) K/sec.

'''This is not a problem!'''

As actual data is written to either side of the array, that data will be immediately copied to both nodes. As such, both nodes will always contain up to date copies of the real data. Given this, the syncer is intentionally set low so as to not put too much load on the underlying disks that could cause slow downs. If you still wish to increase the sync speed, you can do so with the following command.

'''Warning''': If you set the DRBD sync speed too high and saturate your disks' maximum write speed, clvmd daemon will likely fail to start in some cases, leading to fences. For this reason, keep your sync rate to about 2/3rds of the underlying disk maximum write speed and hold off on bumping up the sync speed until you know you will have a period of low activity on your cluster.

<source lang="bash">
drbdsetup /dev/drbd0 syncer -r 35M
</source>

The speed-up will not be instant. It will take a little while for the speed to pick up. Once the sync is finished, it is a good idea to revert to the default sync rate.

<source lang="text">
drbdadm syncer r0
</source>

= Setting Up CLVM =

The goal of DRBD in the cluster is to provide clustered [[LVM]], referred to as [[CLVM]] to the nodes. This is done by turning the DRBD partition into an CLVM physical volume.

So now we will create a [[PV]] on top of the new [[DRBD]] partition, /dev/drbd0, that we created in the previous step. Since this new LVM [[PV]] will exist on top of the shared DRBD partition, whatever get written to it's logical volumes will be immediately available on either node, regardless of which node actually initiated the write.

This capability is the underlying reason for creating this cluster; Neither machine is truly needed so if one machine dies, anything on top of the DRBD partition will still be available. When the failed machine returns, the surviving node will have a list of what blocks changed while the other node was gone and can use this list to quickly re-sync the other server.

== Making LVM Cluster-Aware ==

Normally, LVM is run on a single server. This means that at any time, the LVM can write data to the underlying drive and not need to worry if any other device might change anything. In clusters, this isn't the case. The other node could try to write to the shared storage, so then nodes need to enable "locking" to prevent the two nodes from trying to work on the same bit of data at the same time.

The process of enabling this locking is known as making LVM "cluster-aware".

LVM has tool called lvmconf that can be used to enable LVM locking. This is provided as part of the lvm2-cluster package.

<source lang="bash">
yum -y install lvm2-cluster.x86_64
</source>

Now to enable cluster awareness in LVM, run to following command.

<source lang="bash">
lvmconf --enable-cluster
</source>

By default, clvmd, the cluster lvm daemon, is stopped and not set to run on boot. Now that we've enabled LVM locking, we need to start it:

<source lang="bash">
/etc/init.d/clvmd status
</source>
<source lang="text">
clvmd is stopped
active volumes: (none)
</source>

As expected, it is stopped, so lets start it:

'''Note''': At this point cman is still set to not start a boot. Since we rebooted after creating the partitions that make up /dev/md3, cman will likely, and in my case was still off. clvmd will fail to start because the cluster manager (cman) is not started. --[[User:SRSullivan|SRSullivan]] 17:40, 18 October 2010 (UTC)

<source lang="bash">
/etc/init.d/clvmd start
</source>
<source lang="text">
Activating VGs: No volume groups found
[ OK ]
</source>

'''Note''': I've seen on a few occasions where starting clvmd will time out and, on occasion, fences will be issued. I've not sorted out why, but I have usually been able to resolve this by stopping clvmd and cman, then restarting cman and, finally, restarting clvmd. If I can sort out a way to reliably trigger this problem, I will submit a bug report.

== Filtering Out Devices ==

'''ToDo''': Find a less-aggressive filter.

With the stock /etc/lvm/lvm.conf configuration, all devices on the system will be checked for LVM volumes. This can cause a problem as LVM will give preference to the LVM data on the RAID device over the DRBD device. It sees a duplicate as both are, effectively, one and the same.

To work around this, we need to alter the filter = [] entry. At the time of writing, simply rejecting the underlying /dev/md3 device as a candidate wasn't enough. So for now, we will tell LVM to accept DRBD devices and reject all other devices. To do this, we'll insert "a|/dev/drbd*|" as the first array entry and change the existing entry to "r/.*/".

'''Note''': I would love feedback on a filter argument that successfully ignored just /dev/md3, if anyone can suggest one.

<source lang="bash">
vim /etc/lvm/lvm.conf
</source>
<source lang="text">
# By default we accept every block device:
#filter = [ "a/.*/" ]
filter = [ "a|/dev/drbd*|", "r/.*/" ]
</source>

Now delete the existing cache file so that LVM is forced to rescan the system.

<source lang="bash">
rm -f /etc/lvm/cache/.cache
</source>

The changes take effect immediately.

== Creating a new PV using the DRBD Partition ==

'''Node1'''

We can now proceed with setting up the new DRBD-based LVM physical volume. Once the PV is created, we can create a new volume group and start allocating space to logical volumes.

'''Note''': As we will be using our DRBD device, and as it is a shared block device, most of the following commands only need to be run on one node. Once the block device changes in any way, those changes will near-instantly appear on the other node. For this reason, unless explicitly stated to do so, only run the following commands on one node.

To setup the DRBD partition as an LVM PV, run pvcreate:

<source lang="bash">
pvcreate /dev/drbd0
</source>
<source lang="text">
Physical volume "/dev/drbd0" successfully created
</source>

'''Both'''

Now, on both nodes, check that the new physical volume is visible by using pvdisplay:

<source lang="bash">
pvdisplay
</source>
<source lang="text">
"/dev/drbd0" is a new physical volume of "422.44 GiB"
--- NEW Physical volume ---
PV Name /dev/drbd0
VG Name
PV Size 422.44 GiB
Allocatable NO
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID YHmdip-SuJN-KIEv-2tbK-BT9Q-wfOo-OuQuaW
</source>

If you see PV Name /dev/drbd0 (or your underlying partition) on both nodes, then your DRBD setup and LVM configuration changes are working perfectly!

== Creating a VG on the new PV ==

'''Node1'''

Now we need to create the volume group using the vgcreate command:

<source lang="bash">
vgcreate -c y drbd0_vg0 /dev/drbd0
</source>
<source lang="text">
Clustered volume group "drbd0_vg0" successfully created
</source>

'''Both'''

Now we'll check that the new VG is visible on both nodes using vgdisplay:

<source lang="bash">
vgdisplay
</source>
<source lang="text">
--- Volume group ---
VG Name drbd0_vg0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
Clustered yes
Shared no
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 422.43 GiB
PE Size 4.00 MiB
Total PE 108143
Alloc PE / Size 0 / 0
Free PE / Size 108143 / 422.43 GiB
VG UUID Bb8l9e-es2z-PhaF-Gg3o-2is2-DZ1S-V2RsBF
</source>

If the new VG is visible on both nodes, we are ready to create our first logical volume using the lvcreate tool.

== Creating the First LV on the new VG ==

'''Node1'''

Now we'll create a simple 20 GiB logical volumes. We will use it as a shared GFS2 store for shared files and to store our Xen domU config files later on.

<source lang="bash">
lvcreate -L 20G -n xen_shared drbd0_vg0
</source>
<source lang="text">
Logical volume "xen_shared" created
</source>

'''Both'''

As before, we will check that the new logical volume is visible from both nodes by using the lvdisplay command:

<source lang="bash">
lvdisplay
</source>
<source lang="text">
--- Logical volume ---
LV Name /dev/drbd0_vg0/xen_shared
VG Name drbd0_vg0
LV UUID AqQizc-KBpX-2scN-WFLb-jIeF-QDcM-PlQW84
LV Write Access read/write
LV Status available
# open 0
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
</source>

Again, if this is visible from both nodes, we're set! Repeat this process for all future LVs you will want to create. We will do this a little later to create LVs for Xen VMs.

= Creating A Shared GFS FileSystem =

GFS is a cluster-aware file system that can be simultaneously mounted on two or more nodes at once. We will use it as a place to store ISOs that we'll use to provision our virtual machines.

== Install The GFS2 Utilities ==

Start by installing the [[GFS2]] tools:

<source lang="bash">
yum -y install gfs2-utils.x86_64
</source>

== Format Our CLVM LV With The GFS2 File System ==

'''Node1'''

'''Note''': The following example is designed for the cluster used in the prerequisite HowTo.
* If you have more than 2 nodes, increase the -j 2 to the number of nodes you want to mount this file system on.
* If your cluster is named something other than an-cluster (as set in the cluster.conf file), change -t an-cluster:xen_shared to match you cluster's name. The xen_shared can be whatever you like, but it must be unique in the cluster. I tend to use a name that matches the LV name, but this is my own preference and is not required.

To format the partition run:
<source lang="bash">
mkfs.gfs2 -p lock_dlm -j 2 -t an-cluster:xen_shared /dev/drbd0_vg0/xen_shared
</source>
<source lang="text">
This will destroy any data on /dev/drbd0_vg0/xen_shared.
It appears to contain: symbolic link to `../dm-0'

Are you sure you want to proceed? [y/n]
</source>

Acknowledge the warning, if any, and then press y if you are ready to proceed.

<source lang="bash">
y
</source>
<source lang="text">
Device: /dev/drbd0_vg0/xen_shared
Blocksize: 4096
Device Size 20.00 GB (5242880 blocks)
Filesystem Size: 20.00 GB (5242878 blocks)
Journals: 2
Resource Groups: 80
Locking Protocol: "lock_dlm"
Lock Table: "an-cluster:xen_shared"
UUID: A1487063-2A3F-43B1-3A36-44936B0B4D1E
</source>

Once the format completes, you can mount /dev/drbd0_vg0/xen_shared as you would a normal file system.

'''Both''':

To complete the example, lets mount the GFS2 partition we made just now on /shared and then use df -h to verify.

<source lang="bash">
mkdir /xen_shared
mount /dev/drbd0_vg0/xen_shared /xen_shared
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.8G 34G 8% /
tmpfs 466M 29M 438M 7% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
/dev/dm-0 20G 259M 20G 2% /xen_shared
</source>

You may have noticed that it shows /dev/dm-0 instead of /dev/drbd0_vg0/xen_shared. If you look at the later, you will see that it is simply a [[symlink]] to the former.

<source lang="bash">
ls -lah /dev/drbd0_vg0/xen_shared
</source>
<source lang="text">
lrwxrwxrwx. 1 root root 7 Sep 9 13:24 /dev/drbd0_vg0/xen_shared -> ../dm-0
</source>

== Add An Entry To /etc/fstab ==

The last step is to add an entry for this new partition to each node's /etc/fstab file.

=== Reference The GFS2 Partition By Device Path ===

This is the more traditional method of referencing the GFS2 partition by using it's device path directly.

'''Warning''': An incorrect edit of the /etc/fstab file can leave your system unable to boot! Please review the line generated above to make sure it is accurate and compatible with your setup before proceeding.

<source lang="bash">
vim /etc/fstab
</source>
<source lang="text">
#
# /etc/fstab
# Created by anaconda on Tue Sep 7 06:29:51 2010
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=865690db-85d0-44c5-9f32-ffb6fdf47060 / ext3 defaults 1 1
UUID=8b9822b6-a92e-48c9-96b5-f8943142319e /boot ext3 defaults 1 2
UUID=94b03547-a7e3-45bb-b2d5-837498b370f4 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
xenfs /proc/xen xenfs defaults 0 0
/dev/drbd0_vg0/xen_shared /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

=== Reference The GFS2 Partition By UUID ===

It is sometimes preferable to create an fstab entry that locates the device path via it's [[UUID]]. To do this, you can run the following command which, though a bit cryptic, will print out an /etc/fstab compatible string.

'''Warning''': The same warnings apply here as above

<source lang="bash">
echo `gfs2_edit -p sb /dev/drbd0_vg0/xen_shared | grep sb_uuid | sed -e "s/.*sb_uuid *$.*$/UUID=\L\1\E \/xen_shared\t\tgfs2\trw,suid,dev,exec,nouser,async\t0 0/"`
</source>
<source lang="text">
UUID=a1487063-2a3f-43b1-3a36-44936b0b4d1e /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

You may have noticed that defaults isn't used. Rather, all but the auto option are manually set. This is because the system will drop to single-user mode at boot if it can't mount an auto partition at boot time (auto being implied by defaults). Given that our GFS2 partition sits on top of DRBD and the cluster, there is no way to make it available that early in the boot process.

Further, the gfs2 init script specifically excludes entries in /etc/fstab that have the 'noauto option set. For this reason, we can't simply specify that as we need the init script to see the partition so that it is mounted when GFS2 starts and unmounted when it stops.

Now add this string to /etc/fstab.

<source lang="bash">
vim /etc/fstab
</source>
<source lang="text">
#
# /etc/fstab
# Created by anaconda on Tue Sep 7 06:29:51 2010
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=865690db-85d0-44c5-9f32-ffb6fdf47060 / ext3 defaults 1 1
UUID=8b9822b6-a92e-48c9-96b5-f8943142319e /boot ext3 defaults 1 2
UUID=94b03547-a7e3-45bb-b2d5-837498b370f4 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
xenfs /proc/xen xenfs defaults 0 0
UUID=a1487063-2a3f-43b1-3a36-44936b0b4d1e /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

Please note; At the time of writing this HowTo, there is a bug in findfs and mount. According to [http://www.ietf.org/rfc/rfc4122.txt RFC 4122], programs should accept a [[UUID]] in either upper or lower case. However, this is not currently the case, so you '''must''' pass the UUID in lower-case. Please see bugs [https://bugzilla.redhat.com/show_bug.cgi?id=632373 632373] and [https://bugzilla.redhat.com/show_bug.cgi?id=632385 632385].

== Testing The gfs2 Initialization Script ==

To verify that the new entry is valid, check gfs2's status.

<source lang="bash">
/etc/init.d/gfs2 status
</source>
<source lang="text">
Configured GFS2 mountpoints:
/xen_shared
Active GFS2 mountpoints:
/xen_shared
</source>

Now test stopping and restarting to ensure that the GFS2 partition unmounts and mounts properly.

Stop gfs2.

<source lang="bash">
/etc/init.d/gfs2 stop
</source>
<source lang="text">
Unmounting GFS2 filesystem (/xen_shared): [ OK ]
</source>

Check with df -h to ensure that the mount is gone.

<source lang="bash">
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.0G 35G 6% /
tmpfs 466M 23M 444M 5% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
</source>

Start gfs2 again.

<source lang="bash">
/etc/init.d/gfs2 start
</source>
<source lang="text">
Mounting GFS2 filesystem (/xen_shared): [ OK ]
</source>

Again, check with df -h that it has been remounted.

<source lang="bash">
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.0G 35G 6% /
tmpfs 466M 29M 438M 7% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
/dev/dm-0 20G 259M 20G 2% /xen_shared
</source>

Done!

== Further Reading ==

* [[Grow a GFS2 Partition]]
* [[Hard drive has gone bad in DRBD]]

= Altering Daemon Start Order =

It is important that the various daemons in use by our cluster start in the right order. Most daemons will rely on services provided by another daemon to be running, and will not start or will not operate reliably otherwise.

We need to make sure that xend starts so that the network is stable. Then cman needs to start so that [[fencing]] and [[dlm]] are available. Next, drbd starts so that the clustered storage is available. Then clvmd must start so that the data on the DRBD resource is accessible. Now gfs2 needs to start so that the Xen domU configuration files can be found and finally xendomains must start to boot up the actual domU virtual machines.

To restate as a list, the start order must be:
* xend
* cman
* drbd
* clvmd
* gfs2
* xendomains

To make sure the start order is sane then, we'll edit each of the six daemon's init scripts and alter their Required-Start lines. To make the changes take effect, we will use chkconfig to remove and re-add them to the various start levels.

== Altering xend ==

This should already be done. If it isn't, please see "[[#Make xend play nice with clustering|Making xend play nice with clustering]]" above. If you are revisiting that section, you can skip the cman edit as we will need to make another change in the next step.

== Altering cman ==

This should already be done. If it isn't, please see "[[#Make xend play nice with clustering|Making xend play nice with clustering]]" above. If you are revisiting that section, you can skip the cman edit as we will need to make another change in the next step.

== Altering drbd ==

Now we will tell drbd to start after cman.

This requires the additional step of altering the chkconfig: - 70 08 line to instead read chkconfig: - 20 08. This isn't strictly needed, but will give more room for chkconfig to order the dependent daemons by allowing DRBD to be started as low as position 20, rather than waiting until position 70. This is somewhat more compatible with cman and clvmd which normally start at positions 21 and 24, respectively

<source lang="bash">
vim /etc/init.d/drbd
</source>
<source lang="text">
#!/bin/bash
#
# chkconfig: - 20 08
# description: Loads and unloads the drbd module
#
# Copright 2001-2008 LINBIT Information Technologies
# Philipp Reisner, Lars Ellenberg
#
### BEGIN INIT INFO
# Provides: drbd
# Required-Start: $local_fs $network $syslog cman
# Required-Stop: $local_fs $network $syslog
# Should-Start: sshd multipathd
# Should-Stop: sshd multipathd
# Default-Start:
# Default-Stop:
# Short-Description: Control drbd resources.
### END INIT INFO
</source>

== Altering clvmd ==

Now we will now tell clvmd to start after drbd.

'''Note''': There is currently a minor bug with lvm2-cluster version 2.02.73-2 in that /etc/init.d/clvmd is set by default to mode 0555. This is easily corrected by running the following command. Please check bug [https://bugzilla.redhat.com/show_bug.cgi?id=636066 636066] to see if this has been resolved.

<source lang="bash">
chmod u+w /etc/init.d/clvmd
</source>

Once you've got write access, edit the file.

<source lang="bash">
vim /etc/init.d/clvmd
</source>
<source lang="text">
#!/bin/bash
#
# chkconfig: - 24 76
# description: Starts and stops clvmd
#
# For Red-Hat-based distributions such as Fedora, RHEL, CentOS.
#
### BEGIN INIT INFO
# Provides: clvmd
# Required-Start: $local_fs drbd
# Required-Stop: $local_fs
# Default-Start:
# Default-Stop: 0 1 6
# Short-Description: Clustered LVM Daemon
### END INIT INFO
</source>

== Altering gfs2 ==

Now we will now tell gfs2 to start after clvmd. You will notice that cman is already listed under Required-Start and Required-Stop. It's true that cman must be started, but we've created a chain here so we can safely replace it with clvmd in the start line.

<source lang="bash">
vim /etc/init.d/gfs2
</source>
<source lang="text">
#!/bin/bash
#
# gfs2 mount/unmount helper
#
# chkconfig: - 26 74
# description: mount/unmount gfs2 filesystems configured in /etc/fstab

### BEGIN INIT INFO
# Provides: gfs2
# Required-Start: $network clvmd
# Required-Stop: $network
# Default-Start:
# Default-Stop:
# Short-Description: mount/unmount gfs2 filesystems configured in /etc/fstab
# Description: mount/unmount gfs2 filesystems configured in /etc/fstab
### END INIT INFO
</source>

== Altering xendomains ==

Finally, we will alter xendomains so that it starts last, after gfs2.

<source lang="bash">
vim /etc/init.d/xendomains
</source>
<source lang="text">
#!/bin/bash
#
# /etc/init.d/xendomains
# Start / stop domains automatically when domain 0 boots / shuts down.
#
# chkconfig: 345 99 00
# description: Start / stop Xen domains.
#
# This script offers fairly basic functionality. It should work on Redhat
# but also on LSB-compliant SuSE releases and on Debian with the LSB package
# installed. (LSB is the Linux Standard Base)
#
# Based on the example in the "Designing High Quality Integrated Linux
# Applications HOWTO" by Avi Alkalay
# <http://www.tldp.org/HOWTO/HighQuality-Apps-HOWTO/>
#
### BEGIN INIT INFO
# Provides: xendomains
# Required-Start: $syslog $remote_fs xend gfs2
# Should-Start:
# Required-Stop: $syslog $remote_fs xend
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop secondary xen domains
# Description: Start / stop domains automatically when domain 0
# boots / shuts down.
### END INIT INFO
</source>

== Applying The Changes ==

Change the start order by removing and re-adding all cluster-related daemons using chkconfig.

<source lang="bash">
chkconfig xenstored off; chkconfig xenconsoled off; chkconfig xend off; chkconfig cman off; chkconfig drbd off; chkconfig clvmd off; chkconfig gfs2 off; chkconfig xendomains off
chkconfig xendomains on; chkconfig gfs2 on; chkconfig clvmd on; chkconfig drbd on; chkconfig cman on; chkconfig xend on; chkconfig xenconsoled on; chkconfig xenstored on
</source>

Now verify that the start order is as we want it.

<source lang="bash">
ls -lah /etc/rc3.d/
</source>
<source lang="text">
lrwxrwxrwx. 1 root root 19 Sep 20 13:37 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 21 Sep 20 13:37 S27xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S28xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S29cman -> ../init.d/cman
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S70drbd -> ../init.d/drbd
lrwxrwxrwx. 1 root root 15 Sep 20 13:37 S71clvmd -> ../init.d/clvmd
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S72gfs2 -> ../init.d/gfs2
lrwxrwxrwx. 1 root root 20 Sep 20 13:37 S99xendomains -> ../init.d/xendomains
</source>

= Setting Up Xen =

'''''WARNING''''': Everything below here is pretty '''seriously screwed up'''.

'''Note''': This is not meant to be an extensive tutorial on Xen itself. It covers enough to get domU VMs provisioned in a manner that will take advantage of the cluster. As such, there is minimal explanation of configuration file options. If you need further help, please drop by the ##xen (yes, two ##) [[IRC]] channel on freenode.org.

== Install The Hypervisor Tools ==

These tools are very useful in provisioning and managing domU VMs.

<source lang="bash">
yum -y install virt-install virt-viewer
</source>

== Install The HVM/KVM Tools ==

For hvm (Hardware Virtual Machines), which is required for [http://www.virtuatopia.com/index.php/Virtualizing_Windows_Server_2008_with_Xen paravirtualized Microsoft] VMs, you must install the following packages as well.

<source lang="bash">
yum -y install qemu-kvm.x86_64 qemu-kvm-tools.x86_64
</source>

== Ensure That Virtualization Is Enabled ==

Many motherboards disable hvm by default in their [[BIOS]]. Assuming that you've got a dom0 kernel running at this stage, you can check if this is the case by checking the xm info output.

<source lang="bash">
xm info
</source>
<source lang="text">
host : an-node04.alteeve.com
release : 2.6.32.23-170.dom0_an1.fc13.x86_64
version : #1 SMP Sun Oct 10 20:39:19 EDT 2010
machine : x86_64
nr_cpus : 4
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 2209
hw_caps : 178bf3ff:efd3fbff:00000000:00001310:00802001:00000000:000037ff:00000000
virt_caps : hvm
total_memory : 4063
free_memory : 2987
node_to_cpu : node0:0-3
node_to_memory : node0:2987
node_to_dma32_mem : node0:2928
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline : dom0_mem=1024M
cc_compiler : gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC)
cc_compile_by : root
cc_compile_domain :
cc_compile_date : Mon Oct 11 01:10:38 EDT 2010
xend_config_format : 4
</source>

Look at the virt_caps and xen_caps lines. Notice the hvm entries? This shows that HVM, also known as "secure virtualization", has been enabled. If you do not see this, please check your mainboard manual for information on enabling this on your system.

'''Note''': The next paragraph applies only when running a vanilla kernel.

If you are running a vanilla kernel, you can check to see if your CPU has support for HVM guests but checking /proc/cpuinfo. What you're looking for depends on your CPU manufacturer. If you have an Intel CPU, you need to look for the vmx flag. Likewise, with AMD CPUs, you need to look for the svm flag.

For a more complete, if somewhat dated paper on this topic, please [http://fedoraproject.org/wiki/FedoraXenQuickstartFC6#Fully-virtualized_guests_.28HVM.2FIntel-VT.2FAMD-V.29 Fedora 6 Xen Quickstart Guide, System Requirements].

== Enabling Migration ==

By default, xend will not allow domU VMs from being migrated onto or off of a given dom0 host. Given that we've got a cluster though, we very much want this behaviour, so now we will enable it. This is done by making edits to /etc/xen/xend-config.sxp. Below is a concise list of options that must be set. Some exist already in the file and need to be commented out or altered.

'''Warning''': The values below are '''very''' permissive. Please review each option and improve the security to fit your network before going into production!

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>
<source lang="text">
(xend-http-server yes)
(xend-unix-server yes)
(xend-tcp-xmlrpc-server yes)
(xend-relocation-server yes)
(xend-udev-event-server yes)
(xend-port 8000)
(xend-relocation-port 8002)
(xend-address '')
(xend-relocation-address '')
(xend-relocation-hosts-allow '')
</source>

Once done, restart xend. It is usually safest to stop the cluster before hand to avoid accidental fencing caused by the underlying network being reconfigured.

<source lang="bash">
/etc/init.d/gfs2 stop
/etc/init.d/clvmd stop
/etc/init.d/cman stop
/etc/init.d/xend restart
/etc/init.d/cman start
/etc/init.d/clvmd start
/etc/init.d/gfs2 start
</source>

== Virtual Machine Naming Convention ==

'''Note''': This section acts as a recommendation only. Feel free to alter this to fit your style and needs.

Personally, I like to name my VMs similar to c5_shorewall_01. To elaborate, I like to use the format:
* os_role_seq (<Operating System ID>_<Role of the VM>_<Sequence Integer>).

There are no (known) restrictions on virtual machine names, so feel free to use names that made sense for you. I do strongly recommend that you match the name of your domU VM to the name of it's host LVM logical volume.

== Provisioning domU VMs ==

There are two ways to provision new VMs that we will cover (there are many others); Using virt-install and using xm create -f /path/to/domain.cfg where domain.cfg is a hand-crafted python script.

=== Provisioning with virt-install ===

This uses a long command line argument that provisions the VM and loads it into libvirt. Where possible, this is probably the best way to provision a domU. However, there are occasions where it may not work.

Here is an example where a domU is provisioned for a [[Fedora]] 13, x86_64 VM with a dedicated logical volume on the clustered LVM. The command to create the LV precedes the command to provision the VM. Please review and adjust values as you need. Consult man virt-install for a more complete list of available options and their uses.

Following the provision command are two examples of how to backup the configuration to a flat file. The first directly dumps the configuration into the libvirt format. The second backs up the configuration to an [[XML]] file and then shows and example of how that file can be converted into a standard python script.

<source lang="bash">
# Fedora 13 x86_64 RPM builder VM
lvcreate -L 40G -n f13_builder_01 drbd0_vg0
virt-install --connect xen \
--name f13_builder_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--location http://192.168.1.10/f13/x86_64/img/ \
--os-type linux \
--os-variant fedora13 \
--disk path=/dev/drbd0_vg0/f13_builder_01 \
--network bridge=eth0,mac=00:16:3e:00:10:01 \
--vnc \
--paravirt
# Backup the domU configuration to the libvirt native format.
xm list -l f13_builder_01 > /xen_shared/dom_config/f13_builder.cfg
# Backup the config to an XML file.
virsh dumpxml f13_builder_01 > /xen_shared/domU_config/f13_builder_01.xml
# Convert it to a "traditional" python script. Be sure the edit the resulting .cfg file to remove the 'vifname=' sections.
virsh -c xen:/// domxml-to-native xen-xm f13_builder_01.xml > f13_builder_01.cfg
</source>

The next two examples show the provisioning of a [[CentOS]] v5.5 and [[RHEL]] v6.0, beta 2 machines. The dump and backup methods above should be easily adapted to work with these VMs.

<source lang="bash">
# A CentOS test server
lvcreate -L 40G -n c5_test_01 drbd0_vg0
virt-install --connect xen \
--name c5_test_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1-3 \
--location http://192.168.1.10/c5/x86_64/img/ \
--os-type linux \
--os-variant rhel5.4 \
--disk path=/dev/drbd0_vg0/c5_test_01 \
--network bridge=eth0,mac=00:16:3e:00:10:02 \
--vnc \
--paravirt
</source>

<source lang="bash">
# Red Hat Enterprise Linux 6 beta 2 test server
lvcreate -L 40G -n rh6b2_test_01 drbd0_vg0
virt-install --connect xen \
--name rh6b2_test_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--location http://192.168.1.10/rhel6/x86_64/img/ \
--os-type linux \
--os-variant rhel6 \
--disk path=/dev/drbd0_vg0/rh6b2_test_01 \
--network bridge=eth0,mac=00:16:3e:00:10:03 \
--vnc \
--paravirt
</source>

=== Provisioning With 'xm create' ===

At the time of writing this, I could not sort out the magical incantation for provisioning a [[Windows]] VM using virt-install. Instead, I used the "old" style of crafting a configuration file using a python script. This is useful to know as many templates exist on the web for various VMs. Following the steps below, you should be able to fairly easily adapt them.

In this example, we will provision a VM using [[HVM]] from an [[ISO]] image of the Windows 2008 Server installation DBD. In this case, there is a problem with virt-install finding the qemu-dm file.

This configuration file will be saved in the /xen_shared/domU_config directory, which exists on the shared [[GFS2]] partition we created earlier.

<source lang="bash">
mkdir /xen_shared/domU_config
vim /xen_shared/domU_config/win2008_sql_01.cfg
</source>
<source lang="python">
# This is the Windows 2008 Enterprise Server x86_64 hosting MS SQL Server 2008 Enterprise
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
memory = 1024

# Should be at least 2KB per MB of domain memory, plus a few MB per vcpu.
shadow_memory = 8
name = "win2008_sql_01"
#vif = [ 'type=ioemu, bridge=xenbr0' ]
vif = [ 'type=ioemu, bridge=eth0,mac=00:16:3e:00:30:03' ]
acpi = 1
apic = 1
# Remove the 'file:...' entry (or change it to another ISO) after the install is complete.
disk = [ 'phy:/dev/drbd0_vg0/win2008_sql_01,hda,w', 'file:/xen_shared/iso/MS-Win2008-Ent-x86_64-SP2.iso,hdc:cdrom,r' ]

device_model = '/usr/lib/xen/bin/qemu-dm'

#-----------------------------------------------------------------------------
# boot on floppy (a), hard disk (c) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
boot="dc"
sdl=0
vnc=1
vncconsole=1
vncpasswd=''

serial='pty'
usbdevice='tablet'
</source>

Now provision the VM using xm create.

<source lang="bash">
xm create -f /xen_shared/domU_config/win2008_sql_01.cfg
</source>

At this point, the VM is not loaded into libvirt. This is a problem as, on boot, the node will not know that the VM exists. As a consequence, some tools like virt-manager will not see the VM until it is manually started with xm create -f /xen_shared/domU_config/domain.cfg. Further, and perhaps more troubling, changes made to the VM's config made outside to config file will be lost when you restart from the config file.

To fix this, we'll use a few hacks to load the config into libvirt. This needs to be done while the domU is loaded and it must be done on the node currently hosting the VM.

First, make sure that the domU's configuration is visible. Then, dump the config and filter it through grep and sed to pull out the [[UUID]]. Once you know that you get the proper UUID, create the directory under /var/lib/xend/domains/ and then create the config.sxp file with the domU's configuration.

'''NOTE''': Be sure to change win2008_01 in the examples below to match the name of the domU that you want to setup.

<source lang="bash">
xm list -l win2008_01
xm list -l win2008_01 | grep '^ (uuid' | sed -e "s/ (uuid $.*$)/\1/"
mkdir /var/lib/xend/domains/`xm list -l win2008_01 | grep '^ (uuid' | sed -e "s/ (uuid $.*$)/\1/"`
cd /var/lib/xend/domains/`xm list -l win2008_01 | grep '^ (uuid' | sed -e "s/ (uuid $.*$)/\1/"`
xm list -l win2008_01 > config.sxp
</source>

There are several uuid strings in the output from xm list -l domain. Thankfully, the one we want is the only one indented by four spaces. This is how we can be fairly confident that the UUID returned by sed above is, in fact,


<source lang="bash">
</source>
<source lang="text">
</source>

{{footer}}

Talk:Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

2010-10-15T20:05:10Z

SRSullivan: /* Dealing with Raid 5/6 */

Talk:Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

2010-10-15T20:04:54Z

SRSullivan: /* Dealing with Raid 5/6 */ Sig

Talk:Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

2010-10-15T20:04:35Z

SRSullivan: Copy Paste failure in insturctions when dealing with a non Raid 1

Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

2010-10-15T17:52:33Z

SRSullivan: /* Creating The New /dev/mdX Device */ Clarication that no file system should be made.

{{howto_header}}

'''Warning''': This is currently a dumping ground for notes. '''''DO NOT FOLLOW THIS DOCUMENT'S INSTRUCTIONS'''''. Seriously, it could blow up your computer or cause winter to come early.

----

This HowTo will walk you through setting up [[Xen]] [[VM]]s using [[DRBD]] and [[CLVM]] for high availability.

= Prerequisite =

This talk is an extension of the [[Two Node Fedora 13 Cluster]] HowTo. As such, you will be expected to have a freshly built two-node cluster with spare disk space on either node.

Please do not proceed until you have completed the first tutorial.

= Overview =

This tutorial will cover several topics; [[DRBD]], [[CLVM]], [[GFS2]], [[Xen]] [[dom0]] and [[domU]] [[VM]]s and [[rgmanager]]. Their relationship is thus:

* DRBD provides a mechanism to replicate data across both nodes in real time and guarantees a consistent view of that data from either node. Think of it like [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]], but across machines.
* CLVM sits on the DRBD partition and provides the underlying mechanism for allowing both nodes to access shared data in a clustered environment. It will host a shared filesystem by way of GFS2 as well as [[LV]]s that Xen's domU VMs will use as their disk space.
* GFS2 will be the clustered file system used on one of the DBRD-backed, CLVM-managed partitions. Files that need to be shared between nodes, like the Xen VM configuration files, will exist on this partition.
* Xen will be the hypervisor in use that will manage the various virtual machines. Each virtual machine will exist in an LVM LV.
** Xen's dom0 is the special "host" virtual machine. In this case, dom0 will be the OS installed in the first HowTo.
** Xen's domU virtual machines will be the "floating", highly available servers.
* Lastly, rgmanager will be the component of [[cman]] that will be configured to manage the automatic migration of the virtual machines when failures occur and when nodes recover.

= Setting Up Xen =

It may seem odd to start with [[Xen]] at this stage, but it is going to rather fundamentally alter each node's "host" operating system.

At this point, each node's host OS is a traditional operating system operating on the bare metal. When we install a dom0 kernel though, we tell Xen to boot a mini operating system first, and then to boot our "host" operating system. In effect, this converts the host node's operating system into just another virtual machine, albeit with a special view of the underlying hardware and Xen hypervisor.

This conversion is somewhat disruptive, so I like to get it out of the way right away. We will then do the rest of the setup before returning to Xen later on to create the floating virtual machines.

== A Note On The State Of Xen dom0 Support In Fedora ==

As of Fedora 8, support for Xen [[dom0]] has been removed, but support for the hypervisor and [[domU]] virtual machines remains. Red Hat's position is that KVM will be the supported platform going forward. That said, [http://fedoraproject.org/wiki/Features/XenPvopsDom0 this page] seems to indicate that PV Ops dom0 kernels will be supported in the future. Specifically, when dom0 support is merged into the mainline Linux kernel. When this will be is open to speculation, though "by Fedora 16" seems to be a reasonable educated guess.

What this means for us is that we need to use a non-standard dom0 kernel. Specifically, we will use a kernel created by [http://myoung.fedorapeople.org/ myoung] (Micheal Young) for Fedora 12. This kernel does not directly support DRBD, so be aware that we will need to build new DRBD kernel modules for his kernel and then rebuild the DRBD modules each time his kernel is updated.

== A Note On Rolling Your Own RPMs ==

If you want to roll the source RPMs for both the hypervisor and the [[dom0]] kernel, you will need to make both before you can boot into your dom0 for the first time. This is because the dom0 kernel needs the Xen microkernel provided by the xen hypervisor package to boot.

== Install The Hypervisor ==

We will use [[Xen]] 4.0.1, as provided by [http://pasik.reaktio.net/fedora/ Pasik]. We'll use the [http://pasik.reaktio.net/fedora/xen-4.0.1-0.2.fc13.src.rpm source RPM] to build our own RPM.

Regardless of whether you install from source RPMs or the pre-compiled ones, you will need to install the libvirt, qemu, SDL and PyXML packages from the standard repositories before you can proceed.

<source lang="bash">
yum -y install libvirt PyXML.x86_64 qemu.x86_64 SDL.x86_64
</source>

Please don't remove existing xen utilities and libraries prior to installing the newer version. If you do, core clustering components may be removed. Instead, be sure to use the rpm -Uvh switches to upgrade the existing packages that may be installed already.

=== Installing Prebuilt RPMs ===

These are locally stored copies of the Xen RPMs built for [[x86_64]] on Fedora 13. This requires some dependent packages be installed first.

'''Warning''': These are all recompiled for this website against Fedora 13, x86_64. If you feel more comfortable, please use [[RPM]]s from a source you are familiar with.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/xen-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-doc-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-hypervisor-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-libs-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-runtime-4.0.1-0.2.fc13.x86_64.rpm
rpm -Uvh xen*4.0.1-0.2.fc13.x86_64.rpm
</source>

Optionally, you may wish to install debug and/or devel RPMs as well.

<source lang="bash">
wget -c https://alteeve.com/files/xen-debuginfo-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/xen-devel-4.0.1-0.2.fc13.x86_64.rpm
rpm -Uvh xen-d*4.0.1-0.2.fc13.x86_64.rpm
</source>

=== Building RPMs From Source ===

To build the RPMs from the source, you need to make sure that you have the build environment installed.

'''Note''': If you are following these instruction after having installed a prior dom0 kernel, you will need to comment out exclude=kernel* in /etc/yum.conf so that the kernel-header package can be installed.

<source lang="bash">
vim /etc/yum.conf
</source>
<source lang="text">
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
#exclude=kernel*
</source>

Now install the development environment.

<source lang="bash">
yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
yum install transfig texi2html SDL-devel libX11-devel tetex-latex gtk2-devel libaio-devel dev86 iasl xz-devel e2fsprogs-devel glibc-devel.i686 xmlto asciidoc elfutils-libelf-devel
</source>

'''Note''': If you had to uncomment the exclude=kernel* line earlier, comment it back out now before you forget.

<source lang="bash">
vim /etc/yum.conf
</source>
<source lang="text">
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
exclude=kernel*
</source>

Now we can now build the RPMs from source.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/xen-4.0.1-0.2.fc13.src.rpm
rpm -ivh xen-4.0.1-0.2.fc13.src.rpm
cd /root/rpmbuild/SPECS/
rpmbuild -ba xen.spec
</source>

Once this is done, you will have seven RPMs built. Two of them are not really needed (xen-debuginfo-4.0.1-0.2.fc13.x86_64.rpm and xen-devel-4.0.1-0.2.fc13.x86_64.rpm) and I will leave it up to you may wish to install them or not. If you don't, modify the next command or move the debug and devel RPMs out of the way first.

<source lang="bash">
cd ~/rpmbuild/RPMS/x86_64/
rpm -Uvh xen*4.0.1-0.2.fc13.x86_64.rpm
</source>

== Installing The AN!Cluster dom0 Kernel ==

'''Notice'''!

The kernel provided here was recompiled on Fedora 13 and is a slightly modified version of [[#Installing Micheal Young's dom0 Kernel|Micheal Young's]] kernel available below. I was originally driven to recompile in an effort to solve a DRBD-related kernel oops. For now, unless you have the same DRBD kernel oops, I'd strongly recommend against using the AN!Cluster dom0 kernel until it has been tested much more thoroughly.

With that warning out of the way...

This kernel was compiled on a Fedora 13, x86_64. The DRBD RPMs available a little later where compiled against this dom0 kernel.

'''Note''': The --force is required because the current kernel is newer than 2.6.32 used here. Without this switch, the RPM would not install.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
</source>

If you would like to install the debug, devel and/or header RPMs for this kernel, they are available below.

'''Note''': The kernel-debuginfo-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm RPM is 213 [[MiB]].

<source lang="bash">
wget -c https://alteeve.com/files/an-cluster/kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-debuginfo-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-de*-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
</source>

=== Post AN!Cluster dom0 Install Configuration ===

The entry in grub's /boot/grub/menu.lst won't work. You will need to edit it so that it calls the existing installed operating system as a module.

'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your root= is likely different and your rd_MD_UUID= will definitely be different, even on the same machine across installs. Generally speaking, what follows the kernel /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ... entry made by the dom0 kernel can be copied after the module /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ... entry in the example below.

<source lang="bash">
vim /boot/grub/menu.lst
</source>
<source lang="bash">
title Xen 4.0.x, Linux 2.6.32.23-170.dom0_an1.fc13.x86_64
root (hd0,0)
kernel /xen.gz dom0_mem=1024M
module /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ...
module /initramfs-2.6.32.23-170.dom0_an1.fc13.x86_64.img

</source>

== Installing Micheal Young's dom0 Kernel ==

This uses a kernel built for Fedora 12, but it works on Fedora 13. This step involves either installing it over HTML or adding and enabling his repository and then installing it from there.

=== Installing Via myoung's Repository ===

This is almost always the preferred method. However, do note that when myoung updates his kernel, there will be a lag where the dom0 dependent RPMs provided here will no longer be compatible.

To add the repository, download the myoung.dom0.repo into the /etc/yum.repos.d/ directory.

<source lang="bash">
cd /etc/yum.repos.d/
wget -c http://myoung.fedorapeople.org/dom0/myoung.dom0.repo
</source>

To enable his repository, edit the repository file and change the two enabled=0 entries to enabled=1.

<source lang="bash">
vim /etc/yum.repos.d/myoung.dom0.repo
</source>
<source lang="text">
[myoung-dom0]
name=myoung's repository of Fedora based dom0 kernels - $basearch
baseurl=http://fedorapeople.org/~myoung/dom0/$basearch/
enabled=1
gpgcheck=0

[myoung-dom0-source]
name=myoung's repository of Fedora based dom0 kernels - Source
baseurl=http://fedorapeople.org/~myoung/dom0/src/
enabled=1
gpgcheck=0
</source>

Install the [[Xen]] [[dom0]] kernel (edit the version number with the updated version if it has changed).

<source lang="bash">
yum install kernel-2.6.32.21-170.xendom0.fc12.x86_64
</source>

=== Post Michael Young's dom0 Install Configuration ===

The entry in grub's /boot/grub/menu.lst won't work. You will need to edit it so that it calls the existing installed operating system as a module.

'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your root= is likely different and your rd_MD_UUID= will definitely be different, even on the same machine across installs. Generally speaking, what follows the kernel /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ... entry made by the dom0 kernel can be copied after the module /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ... entry in the example below.

<source lang="bash">
vim /boot/grub/menu.lst
</source>
<source lang="bash">
title Xen 4.0.x, Linux kernel 2.6.32.21-170.xendom0.fc12.x86_64
root (hd0,0)
kernel /xen.gz dom0_mem=1024M
module /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ...
module /initramfs-2.6.32.21-170.xendom0.fc12.x86_64.img
</source>

=== Disabling Automatic Kernel Updates ===

Seeing as we're using an older kernel, yum will want to replace it whenever there is an updated kernel* package available. Likewise if myoung updates his kernel. In the latter case, the updated kernel from Mr. Young would break compatibility with our DRBD module. So to be safe, we want to tell yum to never update the kernel.

To do this, we need to add exclude=kernel* to the /etc/yum.conf file.

<source lang="bash">
echo "exclude=kernel*" >> /etc/yum.conf
cat /etc/yum.conf
</source>
<source lang="text">
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=3
color=never

# This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
# It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m

# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d

exclude=kernel*
</source>

=== Make xend play nice with clustering ===

By default under Fedora 13, cman will start before xend. This is a problem because xend takes the network down as part of it's setup. This causes [[totem]] communication to fail which leads to fencing.

'''Note''': Move xenconsoled and xenstore to 09 and 10 start positions and then make xend depend on the before starting.

To avoid this, edit the initialization scripts for /etc/init.d/xend and it's dependents xenconsoled and xenstore to have a lower minimum start position. We need to maintain the start order of xenstore first, xenconsoled second and lastly xend. By default, their minimum start positions are 96, 97 and 98 respectively. We will change these to 10, 11 and 12, again, respectively.

Note that we are '''not''' altering the start position of xendomains! This is intentional as this daemon will start the [[domU]] VMs. This can not happen until all other cluster related daemons have started.

To change the start order we will change the line chkconfig: 2345 9x 01 lines to chkconfig: 2345 1x 01, where x is the given daemon's start number. Further, we'll make sure that xenstored begins first by add it to xenconsoled's Required-Start line. We'll then make sure that xenconsoled starts before xend by adding it to xend's Required-Start line.

To recap the changes;
* xenstored will start first.
** We'll change it's start position from 96 to 10.
** We will not add anything to it's Required-Start as it must be the first daemon to come up.
* xenconsoled will start second.
** We'll change it's start position from 97 to 11.
** We will add xenstored to it's Required-Start line.
* xend will start third.
** We'll change it's start position from 98 to 12.
** We will add xenconsoled to it's Required-Start line.

When done, the three initialization scripts should look like the examples below.

<source lang="bash">
vim /etc/init.d/xenstored
</source>
<source lang="bash">
#!/bin/bash
#
# xenstored Script to start and stop the Xen control daemon.
#
# Author: Daniel Berrange <berrange@redhat.com
#
# chkconfig: 2345 10 01
# description: Starts and stops the Xen xenstored daemon.
### BEGIN INIT INFO
# Provides: xenstored
# Required-Start: $syslog $remote_fs
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xenstored
# Description: Starts and stops the Xen xenstored daemon.
### END INIT INFO
</source>

<source lang="bash">
vim /etc/init.d/xenconsoled
</source>
<source lang="bash">
#!/bin/bash
#
# xenconsoled Script to start and stop the Xen xenconsoled daemon
#
# Author: Daniel P. Berrange <berrange@redhat.com>
#
# chkconfig: 2345 11 01
# description: Starts and stops the Xen control daemon.
### BEGIN INIT INFO
# Provides: xenconsoled
# Required-Start: $syslog $remote_fs xenstored
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xenconsoled
# Description: Starts and stops the Xen xenconsoled daemon.
### END INIT INFO
</source>

<source lang="bash">
vim /etc/init.d/xend
</source>
<source lang="bash">
#!/bin/bash
#
# xend Script to start and stop the Xen control daemon.
#
# Author: Keir Fraser <keir.fraser@cl.cam.ac.uk>
#
# chkconfig: 2345 12 98
# description: Starts and stops the Xen control daemon.
### BEGIN INIT INFO
# Provides: xend
# Required-Start: $syslog $remote_fs xenconsoled
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xend
# Description: Starts and stops the Xen control daemon.
### END INIT INFO
</source>

With xend set to start at a position lower than 98, we now have room for chkconfig to put other daemons after it in the start order, which will be needed a little later. First and foremost, we now need to tell cman to not start until after xend is up.

As above, we will now edit cman's /etc/init.d/cman script. This time though, we will not edit it's chkconfig line. Instead, we will simply add xend to the Required-Start line.

<source lang="bash">
vim /etc/init.d/cman
</source>
<source lang="bash">
#!/bin/bash
#
# cman - Cluster Manager init script
#
# chkconfig: - 21 79
# description: Starts and stops cman
#
#
### BEGIN INIT INFO
# Provides: cman
# Required-Start: $network $time xend
# Required-Stop: $network $time
# Default-Start:
# Default-Stop:
# Short-Description: Starts and stops cman
# Description: Starts and stops the Cluster Manager set of daemons
### END INIT INFO
</source>

Finally, remove and re-add the xend and cman daemons to re-order them in the start list:

<source lang="bash">
chkconfig xenstored off; chkconfig xenconsoled off; chkconfig xend off; chkconfig cman off;
chkconfig xenstored on; chkconfig xenconsoled on; chkconfig xend on; chkconfig cman on
</source>

Confirm that the order has changed so that xend is earlier in the boot sequence than cman. Assuming you've switched to run-level 3, run:

<source lang="bash">
ls -lah /etc/rc3.d/
</source>

Your start sequence should now look like:

<source lang="text">
lrwxrwxrwx. 1 root root 19 Sep 15 19:29 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 21 Sep 15 19:29 S27xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 14 Sep 15 19:29 S28xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Sep 15 19:29 S29cman -> ../init.d/cman
</source>

=== Booting Into The New dom0 ===

If everything went well, you should be able to boot the new dom0 operating system. If you watch the boot process closely, you will see that the boot process is different. You should now see the Xen hypervisor boot prior to handing off to the "host" operating system. This can be confirmed once the dom0 operating system has booted by checking that the file /proc/xen/capabilities exists. What it contains doesn't matter at this stage, only that it exists at all.

== Configure Networking ==

Networking in Xen, particularly in a cluster, can be confusing. If you are not familiar with networking in Xen, please review to following article before proceeding.

A note of a major change from previous layouts. In Xen 3.x, ethX would be copied to a virtual interface called vethX. Then the real ethX would be renamed to pethX and the virtual interface vethX would be renamed to ethX to take it's place. Finally, a bridge called xenbrX would be created and the real pethX and virtual ethX would be connected to it.

This has been changed somewhat it that now, by default, ethX is left alone and a simple bridge called virbrX would be created. We'll be changing this to be somewhat similar to the old style.

Specifically, the real ethX will be renamed to pethX. Then a bridge will be created called ethX, which plays the role of dom0's interface '''and''' bridges connections from VMs through pethX and out into the real world.

This is explained in more detail, and with diagrams, in the article below.

* [[Networking in Xen]]

=== Adding New NICs to Xen ===

By default, xend manages eth0 only. We need to add eth2. Personally, I don't like to put the storage network ethernet devices (eth1) under Xen's control as this potentially can cause DRBD problems on xend restart. Whether you add it or not I will leave to your preferences.

You can see which, if any, network devices are under Xen's control by running ifconfig and checking to see if there is a virbrX corresponding to a given ethX device.

<source lang="bash">
ifconfig
</source>
<source lang="text">
eth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet addr:192.168.1.74 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:224261 errors:0 dropped:0 overruns:0 frame:0
TX packets:55174 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:319384110 (304.5 MiB) TX bytes:27348739 (26.0 MiB)
Interrupt:225 Base address:0x8000

eth1 Link encap:Ethernet HWaddr 00:1B:21:72:9B:5A
inet addr:192.168.2.74 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:9b5a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:832 (832.0 b) TX bytes:6234 (6.0 KiB)
Memory:feae0000-feb00000

eth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet addr:192.168.3.74 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:818 (818.0 b) TX bytes:6081 (5.9 KiB)
Memory:fe9e0000-fea00000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

virbr0 Link encap:Ethernet HWaddr 02:23:C8:98:31:17
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:4013 (3.9 KiB)
</source>

In the above example, eth0 has a corresponding virbr0 bridge having it's own subnet. In non-clustered systems, this is fine. For our purposes though, it will not do.

=== Removing The qemu virbr0 Bridge ===

By default, [[QEMU]] creates a bridge called virbr0 designed to connect virtual machines to the first eth0 interface. Our system will not need this, so we will remove it. This bridge is configured in the /etc/libvirt/qemu/networks/default.xml file, so to remove this bridge, simply delete the contents of the file.

<source lang="bash">
cat /dev/null >/etc/libvirt/qemu/networks/default.xml
</source>

The next time you reboot, that bridge will be gone.

'''Madi''': Put in the command to delete the bridge before a reboot.

=== Create /etc/xen/scripts/an-network-script ===

This script will be used by Xen to turn the dom0 ethX interfaces into bridges. All traffic to the bridge, be it from dom0 or domU VMs, will be routeable out of the corresponding pethX device. As domU VMs come online, a hotplug script will create virtual interfaces between this new bridge and the domU's interface(s). Think of the vifX.Y devices as being the network cables you'd normally run between a server and a switch.

Before we proceed, please note three things;
# You don't need to use the file name an-network-script. I suggest this name mainly to keep in line with the rest of the 'AN!x' naming used here.
# If you install convirt or other hypervisor tools, they will likely create their own bridge script.
# Adding eth1 is optional, as we know ahead of time that eth1 will not be made available to any virtual machines as it is dedicated to [[DRBD]]. I'm adding it here because I like having things consistent; Do whichever makes more sense to you.

First, touch the file and then chmod it to be executable.

<source lang="bash">
touch /etc/xen/scripts/an-network-script
chmod 755 /etc/xen/scripts/an-network-script
</source>

Now edit it to contain the following:
<source lang="bash">
vim /etc/xen/scripts/an-network-script
</source>
<source lang="text">
#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=eth0
"$dir/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=eth2
</source>

Now tell Xen to execute that script by editing /etc/xen/xend-config.sxp file and changing the network-script argument to point to this new script (this is line 158 in the default xend-config.sxp script):

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>
<source lang="text">
#(network-script network-bridge)
#(network-script /bin/true)
(network-script an-network-script)
</source>

'''Warning''': The next step may trigger fencing of the nodes! As such, be sure that you're not running anything critical. If unsure, please stop cman or reboot the nodes.

<source lang="bash">
/etc/init.d/cman stop
</source>

Now restart xend.

<source lang="bash">
/etc/init.d/xend restart
</source>

If everything worked, you should now be able to run ifconfig and see that all the ethX devices have matching pethX, virtual and bridge devices.

<source lang="bash">
ifconfig
</source>
<source lang="text">
eth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet addr:192.168.1.74 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:78 errors:0 dropped:0 overruns:0 frame:0
TX packets:57 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9796 (9.5 KiB) TX bytes:12574 (12.2 KiB)

eth1 Link encap:Ethernet HWaddr 00:1B:21:72:9B:5A
inet addr:192.168.2.74 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:9b5a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:832 (832.0 b) TX bytes:6234 (6.0 KiB)
Memory:feae0000-feb00000

eth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet addr:192.168.3.74 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:32 errors:0 dropped:0 overruns:0 frame:0
TX packets:29 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5471 (5.3 KiB) TX bytes:5867 (5.7 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

peth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:224486 errors:0 dropped:0 overruns:0 frame:0
TX packets:55349 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:319406626 (304.6 MiB) TX bytes:27384681 (26.1 MiB)
Interrupt:225 Base address:0x8000

peth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:35 errors:0 dropped:0 overruns:0 frame:0
TX packets:70 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6827 (6.6 KiB) TX bytes:12470 (12.1 KiB)
Memory:fe9e0000-fea00000
</source>

'''Note''': The virbr0 may remain until you reboot your nodes.

If you see something like this, then you are ready to proceed! Now start your cluster back up.

<source lang="bash">
/etc/init.d/cman start
</source>

We're done for now. There is more to do in Xen, but this was all we needed to do in order to proceed with the next several steps. Onces we have the clustered storage online, we'll come back to Xen for the domU setup.

= Building the DRBD Array =

Building the DRBD array requires a few steps. First, raw space on either node must be prepared. Next, DRBD must be told that it is to create a resource using this newly configured raw space. Finally, the new array must be initialized.

== A Map of the Cluster's Storage ==

The layout of the storage in the cluster can quickly become difficult to follow. Below is an [[ASCII]] drawing which should help you see how DRBD will tie in to the rest of the cluster's storage. This map assumes a simple [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]] array underlying each node. If your node has a single hard drive, simply collapse the first two layers into one. Similarly, if your underlying storage is a more complex RAID array, simply expand the number of physical devices at the top level.

<source lang="text">
Node1 Node2
_____ _____ _____ _____
| sda | | sdb | | sda | | sdb |
|_____| |_____| |_____| |_____|
|_______| |_______|
_______ ____|___ _______ _______ ____|___ _______
__|__ __|__ __|__ __|__ __|__ __|__ __|__ __|__
| md0 | | md1 | | md2 | | md3 | | md3 | | md2 | | md1 | | md0 |
|_____| |_____| |_____| |_____| |_____| |_____| |_____| |_____|
| | | | | | | |
___|___ _|_ ____|____ |___________| ____|____ _|_ ___|___
| /boot | | / | | <swap> | | | <swap> | | / | | /boot |
|_______| |___| |_________| ______|______ |_________| |___| |_______|
| /dev/drbd0 |
|_____________|
|
____|______
| clvm PV |
|___________|
|
_____|_____
| drbd0_vg0 |
|___________|
|
_____|_____ ___...____
| | |
___|___ ___|___ ___|___
| lv_X | | lv_Y | | lv_N |
|_______| |_______| |_______|
</source>

== Install The DRBD Tools ==

DRBD has two components; The actual application and tools and the kernel module.

There are two options for installing the DRBD user-land tools at this point; AN!Cluster-built RPMs or using the ones shipped with Fedora. Regardless of which method you choose, you will need to either install the AN!Cluster DRBD kernel module RPMs or else rebuild the source RPMs referenced.

=== Install The AN!Cluster DRBD User-Land Tools ===

I am currently experimenting with ways to solve a DRBD triggered kernel oops in the Xen pvops 2.6.32 kernel. For this reason, I've recompiled the following user-land RPMs under the AN!Cluster variant dom0 kernel RPMs referenced earlier in this paper. If you used the AN! RPMs, then I suggest giving these RPMs a try. However, if you are using myoung's dom0, I recommend sticking to the Fedora-provided user-land DRBD tools.

<source lang="bash">
yum -y install bash-completion heartbeat pacemaker
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-utils-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-xen-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-bash-completion-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-udev-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-heartbeat-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-pacemaker-8.3.7-2.fc13.x86_64.rpm
rpm -ivh drbd*8.3.7-2.fc13.x86_64.rpm
</source>

=== Install The Stock Fedora DRBD User-Land Tools ===

You will need to install the following tools:

<source lang="bash">
yum install drbd.x86_64 drbd-xen.x86_64 drbd-utils.x86_64
</source>

=== Disable heartbeat ===

These packages require that the heartbeat packages be installed. This is for a different cluster platform which we are not using here, so we will disable it from starting with the system.

<source lang="bash">
chkconfig heartbeat off
</source>

== Install The DRBD Kernel Module ==

The kernel module '''must''' match the [[dom0]] kernel that is running. If you update the kernel and neglect to update the DRBD kernel module, the DRBD array '''will not start'''.

To help simplify things, links to pre-compiled DRBD kernel modules are provided. If the kernel version you have installed doesn't match your kernel, instructions on recompiling the DRBD kernel module from source RPM is provided as well.

=== Install Pre-Compiled DRBD Kernel Module RPMs ===

'''Warning''': The RPM provided here '''''will only work''''' with the kernel-2.6.32.21-168.xendom0_an1.fc13.x86_64.rpm kernel. If you are using Michael Young's dom0 kernel, please skip to [[#Building DRBD Kernel Module RPMs From Source|the next section]].

This RPM provides the DRBD kernel module. Note that these RPMs are compiled against the AN!Cluster variant of myoung's 2.6.32.21_168 dom0 kernel.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-km-2.6.32.23_170.dom0_an1.fc13.x86_64-8.3.7-12.fc13.x86_64.rpm
rpm -ivh drbd-km-2.6.32.23_170.dom0_an1.fc13.x86_64-8.3.7-12.fc13.x86_64.rpm
</source>

If you would like to install the debuginfo

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm
rpm -ivh drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm
</source>

=== Building DRBD Kernel Module RPMs From Source ===

If the above RPMs don't work or if the dom0 kernel you are using in any way differs, please follow the steps here to create a DRBD kernel module matched to your running dom0.

First, install the build environment.

<source lang="bash">
yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
</source>

Install the kernel headers and development library for the dom0 kernel:

'''Note''': The following commands use --force to get past the fact that the headers for the 2.6.33 are already installed, thus making RPM think that these are too old and will conflict. Please proceed with caution.

* If you are using Michael Young's kernel:

<source lang="bash">
cd ~
wget -c http://fedorapeople.org/~myoung/dom0/x86_64/kernel-headers-2.6.32.21-170.xendom0.fc12.x86_64.rpm
wget -c http://fedorapeople.org/~myoung/dom0/x86_64/kernel-devel-2.6.32.21-170.xendom0.fc12.x86_64.rpm
rpm -ivh --force kernel*2.6.32.21-170.xendom0.fc12.x86_64.rpm
</source>

* If you are using the AN!Cluster dom0 kernel:

<source lang="bash">
wget -c https://alteeve.com/files/an-cluster/kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm</source>

Now you need to download, prepare, build and install the source RPM.

<source lang="bash">
rpm -ivh http://fedora.mirror.iweb.ca/releases/13/Everything/source/SRPMS/drbd-8.3.7-2.fc13.src.rpm
cd /root/rpmbuild/SPECS/
rpmbuild -bp drbd.spec
cd /root/rpmbuild/BUILD/drbd-8.3.7/
./configure --enable-spec --with-km
cp /root/rpmbuild/BUILD/drbd-8.3.7/drbd-km.spec /root/rpmbuild/SPECS/
cd /root/rpmbuild/SPECS/
rpmbuild -ba drbd-km.spec
cd /root/rpmbuild/RPMS/x86_64
rpm -Uvh drbd-km-*
</source>

This may be needed if drbd-utils, the user-land DRBD tools, is listed as a requirement when trying to install the drbd-km* RPMs. This step will build all of the DRBD tools RPMs.

<source lang="bash">
yum install bash-completion heartbeat pacemaker
cd /root/rpmbuild/SPECS/
rpmbuild -ba drbd.spec
cd ~/rpmbuild/RPMS/x86_64/
rpm -Uvh drbd-*
chkconfig off heartbeat
</source>

You should be good to go now!

== Allocating Raw Space For DRBD On Each Node ==

If you followed the setup steps provided for in "[[Two Node Fedora 13 Cluster]]", you will have a set amount of unconfigured hard drive space. This is what we will use for the DRBD space on either node. If you've got a different setup, you will need to allocate some raw space before proceeding.

=== Create a Simple Partition ===

If you do not have two drives, please follow the next section's steps, but pay attention to the "'''note'''"s. In short, you will need to create one partition, leave the default type of the partition as 83, write the changes to disk and the proceed to the [[#DRBD Configuration Files|DRBD Configuration Files]] section.

=== Creating a RAID level 1 'md' Device ===

This assumes that you have two raw drives, /dev/sda and /dev/sdb. It further assumes that you've created three partitions which have been assigned to three existing /dev/mdX devices. With these assumptions, we will create /dev/sda4 and /dev/sdb4 and, using them, create a new /dev/md3 device that will host the DRBD partition.

If you have multiple drives and plan to use a different [[TLUG_Talk:_Storage_Technologies_and_Theory#RAID_Levels|RAID levels]], please adjust the follow commands accordingly.

==== Creating The New Partitions ====

'''Warning''': The next steps will have you directly accessing your server's hard drive configuration. Please do not proceed on a live server until you've had a chance to work through these steps on a test server. One mistake can '''''blow away all your data'''''.

Start the fdisk shell for the first hard drive; /dev/sda.

<source lang="bash">
fdisk /dev/sda
</source>
<source lang="text">
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help):
</source>

'''Note''': Depending on your configuration, you may not see the above warning or you may see a different warning. Note it, but it is likely nothing to worry about it.

View the current configuration with the '''p'''rint option

<source lang="bash">
p
</source>
<source lang="text">
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6fe1

Device Boot Start End Blocks Id System
/dev/sda1 1 5100 40960000 fd Linux raid autodetect
/dev/sda2 5100 5622 4194304 fd Linux raid autodetect
/dev/sda3 * 5622 5654 256000 fd Linux raid autodetect

Command (m for help):
</source>

Now we know for sure that the next free partition number is 4. We will now create the '''n'''ew partition.

<source lang="bash">
n
</source>
<source lang="text">
Command action
e extended
p primary partition (1-4)
</source>

We will make it a '''p'''rimary partition

<source lang="bash">
p
</source>
<source lang="text">
Selected partition 4
First cylinder (5654-60801, default 5654):
</source>

Then we simply hit <enter> to select the default starting block.

<source lang="bash">
<enter>
</source>
<source lang="text">
Using default value 5654
Last cylinder, +cylinders or +size{K,M,G} (5654-60801, default 60801):
</source>

Once again we will press <enter> to select the default ending block.

<source lang="bash">
<enter>
</source>
<source lang="text">
Using default value 60801

Command (m for help):
</source>

'''Note''': If you only have one drive and are not creating a [[RAID]] array, you do not to change the type of the partition so you can skip the next few steps. Continue at the step where you write the changes.

Now we need to change the type of partition that it is.

<source lang="bash">
t
</source>
<source lang="text">
Partition number (1-4):
</source>

We know that we are modifying partition number 4.

<source lang="bash">
4
</source>
<source lang="text">
Hex code (type L to list codes):
</source>

Now we need to set the [[hex]] code for the [[Filesystem_List#List_of_Linux_Partition_Types|partition type]] to set. We want to set fd, which defines Linux raid autodetect.

<source lang="bash">
fd
</source>
<source lang="text">
Changed system type of partition 4 to fd (Linux raid autodetect)
</source>

Now check that everything went as expected by once again '''p'''rinting the partition table.

<source lang="bash">
p
</source>
<source lang="text">
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6fe1

Device Boot Start End Blocks Id System
/dev/sda1 1 5100 40960000 fd Linux raid autodetect
/dev/sda2 5100 5622 4194304 fd Linux raid autodetect
/dev/sda3 * 5622 5654 256000 fd Linux raid autodetect
/dev/sda4 5654 60801 442972704+ fd Linux raid autodetect

Command (m for help):
</source>

'''Note''': If you only have one drive, your partitions will be 83 Linux or 82 Linux swap / Solaris, instead of fd Linux raid autodetect.

There it is. So finally, we need to '''w'''rite the changes to the disk.

<source lang="bash">
w
</source>
<source lang="text">
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
</source>

'''Note''': If you only have one drive, reboot now if you got the message above and then skip forward to the "[[#DRBD Configuration Files|DRBD Configuration Files]]" section.

If you see the above message, '''do not''' reboot yet. repeat these steps for the second drive, /dev/sdb, and then reboot.

==== Creating The New /dev/mdX Device ====

''If you only have one drive, skip this step.''

Now we need to use mdadm to create the new [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]] device. This will be used as the device that DRBD will directly access.

<source lang="bash">
mdadm --create /dev/md3 --homehost=localhost.localdomain --raid-devices=2 --level=1 /dev/sda4 /dev/sdb4
</source>
<source lang="text">
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
</source>

Seeing as /boot doesn't exist on this device, we can safely ignore this warning.

<source lang="bash">
y
</source>
<source lang="text">
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/md4 started.
</source>

You can now cat /proc/mdstat to verify that it indeed built. If you're interested, you could open a new terminal window and use watch cat /proc/mdstat and watch the array build.

<source lang="bash">
cat /proc/mdstat
</source>
<source lang="text">
md3 : active raid1 sdb4[1] sda4[0]
442971544 blocks super 1.2 [2/2] [UU]
[>....................] resync = 0.8% (3678976/442971544) finish=111.0min speed=65920K/sec

md2 : active raid1 sda2[0] sdb2[1]
4193272 blocks super 1.1 [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]
40958908 blocks super 1.1 [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk

md0 : active raid1 sda3[0] sdb3[1]
255988 blocks super 1.0 [2/2] [UU]

unused devices: <none>
</source>

Finally, we need to make sure that the new array will start when the system boots. To do this, we'll again use mdadm, but with different options that will have it output data in a format suitable for the /etc/mdadm.conf file. We'll redirect this output to that config file, thus updating it.

<source lang="bash">
mdadm --detail --scan | grep md3 >> /etc/mdadm.conf
cat /etc/mdadm.conf
</source>
<source lang="text">
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=b58df6d0:d925e7bb:c156168d:47c01718
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=ac2cf39c:77cd0314:fedb8407:9b945bb5
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=4e513936:4a966f4e:0dd8402e:6403d10d
ARRAY /dev/md3 metadata=1.2 name=localhost.localdomain:3 UUID=f0b6d0c1:490d47e7:91c7e63a:f8dacc21
</source>

You'll note that the last line, which we just added, is different from the previous lines. This isn't a concern, but you are welcome to re-write it to match the existing format if you wish.

Before you proceed, it is strongly advised that you reboot each node and then verify that the new array did in fact start with the system. You ''do not'' need to wait for the sync to finish before rebooting. It will pick up where you left off once rebooted.

'''Note''': You'll notice we did not format a file system on this raid array, this is intentional. DRBD use the raw device and does not need a file system on it.

== DRBD Configuration Files ==

DRBD uses a global configuration file, /etc/drbd.d/global_common.conf, and one or more resource files. The resource files need to be created in the /etc/drbd.d/ directory and must have the suffix .res. For this example, we will create a single resource called r0 which we will configure in /etc/drbd.d/r0.res.

=== /etc/drbd.d/global_common.conf ===

The stock /etc/drbd.d/global_common.conf is sane, so we won't bother altering it here.

Full details on all the drbd.conf configuration file directives and arguments can be found [http://www.drbd.org/users-guide/re-drbdconf.html here]. '''Note''': That link doesn't show this new configuration format. Please see [http://www.novell.com/documentation/sle_ha/book_sleha/?page=/documentation/sle_ha/book_sleha/data/sec_ha_drbd_configure.html Novell's] link.

=== /etc/drbd.d/r0.res ===

This is the important part. This defines the resource to use, and must reflect the IP addresses and storage devices that DRBD will use for this resource.

<source lang="bash">
vim /etc/drbd.d/r0.res
</source>
<source lang="bash">
# This is the name of the resource and it's settings. Generally, 'r0' is used
# as the name of the first resource. This is by convention only, though.
resource r0
{
# This tells DRBD where to make the new resource available at on each
# node. This is, again, by convention only.
device /dev/drbd0;

# The main argument here tells DRBD that we will have proper locking
# and fencing, and as such, to allow both nodes to set the resource to
# 'primary' simultaneously.
net
{
allow-two-primaries;
}

# This tells DRBD to automatically set both nodes to 'primary' when the
# nodes start.
startup
{
become-primary-on both;
}

# This tells DRBD to look for and store it's meta-data on the resource
# itself.
meta-disk internal;

# The name below must match the output from `uname -n` on each node.
on an-node01.alteeve.com
{
# This must be the IP address of the interface on the storage
# network (an-node01.sn, in this case).
address 192.168.2.71:7789;

# This is the underlying partition to use for this resource on
# this node.
disk /dev/md3;
}

# Repeat as above, but for the other node.
on an-node02.alteeve.com
{
address 192.168.2.72:7789;
disk /dev/md3;
}
}
</source>

This file must be copied to '''BOTH''' nodes and must match before you proceed.

== Starting The DRBD Resource ==

From the rest of this section, pay attention to whether you see
* '''Node1'''
* '''Node2'''
* '''Both'''

These indicate which node to run the following commands on. There is no functional difference between either node, so just randomly choose one to be '''Node1''' and the other will be '''Node2'''. Once you've chosen which is which, be consistent with which node you run the commands on. Of course, if a command block is proceeded by '''Both''', run the following code block on both nodes.

=== Loading the 'drbd' Module ===

'''Both'''

Normally, we'd load the drbd module by simply starting the /etc/init.d/drbd daemon. However, if we did that at this stage, we'd generate errors because there isn't an UpToDate disk in the array. To get around this, we'll manually load the drbd kernel module using modprobe.

<source lang="bash">
modprobe drbd
</source>

This won't return any output, but if you check, you should now see the special /proc/drbd file.

=== Monitoring Progress ===

'''Both'''

I find it very useful to monitor DRBD while running the rest of the setup. To do this, open a second terminal on each node and use watch to keep an eye on /proc/drbd. This way you will be able to monitor the progress of the array in near-real time.

'''Both'''
<source lang="bash">
watch cat /proc/drbd
</source>

At this stage, it should look like this:

<source lang="text">
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:Unconfigured
</source>

=== Initialize The Resource ===

'''Both'''

This step creates the DRBD meta-data on the new DRBD resource's backing devices. It is only needed when creating new DRBD partitions.

<source lang="bash">
drbdadm create-md r0
</source>
<source lang="text">
--== Thank you for participating in the global usage survey ==--
The server's response is:

you are the 9507th user to install this version
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
</source>

The /proc/drbd output should not have changed at this stage.

=== Starting the Resource ===

'''Both'''

This will attach the backing device, /dev/md3 in our case, and then start the new resource r0.

<source lang="bash">
drbdadm up r0
</source>

There will be no output at the command line. If you are watching /proc/drbd though, you should now see something like this:

<source lang="text">
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:442957988
</source>

That it is Secondary/Secondary and Inconsistent/Inconsistent is expected.

=== Setting the First Primary Node ===

'''Node1'''

As this is a totally new resource, DRBD doesn't know which side of the array is "more valid" than the other. In reality, neither is as there was no existing data of note on either node. This means that we now need to choose a node and tell DRBD to treat it as the "source" node. This step will also tell DRBD to make the "source" node primary. Once set, DRBD will begin sync'ing in the background.

<source lang="bash">
drbdadm -- --overwrite-data-of-peer primary r0
</source>

As before, there will be no output at the command line, but /proc/drbd will change to show the following:

<source lang="text">
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:69024 nr:0 dw:0 dr:69232 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:442888964
[>....................] sync'ed: 0.1% (432508/432576)M
finish: 307:33:42 speed: 320 (320) K/sec
</source>

If you're watching the secondary node, the /proc/drbd will show ro:Secondary/Primary ds:Inconsistent/UpToDate. This is, as you can guess, simply a reflection of it being the "over-written" node.

=== Setting the Second Node to Primary ===

'''Node2'''

The last step to complete the array is to tell the second node to also become primary.

<source lang="bash">
drbdadm primary r0
</source>

As with many drbdadm commands, nothing will be printed to the console. If you're watching the /proc/drbd though, you should see something like Primary/Primary ds:UpToDate/Inconsistent. The Inconsistent flag will remain until the sync is complete.

=== A Note On sync Speed ===

You will notice in the previous step that the sync speed seems awfully slow at 320 (320) K/sec.

'''This is not a problem!'''

As actual data is written to either side of the array, that data will be immediately copied to both nodes. As such, both nodes will always contain up to date copies of the real data. Given this, the syncer is intentionally set low so as to not put too much load on the underlying disks that could cause slow downs. If you still wish to increase the sync speed, you can do so with the following command.

'''Warning''': If you set the DRBD sync speed too high and saturate your disks' maximum write speed, clvmd daemon will likely fail to start in some cases, leading to fences. For this reason, keep your sync rate to about 2/3rds of the underlying disk maximum write speed and hold off on bumping up the sync speed until you know you will have a period of low activity on your cluster.

<source lang="bash">
drbdsetup /dev/drbd0 syncer -r 35M
</source>

The speed-up will not be instant. It will take a little while for the speed to pick up. Once the sync is finished, it is a good idea to revert to the default sync rate.

<source lang="text">
drbdadm syncer r0
</source>

= Setting Up CLVM =

The goal of DRBD in the cluster is to provide clustered [[LVM]], referred to as [[CLVM]] to the nodes. This is done by turning the DRBD partition into an CLVM physical volume.

So now we will create a [[PV]] on top of the new [[DRBD]] partition, /dev/drbd0, that we created in the previous step. Since this new LVM [[PV]] will exist on top of the shared DRBD partition, whatever get written to it's logical volumes will be immediately available on either node, regardless of which node actually initiated the write.

This capability is the underlying reason for creating this cluster; Neither machine is truly needed so if one machine dies, anything on top of the DRBD partition will still be available. When the failed machine returns, the surviving node will have a list of what blocks changed while the other node was gone and can use this list to quickly re-sync the other server.

== Making LVM Cluster-Aware ==

Normally, LVM is run on a single server. This means that at any time, the LVM can write data to the underlying drive and not need to worry if any other device might change anything. In clusters, this isn't the case. The other node could try to write to the shared storage, so then nodes need to enable "locking" to prevent the two nodes from trying to work on the same bit of data at the same time.

The process of enabling this locking is known as making LVM "cluster-aware".

LVM has tool called lvmconf that can be used to enable LVM locking. This is provided as part of the lvm2-cluster package.

<source lang="bash">
yum -y install lvm2-cluster.x86_64
</source>

Now to enable cluster awareness in LVM, run to following command.

<source lang="bash">
lvmconf --enable-cluster
</source>

By default, clvmd, the cluster lvm daemon, is stopped and not set to run on boot. Now that we've enabled LVM locking, we need to start it:

<source lang="bash">
/etc/init.d/clvmd status
</source>
<source lang="text">
clvmd is stopped
active volumes: (none)
</source>

As expected, it is stopped, so lets start it:

<source lang="bash">
/etc/init.d/clvmd start
</source>
<source lang="text">
Activating VGs: No volume groups found
[ OK ]
</source>

'''Note''': I've seen on a few occasions where starting clvmd will time out and, on occasion, fences will be issued. I've not sorted out why, but I have usually been able to resolve this by stopping clvmd and cman, then restarting cman and, finally, restarting clvmd. If I can sort out a way to reliably trigger this problem, I will submit a bug report.

== Filtering Out Devices ==

'''ToDo''': Find a less-aggressive filter.

With the stock /etc/lvm/lvm.conf configuration, all devices on the system will be checked for LVM volumes. This can cause a problem as LVM will give preference to the LVM data on the RAID device over the DRBD device. It sees a duplicate as both are, effectively, one and the same.

To work around this, we need to alter the filter = [] entry. At the time of writing, simply rejecting the underlying /dev/md3 device as a candidate wasn't enough. So for now, we will tell LVM to accept DRBD devices and reject all other devices. To do this, we'll insert "a|/dev/drbd*|" as the first array entry and change the existing entry to "r/.*/".

'''Note''': I would love feedback on a filter argument that successfully ignored just /dev/md3, if anyone can suggest one.

<source lang="bash">
vim /etc/lvm/lvm.conf
</source>
<source lang="text">
# By default we accept every block device:
#filter = [ "a/.*/" ]
filter = [ "a|/dev/drbd*|", "r/.*/" ]
</source>

Now delete the existing cache file so that LVM is forced to rescan the system.

<source lang="bash">
rm -f /etc/lvm/cache/.cache
</source>

The changes take effect immediately.

== Creating a new PV using the DRBD Partition ==

'''Node1'''

We can now proceed with setting up the new DRBD-based LVM physical volume. Once the PV is created, we can create a new volume group and start allocating space to logical volumes.

'''Note''': As we will be using our DRBD device, and as it is a shared block device, most of the following commands only need to be run on one node. Once the block device changes in any way, those changes will near-instantly appear on the other node. For this reason, unless explicitly stated to do so, only run the following commands on one node.

To setup the DRBD partition as an LVM PV, run pvcreate:

<source lang="bash">
pvcreate /dev/drbd0
</source>
<source lang="text">
Physical volume "/dev/drbd0" successfully created
</source>

'''Both'''

Now, on both nodes, check that the new physical volume is visible by using pvdisplay:

<source lang="bash">
pvdisplay
</source>
<source lang="text">
"/dev/drbd0" is a new physical volume of "422.44 GiB"
--- NEW Physical volume ---
PV Name /dev/drbd0
VG Name
PV Size 422.44 GiB
Allocatable NO
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID YHmdip-SuJN-KIEv-2tbK-BT9Q-wfOo-OuQuaW
</source>

If you see PV Name /dev/drbd0 (or your underlying partition) on both nodes, then your DRBD setup and LVM configuration changes are working perfectly!

== Creating a VG on the new PV ==

'''Node1'''

Now we need to create the volume group using the vgcreate command:

<source lang="bash">
vgcreate -c y drbd0_vg0 /dev/drbd0
</source>
<source lang="text">
Clustered volume group "drbd0_vg0" successfully created
</source>

'''Both'''

Now we'll check that the new VG is visible on both nodes using vgdisplay:

<source lang="bash">
vgdisplay
</source>
<source lang="text">
--- Volume group ---
VG Name drbd0_vg0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
Clustered yes
Shared no
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 422.43 GiB
PE Size 4.00 MiB
Total PE 108143
Alloc PE / Size 0 / 0
Free PE / Size 108143 / 422.43 GiB
VG UUID Bb8l9e-es2z-PhaF-Gg3o-2is2-DZ1S-V2RsBF
</source>

If the new VG is visible on both nodes, we are ready to create our first logical volume using the lvcreate tool.

== Creating the First LV on the new VG ==

'''Node1'''

Now we'll create a simple 20 GiB logical volumes. We will use it as a shared GFS2 store for shared files and to store our Xen domU config files later on.

<source lang="bash">
lvcreate -L 20G -n xen_shared drbd0_vg0
</source>
<source lang="text">
Logical volume "xen_shared" created
</source>

'''Both'''

As before, we will check that the new logical volume is visible from both nodes by using the lvdisplay command:

<source lang="bash">
lvdisplay
</source>
<source lang="text">
--- Logical volume ---
LV Name /dev/drbd0_vg0/xen_shared
VG Name drbd0_vg0
LV UUID AqQizc-KBpX-2scN-WFLb-jIeF-QDcM-PlQW84
LV Write Access read/write
LV Status available
# open 0
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
</source>

Again, if this is visible from both nodes, we're set! Repeat this process for all future LVs you will want to create. We will do this a little later to create LVs for Xen VMs.

= Creating A Shared GFS FileSystem =

GFS is a cluster-aware file system that can be simultaneously mounted on two or more nodes at once. We will use it as a place to store ISOs that we'll use to provision our virtual machines.

== Install The GFS2 Utilities ==

Start by installing the [[GFS2]] tools:

<source lang="bash">
yum -y install gfs2-utils.x86_64
</source>

== Format Our CLVM LV With The GFS2 File System ==

'''Node1'''

'''Note''': The following example is designed for the cluster used in the prerequisite HowTo.
* If you have more than 2 nodes, increase the -j 2 to the number of nodes you want to mount this file system on.
* If your cluster is named something other than an-cluster (as set in the cluster.conf file), change -t an-cluster:xen_shared to match you cluster's name. The xen_shared can be whatever you like, but it must be unique in the cluster. I tend to use a name that matches the LV name, but this is my own preference and is not required.

To format the partition run:
<source lang="bash">
mkfs.gfs2 -p lock_dlm -j 2 -t an-cluster:xen_shared /dev/drbd0_vg0/xen_shared
</source>
<source lang="text">
This will destroy any data on /dev/drbd0_vg0/xen_shared.
It appears to contain: symbolic link to `../dm-0'

Are you sure you want to proceed? [y/n]
</source>

Acknowledge the warning, if any, and then press y if you are ready to proceed.

<source lang="bash">
y
</source>
<source lang="text">
Device: /dev/drbd0_vg0/xen_shared
Blocksize: 4096
Device Size 20.00 GB (5242880 blocks)
Filesystem Size: 20.00 GB (5242878 blocks)
Journals: 2
Resource Groups: 80
Locking Protocol: "lock_dlm"
Lock Table: "an-cluster:xen_shared"
UUID: A1487063-2A3F-43B1-3A36-44936B0B4D1E
</source>

Once the format completes, you can mount /dev/drbd0_vg0/xen_shared as you would a normal file system.

'''Both''':

To complete the example, lets mount the GFS2 partition we made just now on /shared and then use df -h to verify.

<source lang="bash">
mkdir /xen_shared
mount /dev/drbd0_vg0/xen_shared /xen_shared
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.8G 34G 8% /
tmpfs 466M 29M 438M 7% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
/dev/dm-0 20G 259M 20G 2% /xen_shared
</source>

You may have noticed that it shows /dev/dm-0 instead of /dev/drbd0_vg0/xen_shared. If you look at the later, you will see that it is simply a [[symlink]] to the former.

<source lang="bash">
ls -lah /dev/drbd0_vg0/xen_shared
</source>
<source lang="text">
lrwxrwxrwx. 1 root root 7 Sep 9 13:24 /dev/drbd0_vg0/xen_shared -> ../dm-0
</source>

== Add An Entry To /etc/fstab ==

The last step is to add an entry for this new partition to each node's /etc/fstab file.

=== Reference The GFS2 Partition By Device Path ===

This is the more traditional method of referencing the GFS2 partition by using it's device path directly.

'''Warning''': An incorrect edit of the /etc/fstab file can leave your system unable to boot! Please review the line generated above to make sure it is accurate and compatible with your setup before proceeding.

<source lang="bash">
vim /etc/fstab
</source>
<source lang="text">
#
# /etc/fstab
# Created by anaconda on Tue Sep 7 06:29:51 2010
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=865690db-85d0-44c5-9f32-ffb6fdf47060 / ext3 defaults 1 1
UUID=8b9822b6-a92e-48c9-96b5-f8943142319e /boot ext3 defaults 1 2
UUID=94b03547-a7e3-45bb-b2d5-837498b370f4 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
xenfs /proc/xen xenfs defaults 0 0
/dev/drbd0_vg0/xen_shared /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

=== Reference The GFS2 Partition By UUID ===

It is sometimes preferable to create an fstab entry that locates the device path via it's [[UUID]]. To do this, you can run the following command which, though a bit cryptic, will print out an /etc/fstab compatible string.

'''Warning''': The same warnings apply here as above

<source lang="bash">
echo `gfs2_edit -p sb /dev/drbd0_vg0/xen_shared | grep sb_uuid | sed -e "s/.*sb_uuid *$.*$/UUID=\L\1\E \/xen_shared\t\tgfs2\trw,suid,dev,exec,nouser,async\t0 0/"`
</source>
<source lang="text">
UUID=a1487063-2a3f-43b1-3a36-44936b0b4d1e /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

You may have noticed that defaults isn't used. Rather, all but the auto option are manually set. This is because the system will drop to single-user mode at boot if it can't mount an auto partition at boot time (auto being implied by defaults). Given that our GFS2 partition sits on top of DRBD and the cluster, there is no way to make it available that early in the boot process.

Further, the gfs2 init script specifically excludes entries in /etc/fstab that have the 'noauto option set. For this reason, we can't simply specify that as we need the init script to see the partition so that it is mounted when GFS2 starts and unmounted when it stops.

Now add this string to /etc/fstab.

<source lang="bash">
vim /etc/fstab
</source>
<source lang="text">
#
# /etc/fstab
# Created by anaconda on Tue Sep 7 06:29:51 2010
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=865690db-85d0-44c5-9f32-ffb6fdf47060 / ext3 defaults 1 1
UUID=8b9822b6-a92e-48c9-96b5-f8943142319e /boot ext3 defaults 1 2
UUID=94b03547-a7e3-45bb-b2d5-837498b370f4 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
xenfs /proc/xen xenfs defaults 0 0
UUID=a1487063-2a3f-43b1-3a36-44936b0b4d1e /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

Please note; At the time of writing this HowTo, there is a bug in findfs and mount. According to [http://www.ietf.org/rfc/rfc4122.txt RFC 4122], programs should accept a [[UUID]] in either upper or lower case. However, this is not currently the case, so you '''must''' pass the UUID in lower-case. Please see bugs [https://bugzilla.redhat.com/show_bug.cgi?id=632373 632373] and [https://bugzilla.redhat.com/show_bug.cgi?id=632385 632385].

== Testing The gfs2 Initialization Script ==

To verify that the new entry is valid, check gfs2's status.

<source lang="bash">
/etc/init.d/gfs2 status
</source>
<source lang="text">
Configured GFS2 mountpoints:
/xen_shared
Active GFS2 mountpoints:
/xen_shared
</source>

Now test stopping and restarting to ensure that the GFS2 partition unmounts and mounts properly.

Stop gfs2.

<source lang="bash">
/etc/init.d/gfs2 stop
</source>
<source lang="text">
Unmounting GFS2 filesystem (/xen_shared): [ OK ]
</source>

Check with df -h to ensure that the mount is gone.

<source lang="bash">
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.0G 35G 6% /
tmpfs 466M 23M 444M 5% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
</source>

Start gfs2 again.

<source lang="bash">
/etc/init.d/gfs2 start
</source>
<source lang="text">
Mounting GFS2 filesystem (/xen_shared): [ OK ]
</source>

Again, check with df -h that it has been remounted.

<source lang="bash">
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.0G 35G 6% /
tmpfs 466M 29M 438M 7% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
/dev/dm-0 20G 259M 20G 2% /xen_shared
</source>

Done!

== Further Reading ==

* [[Grow a GFS2 Partition]]
* [[Hard drive has gone bad in DRBD]]

= Altering Daemon Start Order =

It is important that the various daemons in use by our cluster start in the right order. Most daemons will rely on services provided by another daemon to be running, and will not start or will not operate reliably otherwise.

We need to make sure that xend starts so that the network is stable. Then cman needs to start so that [[fencing]] and [[dlm]] are available. Next, drbd starts so that the clustered storage is available. Then clvmd must start so that the data on the DRBD resource is accessible. Now gfs2 needs to start so that the Xen domU configuration files can be found and finally xendomains must start to boot up the actual domU virtual machines.

To restate as a list, the start order must be:
* xend
* cman
* drbd
* clvmd
* gfs2
* xendomains

To make sure the start order is sane then, we'll edit each of the six daemon's init scripts and alter their Required-Start lines. To make the changes take effect, we will use chkconfig to remove and re-add them to the various start levels.

== Altering xend ==

This should already be done. If it isn't, please see "[[#Make xend play nice with clustering|Making xend play nice with clustering]]" above. If you are revisiting that section, you can skip the cman edit as we will need to make another change in the next step.

== Altering cman ==

This should already be done. If it isn't, please see "[[#Make xend play nice with clustering|Making xend play nice with clustering]]" above. If you are revisiting that section, you can skip the cman edit as we will need to make another change in the next step.

== Altering drbd ==

Now we will tell drbd to start after cman.

This requires the additional step of altering the chkconfig: - 70 08 line to instead read chkconfig: - 20 08. This isn't strictly needed, but will give more room for chkconfig to order the dependent daemons by allowing DRBD to be started as low as position 20, rather than waiting until position 70. This is somewhat more compatible with cman and clvmd which normally start at positions 21 and 24, respectively

<source lang="bash">
vim /etc/init.d/drbd
</source>
<source lang="text">
#!/bin/bash
#
# chkconfig: - 20 08
# description: Loads and unloads the drbd module
#
# Copright 2001-2008 LINBIT Information Technologies
# Philipp Reisner, Lars Ellenberg
#
### BEGIN INIT INFO
# Provides: drbd
# Required-Start: $local_fs $network $syslog cman
# Required-Stop: $local_fs $network $syslog
# Should-Start: sshd multipathd
# Should-Stop: sshd multipathd
# Default-Start:
# Default-Stop:
# Short-Description: Control drbd resources.
### END INIT INFO
</source>

== Altering clvmd ==

Now we will now tell clvmd to start after drbd.

'''Note''': There is currently a minor bug with lvm2-cluster version 2.02.73-2 in that /etc/init.d/clvmd is set by default to mode 0555. This is easily corrected by running the following command. Please check bug [https://bugzilla.redhat.com/show_bug.cgi?id=636066 636066] to see if this has been resolved.

<source lang="bash">
chmod u+w /etc/init.d/clvmd
</source>

Once you've got write access, edit the file.

<source lang="bash">
vim /etc/init.d/clvmd
</source>
<source lang="text">
#!/bin/bash
#
# chkconfig: - 24 76
# description: Starts and stops clvmd
#
# For Red-Hat-based distributions such as Fedora, RHEL, CentOS.
#
### BEGIN INIT INFO
# Provides: clvmd
# Required-Start: $local_fs drbd
# Required-Stop: $local_fs
# Default-Start:
# Default-Stop: 0 1 6
# Short-Description: Clustered LVM Daemon
### END INIT INFO
</source>

== Altering gfs2 ==

Now we will now tell gfs2 to start after clvmd. You will notice that cman is already listed under Required-Start and Required-Stop. It's true that cman must be started, but we've created a chain here so we can safely replace it with clvmd in the start line.

<source lang="bash">
vim /etc/init.d/gfs2
</source>
<source lang="text">
#!/bin/bash
#
# gfs2 mount/unmount helper
#
# chkconfig: - 26 74
# description: mount/unmount gfs2 filesystems configured in /etc/fstab

### BEGIN INIT INFO
# Provides: gfs2
# Required-Start: $network clvmd
# Required-Stop: $network
# Default-Start:
# Default-Stop:
# Short-Description: mount/unmount gfs2 filesystems configured in /etc/fstab
# Description: mount/unmount gfs2 filesystems configured in /etc/fstab
### END INIT INFO
</source>

== Altering xendomains ==

Finally, we will alter xendomains so that it starts last, after gfs2.

<source lang="bash">
vim /etc/init.d/xendomains
</source>
<source lang="text">
#!/bin/bash
#
# /etc/init.d/xendomains
# Start / stop domains automatically when domain 0 boots / shuts down.
#
# chkconfig: 345 99 00
# description: Start / stop Xen domains.
#
# This script offers fairly basic functionality. It should work on Redhat
# but also on LSB-compliant SuSE releases and on Debian with the LSB package
# installed. (LSB is the Linux Standard Base)
#
# Based on the example in the "Designing High Quality Integrated Linux
# Applications HOWTO" by Avi Alkalay
# <http://www.tldp.org/HOWTO/HighQuality-Apps-HOWTO/>
#
### BEGIN INIT INFO
# Provides: xendomains
# Required-Start: $syslog $remote_fs xend gfs2
# Should-Start:
# Required-Stop: $syslog $remote_fs xend
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop secondary xen domains
# Description: Start / stop domains automatically when domain 0
# boots / shuts down.
### END INIT INFO
</source>

== Applying The Changes ==

Change the start order by removing and re-adding all cluster-related daemons using chkconfig.

<source lang="bash">
chkconfig xenstored off; chkconfig xenconsoled off; chkconfig xend off; chkconfig cman off; chkconfig drbd off; chkconfig clvmd off; chkconfig gfs2 off; chkconfig xendomains off
chkconfig xendomains on; chkconfig gfs2 on; chkconfig clvmd on; chkconfig drbd on; chkconfig cman on; chkconfig xend on; chkconfig xenconsoled on; chkconfig xenstored on
</source>

Now verify that the start order is as we want it.

<source lang="bash">
ls -lah /etc/rc3.d/
</source>
<source lang="text">
lrwxrwxrwx. 1 root root 19 Sep 20 13:37 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 21 Sep 20 13:37 S27xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S28xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S29cman -> ../init.d/cman
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S70drbd -> ../init.d/drbd
lrwxrwxrwx. 1 root root 15 Sep 20 13:37 S71clvmd -> ../init.d/clvmd
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S72gfs2 -> ../init.d/gfs2
lrwxrwxrwx. 1 root root 20 Sep 20 13:37 S99xendomains -> ../init.d/xendomains
</source>

= Setting Up Xen =

'''''WARNING''''': Everything below here is pretty '''seriously screwed up'''.

'''Note''': This is not meant to be an extensive tutorial on Xen itself. It covers enough to get domU VMs provisioned in a manner that will take advantage of the cluster. As such, there is minimal explanation of configuration file options. If you need further help, please drop by the ##xen (yes, two ##) [[IRC]] channel on freenode.org.

== Install The Hypervisor Tools ==

These tools are very useful in provisioning and managing domU VMs.

<source lang="bash">
yum -y install virt-install virt-viewer
</source>

== Install The HVM/KVM Tools ==

For hvm (Hardware Virtual Machines), which is required for [http://www.virtuatopia.com/index.php/Virtualizing_Windows_Server_2008_with_Xen paravirtualized Microsoft] VMs, you must install the following packages as well.

<source lang="bash">
yum -y install qemu-kvm.x86_64 qemu-kvm-tools.x86_64
</source>

== Ensure That Virtualization Is Enabled ==

Many motherboards disable hvm by default in their [[BIOS]]. Assuming that you've got a dom0 kernel running at this stage, you can check if this is the case by checking the xm info output.

<source lang="bash">
xm info
</source>
<source lang="text">
host : an-node04.alteeve.com
release : 2.6.32.23-170.dom0_an1.fc13.x86_64
version : #1 SMP Sun Oct 10 20:39:19 EDT 2010
machine : x86_64
nr_cpus : 4
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 2209
hw_caps : 178bf3ff:efd3fbff:00000000:00001310:00802001:00000000:000037ff:00000000
virt_caps : hvm
total_memory : 4063
free_memory : 2987
node_to_cpu : node0:0-3
node_to_memory : node0:2987
node_to_dma32_mem : node0:2928
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline : dom0_mem=1024M
cc_compiler : gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC)
cc_compile_by : root
cc_compile_domain :
cc_compile_date : Mon Oct 11 01:10:38 EDT 2010
xend_config_format : 4
</source>

Look at the virt_caps and xen_caps lines. Notice the hvm entries? This shows that HVM, also known as "secure virtualization", has been enabled. If you do not see this, please check your mainboard manual for information on enabling this on your system.

'''Note''': The next paragraph applies only when running a vanilla kernel.

If you are running a vanilla kernel, you can check to see if your CPU has support for HVM guests but checking /proc/cpuinfo. What you're looking for depends on your CPU manufacturer. If you have an Intel CPU, you need to look for the vmx flag. Likewise, with AMD CPUs, you need to look for the svm flag.

For a more complete, if somewhat dated paper on this topic, please [http://fedoraproject.org/wiki/FedoraXenQuickstartFC6#Fully-virtualized_guests_.28HVM.2FIntel-VT.2FAMD-V.29 Fedora 6 Xen Quickstart Guide, System Requirements].

== Enabling Migration ==

By default, xend will not allow domU VMs from being migrated onto or off of a given dom0 host. Given that we've got a cluster though, we very much want this behaviour, so now we will enable it. This is done by making edits to /etc/xen/xend-config.sxp. Below is a concise list of options that must be set. Some exist already in the file and need to be commented out or altered.

'''Warning''': The values below are '''very''' permissive. Please review each option and improve the security to fit your network before going into production!

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>
<source lang="text">
(xend-http-server yes)
(xend-unix-server yes)
(xend-tcp-xmlrpc-server yes)
(xend-relocation-server yes)
(xend-udev-event-server yes)
(xend-port 8000)
(xend-relocation-port 8002)
(xend-address '')
(xend-relocation-address '')
(xend-relocation-hosts-allow '')
</source>

Once done, restart xend. It is usually safest to stop the cluster before hand to avoid accidental fencing caused by the underlying network being reconfigured.

<source lang="bash">
/etc/init.d/gfs2 stop
/etc/init.d/clvmd stop
/etc/init.d/cman stop
/etc/init.d/xend restart
/etc/init.d/cman start
/etc/init.d/clvmd start
/etc/init.d/gfs2 start
</source>

== Virtual Machine Naming Convention ==

'''Note''': This section acts as a recommendation only. Feel free to alter this to fit your style and needs.

Personally, I like to name my VMs similar to c5_shorewall_01. To elaborate, I like to use the format:
* os_role_seq (<Operating System ID>_<Role of the VM>_<Sequence Integer>).

There are no (known) restrictions on virtual machine names, so feel free to use names that made sense for you. I do strongly recommend that you match the name of your domU VM to the name of it's host LVM logical volume.

== Provisioning domU VMs ==

There are two ways to provision new VMs. The first examples below show provisioning of VMs using the 

<source lang="bash">
# Fedora 13 x86_64 RPM builder VM
lvcreate -L 40G -n f13_builder_01 drbd0_vg0
virt-install --connect xen \
--name f13_builder_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--location http://192.168.1.10/f13/x86_64/img/ \
--os-type linux \
--os-variant fedora13 \
--disk path=/dev/drbd0_vg0/f13_builder_01 \
--network bridge=eth0,mac=00:16:3e:00:10:01 \
--vnc \
--paravirt
# Backup the config to an XML file.
virsh dumpxml f13_builder_01 > /xen_shared/domU_config/f13_builder_01.xml
# Convert it to a "traditional" python script. Be sure the edit the resulting .cfg file to remove the 'vifname=' sections.
virsh -c xen:/// domxml-to-native xen-xm f13_builder_01.xml > f13_builder_01.cfg

# A CentOS test server
lvcreate -L 40G -n c5_test_01 drbd0_vg0
virt-install --connect xen \
--name c5_test_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1-3 \
--location http://192.168.1.10/c5/x86_64/img/ \
--os-type linux \
--os-variant rhel5.4 \
--disk path=/dev/drbd0_vg0/c5_test_01 \
--network bridge=eth0,mac=00:16:3e:00:10:02 \
--vnc \
--paravirt
# Backup the config to an XML file.
virsh dumpxml c5_shorewall_01 > /xen_shared/domU_config/c5_shorewall_01.xml

# Red Hat Enterprise Linux 6 beta 2 test server
lvcreate -L 40G -n rh6b2_test_01 drbd0_vg0
virt-install --connect xen \
--name rh6b2_test_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--location http://192.168.1.10/rhel6/x86_64/img/ \
--os-type linux \
--os-variant rhel6 \
--disk path=/dev/drbd0_vg0/rh6b2_test_01 \
--network bridge=eth0,mac=00:16:3e:00:10:03 \
--vnc \
--paravirt
</source>

HVM + ISO-based install of Windows 2008 Server. This uses a config file right off the bat as there is a problem with virt-install finding the qemu-dm file. You'll not for the same reason that the arch_libdir variable is not used. I will try to resolve this later.

This configuration file will be saved in the /xen_shared/domU_config directory, which exists on the shared [[GFS2]] partition we created earlier.

<source lang="bash">
mkdir /xen_shared/domU_config
vim /xen_shared/domU_config/win2008_sql_01.cfg
</source>
<source lang="python">
# This is the Windows 2008 Enterprise Server x86_64 hosting MS SQL Server 2008 Enterprise
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
memory = 1024

# Should be at least 2KB per MB of domain memory, plus a few MB per vcpu.
shadow_memory = 8
name = "win2008_sql_01"
#vif = [ 'type=ioemu, bridge=xenbr0' ]
vif = [ 'type=ioemu, bridge=eth0,mac=00:16:3e:00:30:03' ]
acpi = 1
apic = 1
# Remove the 'file:...' entry (or change it to another ISO) after the install is complete.
disk = [ 'phy:/dev/drbd0_vg0/win2008_sql_01,hda,w', 'file:/xen_shared/iso/MS-Win2008-Ent-x86_64-SP2.iso,hdc:cdrom,r' ]

device_model = '/usr/lib/xen/bin/qemu-dm'

#-----------------------------------------------------------------------------
# boot on floppy (a), hard disk (c) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
boot="dc"
sdl=0
vnc=1
vncconsole=1
vncpasswd=''

serial='pty'
usbdevice='tablet'
</source>
<source lang="bash">
xm create -f /xen_shared/domU_config/win2008_sql_01.cfg
</source>
Convert the "traditional" config into an XML
<source lang="bash">

</source>

=== Install virt-manager ===

We will now start using a graphical tool called virt-manager. It simplifies the creation of domU VMs quite a bit.

<source lang="bash">
yum install virt-manager
</source>

Be sure that you've enabled X-Forwarding if you've ssh'ed into your nodes. This may require modifying your /etc/sshd_config file and changing X11Forwarding to yes. Then when you connect to the node, you will need to use the ssh -X user@node switch.

We'll come back to this program shortly.

=== Create The CLVM LV ===

We will create a new [[CLVM]] [[LV]] for this [[VM]]. It will act as the VM's hard drive, so it needs to have enough capacity for the OS and any data you expect the VM to need. As we're creating a hypothetical router/firewall, we will not need much space. We'll use 20 [[GiB]], which should be more than enough.

As before, this will only need to be done on one node. Being that it is on top of the DRBD resource, it will be visible to both nodes.

'''Node1'''

<source lang="bash">
lvcreate -L 20G -n c5_test_01 drbd0_vg0
</source>
<source lang="text">
Logical volume "c5_test_01" created
</source>

'''Both'''

Verify that the new LV is indeed visible from both nodes now.

<source lang="bash">
lvdisplay
</source>
<source lang="text">
--- Logical volume ---
LV Name /dev/drbd0_vg0/xen_shared
VG Name drbd0_vg0
LV UUID yjt41B-cdc4-LGV4-Ds0m-eRVv-FIY7-3FK4Kb
LV Write Access read/write
LV Status available
# open 1
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0

--- Logical volume ---
LV Name /dev/drbd0_vg0/c5_test_01
VG Name drbd0_vg0
LV UUID eOcBZz-nJzz-wHwg-4cBL-ZZHm-1cm6-Sf82xC
LV Write Access read/write
LV Status available
# open 0
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1
</source>

Now we've got the space, we're ready to provision!

=== Connecting To The Hypervisor From virt-manager ===

At this point, we'll be using the graphical virt-manager tool.

==== Running virt-manager Directly On a Node ====

If you are working on the nodes directly, you can run virt-manager but you will need to have Gnome or KDE started first. The easiest way to do this is to switch to run level 5. As an unprivileged user, run the following command.

<source lang="bash">
startx
</source>

From there, click on Application -> System Tools -> Virtual machine Manager

[[Image:virt-manager_local-node_01.png|thumb|500px|center|Where to find virt-manager in Fedora 13]]

==== Running virt-manager Remotely Over X-Forwarding ====

Some features in virt-manager don't work well unless they are run off of the local machine. The best way to accomplish this from a remote system is to use X-Forwarding, which was discussed in the [[#Install virt-manager|Install virt-manager]] above.

Generally speaking, I prefer to provision VMs using virt-manager directly on a cluster node. This avoids some problems, like accessing underlying LVM volumes. So to do this when you can't directly sit at a node, ssh into your node using the -X switch.

<source lang="bash">
ssh -X root@an-node01
</source>
<source lang="text">
Last login: Sun Sep 19 22:02:38 2010
[root@an-node01 ~]#
</source>

Now start virt-manager at the command line and wait for it to open on your local workstation.

<source lang="bash">
virt-manager
</source>

'''Note''': If your local workstation has a different version of X server or Gnome, the virt-manager window could look fairly primitive. It will work just the same though, and all the icons will be in the same place.

==== Running virt-manager Locally On a Workstation ====

The virt-manager tool can be installed on a computer outside of the cluster and then used to connect to the hypervisor on either node. This is very useful for daily usage of your virtual machines, but isn't recommended during the provisioning stage. In some cases, running virt-manager remotely will cause it to not see the LVM volumes properly, preventing you from choosing the LV for your new VM.

To connect to the hypervisor on the cluster nodes, click on File -> Add Connection. Change the Hypervison to Xen and then change the Connection to Remote tunnel over SSH.

[[Image:virt-manager_remote-workstation_01.png|thumb|500px|center|Adding a remote connection in virt-manager from a workstation]]

In some cases, not all of the nodes on the network will be seen. In the screen shot above, you can see that an-node02 can be seen, but an-node01 wasn't. To add the missing one, just type node-name.local (an-node01.local above). Once selected, hit Connect.

Once the two nodes have been connected, you should see them on the virt-manager main page.

[[Image:virt-manager_remote-workstation_02.png|thumb|500px|center|Remote connection in virt-manager to both nodes in the cluster]]

Once you seen your nodes, you can proceed to use virt-manager the same as if you were running it on one of the nodes directly.

=== Using virt-manager To Provision Our VM ===

How you use virt-manager will be the same regardless of how or where you started it. The following steps will walk you through the program's use.

With the Virtual Machine Manager main menu connected to localhost (QEMU), we can now begin the process of provisioning a new Xen domU virtual machine! The top-left icon, which looks like a monitor with the "play" button on it's screen, will be used to provision our virtual machine.

[[Image:virt-manager_local-node_02.png|thumb|500px|center|The virt-manager main menu]]

'''Madi''': You left off here.

Once you press the Create a new virtual machine button you will be asked to name the VM and to choose the install type. Even if you have a [[PXE]] server, choose Network Install.

[[Image:virt-manager_provision_02.png|thumb|500px|center|Provisioning a Xen domU - Naming the VM and choosing the install source]

=== Storing The domU Configuration Files ===

'''ToDo'''

'''Node1'''

As this is our first [[domU]], we'll start by creating our shared configuration directory on the /xen_shared [[GFS2]] partition we just created.

<source lang="bash">
mkdir /xen_shared/config
</source>

To confirm again that the GFS2 partition is working, verify that this directory is visible from both nodes.

<source lang="bash">
ls -lah /xen_shared/
</source>
<source lang="text">
total 20K
drwxr-xr-x. 3 root root 3.8K Sep 13 12:57 .
dr-xr-xr-x. 27 root root 4.0K Sep 13 11:32 ..
drwxr-xr-x. 2 root root 3.8K Sep 13 12:57 config
</source>


<source lang="bash">
</source>
<source lang="text">
</source>

{{footer}}

Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

2010-10-13T19:51:14Z

SRSullivan: /* Adding New NICs to Xen */ Clarity about eth1

{{howto_header}}

'''Warning''': This is currently a dumping ground for notes. '''''DO NOT FOLLOW THIS DOCUMENT'S INSTRUCTIONS'''''. Seriously, it could blow up your computer or cause winter to come early.

----

This HowTo will walk you through setting up [[Xen]] [[VM]]s using [[DRBD]] and [[CLVM]] for high availability.

= Prerequisite =

This talk is an extension of the [[Two Node Fedora 13 Cluster]] HowTo. As such, you will be expected to have a freshly built two-node cluster with spare disk space on either node.

Please do not proceed until you have completed the first tutorial.

= Overview =

This tutorial will cover several topics; [[DRBD]], [[CLVM]], [[GFS2]], [[Xen]] [[dom0]] and [[domU]] [[VM]]s and [[rgmanager]]. Their relationship is thus:

* DRBD provides a mechanism to replicate data across both nodes in real time and guarantees a consistent view of that data from either node. Think of it like [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]], but across machines.
* CLVM sits on the DRBD partition and provides the underlying mechanism for allowing both nodes to access shared data in a clustered environment. It will host a shared filesystem by way of GFS2 as well as [[LV]]s that Xen's domU VMs will use as their disk space.
* GFS2 will be the clustered file system used on one of the DBRD-backed, CLVM-managed partitions. Files that need to be shared between nodes, like the Xen VM configuration files, will exist on this partition.
* Xen will be the hypervisor in use that will manage the various virtual machines. Each virtual machine will exist in an LVM LV.
** Xen's dom0 is the special "host" virtual machine. In this case, dom0 will be the OS installed in the first HowTo.
** Xen's domU virtual machines will be the "floating", highly available servers.
* Lastly, rgmanager will be the component of [[cman]] that will be configured to manage the automatic migration of the virtual machines when failures occur and when nodes recover.

= Setting Up Xen =

It may seem odd to start with [[Xen]] at this stage, but it is going to rather fundamentally alter each node's "host" operating system.

At this point, each node's host OS is a traditional operating system operating on the bare metal. When we install a dom0 kernel though, we tell Xen to boot a mini operating system first, and then to boot our "host" operating system. In effect, this converts the host node's operating system into just another virtual machine, albeit with a special view of the underlying hardware and Xen hypervisor.

This conversion is somewhat disruptive, so I like to get it out of the way right away. We will then do the rest of the setup before returning to Xen later on to create the floating virtual machines.

== A Note On The State Of Xen dom0 Support In Fedora ==

As of Fedora 8, support for Xen [[dom0]] has been removed, but support for the hypervisor and [[domU]] virtual machines remains. Red Hat's position is that KVM will be the supported platform going forward. That said, [http://fedoraproject.org/wiki/Features/XenPvopsDom0 this page] seems to indicate that PV Ops dom0 kernels will be supported in the future. Specifically, when dom0 support is merged into the mainline Linux kernel. When this will be is open to speculation, though "by Fedora 16" seems to be a reasonable educated guess.

What this means for us is that we need to use a non-standard dom0 kernel. Specifically, we will use a kernel created by [http://myoung.fedorapeople.org/ myoung] (Micheal Young) for Fedora 12. This kernel does not directly support DRBD, so be aware that we will need to build new DRBD kernel modules for his kernel and then rebuild the DRBD modules each time his kernel is updated.

== A Note On Rolling Your Own RPMs ==

If you want to roll the source RPMs for both the hypervisor and the [[dom0]] kernel, you will need to make both before you can boot into your dom0 for the first time. This is because the dom0 kernel needs the Xen microkernel provided by the xen hypervisor package to boot.

== Install The Hypervisor ==

We will use [[Xen]] 4.0.1, as provided by [http://pasik.reaktio.net/fedora/ Pasik]. We'll use the [http://pasik.reaktio.net/fedora/xen-4.0.1-0.2.fc13.src.rpm source RPM] to build our own RPM.

Regardless of whether you install from source RPMs or the pre-compiled ones, you will need to install the libvirt, qemu, SDL and PyXML packages from the standard repositories before you can proceed.

<source lang="bash">
yum -y install libvirt PyXML.x86_64 qemu.x86_64 SDL.x86_64
</source>

Please don't remove existing xen utilities and libraries prior to installing the newer version. If you do, core clustering components may be removed. Instead, be sure to use the rpm -Uvh switches to upgrade the existing packages that may be installed already.

=== Installing Prebuilt RPMs ===

These are locally stored copies of the Xen RPMs built for [[x86_64]] on Fedora 13. This requires some dependent packages be installed first.

'''Warning''': These are all recompiled for this website against Fedora 13, x86_64. If you feel more comfortable, please use [[RPM]]s from a source you are familiar with.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/xen-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-doc-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-hypervisor-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-libs-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-runtime-4.0.1-0.2.fc13.x86_64.rpm
rpm -Uvh xen*4.0.1-0.2.fc13.x86_64.rpm
</source>

Optionally, you may wish to install debug and/or devel RPMs as well.

<source lang="bash">
wget -c https://alteeve.com/files/xen-debuginfo-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/xen-devel-4.0.1-0.2.fc13.x86_64.rpm
rpm -Uvh xen-d*4.0.1-0.2.fc13.x86_64.rpm
</source>

=== Building RPMs From Source ===

To build the RPMs from the source, you need to make sure that you have the build environment installed.

'''Note''': If you are following these instruction after having installed a prior dom0 kernel, you will need to comment out exclude=kernel* in /etc/yum.conf so that the kernel-header package can be installed.

<source lang="bash">
vim /etc/yum.conf
</source>
<source lang="text">
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
#exclude=kernel*
</source>

Now install the development environment.

<source lang="bash">
yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
yum install transfig texi2html SDL-devel libX11-devel tetex-latex gtk2-devel libaio-devel dev86 iasl xz-devel e2fsprogs-devel glibc-devel.i686 xmlto asciidoc elfutils-libelf-devel
</source>

'''Note''': If you had to uncomment the exclude=kernel* line earlier, comment it back out now before you forget.

<source lang="bash">
vim /etc/yum.conf
</source>
<source lang="text">
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
exclude=kernel*
</source>

Now we can now build the RPMs from source.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/xen-4.0.1-0.2.fc13.src.rpm
rpm -ivh xen-4.0.1-0.2.fc13.src.rpm
cd /root/rpmbuild/SPECS/
rpmbuild -ba xen.spec
</source>

Once this is done, you will have seven RPMs built. Two of them are not really needed (xen-debuginfo-4.0.1-0.2.fc13.x86_64.rpm and xen-devel-4.0.1-0.2.fc13.x86_64.rpm) and I will leave it up to you may wish to install them or not. If you don't, modify the next command or move the debug and devel RPMs out of the way first.

<source lang="bash">
cd ~/rpmbuild/RPMS/x86_64/
rpm -Uvh xen*4.0.1-0.2.fc13.x86_64.rpm
</source>

== Installing The AN!Cluster dom0 Kernel ==

'''Notice'''!

The kernel provided here was recompiled on Fedora 13 and is a slightly modified version of [[#Installing Micheal Young's dom0 Kernel|Micheal Young's]] kernel available below. I was originally driven to recompile in an effort to solve a DRBD-related kernel oops. For now, unless you have the same DRBD kernel oops, I'd strongly recommend against using the AN!Cluster dom0 kernel until it has been tested much more thoroughly.

With that warning out of the way...

This kernel was compiled on a Fedora 13, x86_64. The DRBD RPMs available a little later where compiled against this dom0 kernel.

'''Note''': The --force is required because the current kernel is newer than 2.6.32 used here. Without this switch, the RPM would not install.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
</source>

If you would like to install the debug, devel and/or header RPMs for this kernel, they are available below.

'''Note''': The kernel-debuginfo-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm RPM is 213 [[MiB]].

<source lang="bash">
wget -c https://alteeve.com/files/an-cluster/kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-debuginfo-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-de*-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
</source>

=== Post AN!Cluster dom0 Install Configuration ===

The entry in grub's /boot/grub/menu.lst won't work. You will need to edit it so that it calls the existing installed operating system as a module.

'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your root= is likely different and your rd_MD_UUID= will definitely be different, even on the same machine across installs. Generally speaking, what follows the kernel /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ... entry made by the dom0 kernel can be copied after the module /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ... entry in the example below.

<source lang="bash">
vim /boot/grub/menu.lst
</source>
<source lang="bash">
title Xen 4.0.x, Linux 2.6.32.23-170.dom0_an1.fc13.x86_64
root (hd0,0)
kernel /xen.gz dom0_mem=1024M
module /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ...
module /initramfs-2.6.32.23-170.dom0_an1.fc13.x86_64.img

</source>

== Installing Micheal Young's dom0 Kernel ==

This uses a kernel built for Fedora 12, but it works on Fedora 13. This step involves either installing it over HTML or adding and enabling his repository and then installing it from there.

=== Installing Via myoung's Repository ===

This is almost always the preferred method. However, do note that when myoung updates his kernel, there will be a lag where the dom0 dependent RPMs provided here will no longer be compatible.

To add the repository, download the myoung.dom0.repo into the /etc/yum.repos.d/ directory.

<source lang="bash">
cd /etc/yum.repos.d/
wget -c http://myoung.fedorapeople.org/dom0/myoung.dom0.repo
</source>

To enable his repository, edit the repository file and change the two enabled=0 entries to enabled=1.

<source lang="bash">
vim /etc/yum.repos.d/myoung.dom0.repo
</source>
<source lang="text">
[myoung-dom0]
name=myoung's repository of Fedora based dom0 kernels - $basearch
baseurl=http://fedorapeople.org/~myoung/dom0/$basearch/
enabled=1
gpgcheck=0

[myoung-dom0-source]
name=myoung's repository of Fedora based dom0 kernels - Source
baseurl=http://fedorapeople.org/~myoung/dom0/src/
enabled=1
gpgcheck=0
</source>

Install the [[Xen]] [[dom0]] kernel (edit the version number with the updated version if it has changed).

<source lang="bash">
yum install kernel-2.6.32.21-170.xendom0.fc12.x86_64
</source>

=== Post Michael Young's dom0 Install Configuration ===

The entry in grub's /boot/grub/menu.lst won't work. You will need to edit it so that it calls the existing installed operating system as a module.

'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your root= is likely different and your rd_MD_UUID= will definitely be different, even on the same machine across installs. Generally speaking, what follows the kernel /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ... entry made by the dom0 kernel can be copied after the module /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ... entry in the example below.

<source lang="bash">
vim /boot/grub/menu.lst
</source>
<source lang="bash">
title Xen 4.0.x, Linux kernel 2.6.32.21-170.xendom0.fc12.x86_64
root (hd0,0)
kernel /xen.gz dom0_mem=1024M
module /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ...
module /initramfs-2.6.32.21-170.xendom0.fc12.x86_64.img
</source>

=== Disabling Automatic Kernel Updates ===

Seeing as we're using an older kernel, yum will want to replace it whenever there is an updated kernel* package available. Likewise if myoung updates his kernel. In the latter case, the updated kernel from Mr. Young would break compatibility with our DRBD module. So to be safe, we want to tell yum to never update the kernel.

To do this, we need to add exclude=kernel* to the /etc/yum.conf file.

<source lang="bash">
echo "exclude=kernel*" >> /etc/yum.conf
cat /etc/yum.conf
</source>
<source lang="text">
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=3
color=never

# This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
# It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m

# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d

exclude=kernel*
</source>

=== Make xend play nice with clustering ===

By default under Fedora 13, cman will start before xend. This is a problem because xend takes the network down as part of it's setup. This causes [[totem]] communication to fail which leads to fencing.

'''Note''': Move xenconsoled and xenstore to 09 and 10 start positions and then make xend depend on the before starting.

To avoid this, edit the initialization scripts for /etc/init.d/xend and it's dependents xenconsoled and xenstore to have a lower minimum start position. We need to maintain the start order of xenstore first, xenconsoled second and lastly xend. By default, their minimum start positions are 96, 97 and 98 respectively. We will change these to 10, 11 and 12, again, respectively.

Note that we are '''not''' altering the start position of xendomains! This is intentional as this daemon will start the [[domU]] VMs. This can not happen until all other cluster related daemons have started.

To change the start order we will change the line chkconfig: 2345 9x 01 lines to chkconfig: 2345 1x 01, where x is the given daemon's start number. Further, we'll make sure that xenstored begins first by add it to xenconsoled's Required-Start line. We'll then make sure that xenconsoled starts before xend by adding it to xend's Required-Start line.

To recap the changes;
* xenstored will start first.
** We'll change it's start position from 96 to 10.
** We will not add anything to it's Required-Start as it must be the first daemon to come up.
* xenconsoled will start second.
** We'll change it's start position from 97 to 11.
** We will add xenstored to it's Required-Start line.
* xend will start third.
** We'll change it's start position from 98 to 12.
** We will add xenconsoled to it's Required-Start line.

When done, the three initialization scripts should look like the examples below.

<source lang="bash">
vim /etc/init.d/xenstored
</source>
<source lang="bash">
#!/bin/bash
#
# xenstored Script to start and stop the Xen control daemon.
#
# Author: Daniel Berrange <berrange@redhat.com
#
# chkconfig: 2345 10 01
# description: Starts and stops the Xen xenstored daemon.
### BEGIN INIT INFO
# Provides: xenstored
# Required-Start: $syslog $remote_fs
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xenstored
# Description: Starts and stops the Xen xenstored daemon.
### END INIT INFO
</source>

<source lang="bash">
vim /etc/init.d/xenconsoled
</source>
<source lang="bash">
#!/bin/bash
#
# xenconsoled Script to start and stop the Xen xenconsoled daemon
#
# Author: Daniel P. Berrange <berrange@redhat.com>
#
# chkconfig: 2345 11 01
# description: Starts and stops the Xen control daemon.
### BEGIN INIT INFO
# Provides: xenconsoled
# Required-Start: $syslog $remote_fs xenstored
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xenconsoled
# Description: Starts and stops the Xen xenconsoled daemon.
### END INIT INFO
</source>

<source lang="bash">
vim /etc/init.d/xend
</source>
<source lang="bash">
#!/bin/bash
#
# xend Script to start and stop the Xen control daemon.
#
# Author: Keir Fraser <keir.fraser@cl.cam.ac.uk>
#
# chkconfig: 2345 12 98
# description: Starts and stops the Xen control daemon.
### BEGIN INIT INFO
# Provides: xend
# Required-Start: $syslog $remote_fs xenconsoled
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xend
# Description: Starts and stops the Xen control daemon.
### END INIT INFO
</source>

With xend set to start at a position lower than 98, we now have room for chkconfig to put other daemons after it in the start order, which will be needed a little later. First and foremost, we now need to tell cman to not start until after xend is up.

As above, we will now edit cman's /etc/init.d/cman script. This time though, we will not edit it's chkconfig line. Instead, we will simply add xend to the Required-Start line.

<source lang="bash">
vim /etc/init.d/cman
</source>
<source lang="bash">
#!/bin/bash
#
# cman - Cluster Manager init script
#
# chkconfig: - 21 79
# description: Starts and stops cman
#
#
### BEGIN INIT INFO
# Provides: cman
# Required-Start: $network $time xend
# Required-Stop: $network $time
# Default-Start:
# Default-Stop:
# Short-Description: Starts and stops cman
# Description: Starts and stops the Cluster Manager set of daemons
### END INIT INFO
</source>

Finally, remove and re-add the xend and cman daemons to re-order them in the start list:

<source lang="bash">
chkconfig xenstored off; chkconfig xenconsoled off; chkconfig xend off; chkconfig cman off;
chkconfig xenstored on; chkconfig xenconsoled on; chkconfig xend on; chkconfig cman on
</source>

Confirm that the order has changed so that xend is earlier in the boot sequence than cman. Assuming you've switched to run-level 3, run:

<source lang="bash">
ls -lah /etc/rc3.d/
</source>

Your start sequence should now look like:

<source lang="text">
lrwxrwxrwx. 1 root root 19 Sep 15 19:29 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 21 Sep 15 19:29 S27xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 14 Sep 15 19:29 S28xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Sep 15 19:29 S29cman -> ../init.d/cman
</source>

=== Booting Into The New dom0 ===

If everything went well, you should be able to boot the new dom0 operating system. If you watch the boot process closely, you will see that the boot process is different. You should now see the Xen hypervisor boot prior to handing off to the "host" operating system. This can be confirmed once the dom0 operating system has booted by checking that the file /proc/xen/capabilities exists. What it contains doesn't matter at this stage, only that it exists at all.

== Configure Networking ==

Networking in Xen, particularly in a cluster, can be confusing. If you are not familiar with networking in Xen, please review to following article before proceeding.

A note of a major change from previous layouts. In Xen 3.x, ethX would be copied to a virtual interface called vethX. Then the real ethX would be renamed to pethX and the virtual interface vethX would be renamed to ethX to take it's place. Finally, a bridge called xenbrX would be created and the real pethX and virtual ethX would be connected to it.

This has been changed somewhat it that now, by default, ethX is left alone and a simple bridge called virbrX would be created. We'll be changing this to be somewhat similar to the old style.

Specifically, the real ethX will be renamed to pethX. Then a bridge will be created called ethX, which plays the role of dom0's interface '''and''' bridges connections from VMs through pethX and out into the real world.

This is explained in more detail, and with diagrams, in the article below.

* [[Networking in Xen]]

=== Adding New NICs to Xen ===

By default, xend manages eth0 only. We need to add eth2. Personally, I don't like to put the storage network ethernet devices (eth1) under Xen's control as this potentially can cause DRBD problems on xend restart. Whether you add it or not I will leave to your preferences.

You can see which, if any, network devices are under Xen's control by running ifconfig and checking to see if there is a virbrX corresponding to a given ethX device.

<source lang="bash">
ifconfig
</source>
<source lang="text">
eth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet addr:192.168.1.74 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:224261 errors:0 dropped:0 overruns:0 frame:0
TX packets:55174 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:319384110 (304.5 MiB) TX bytes:27348739 (26.0 MiB)
Interrupt:225 Base address:0x8000

eth1 Link encap:Ethernet HWaddr 00:1B:21:72:9B:5A
inet addr:192.168.2.74 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:9b5a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:832 (832.0 b) TX bytes:6234 (6.0 KiB)
Memory:feae0000-feb00000

eth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet addr:192.168.3.74 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:818 (818.0 b) TX bytes:6081 (5.9 KiB)
Memory:fe9e0000-fea00000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

virbr0 Link encap:Ethernet HWaddr 02:23:C8:98:31:17
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:4013 (3.9 KiB)
</source>

In the above example, eth0 has a corresponding virbr0 bridge having it's own subnet. In non-clustered systems, this is fine. For our purposes though, it will not do.

=== Removing The qemu virbr0 Bridge ===

By default, [[QEMU]] creates a bridge called virbr0 designed to connect virtual machines to the first eth0 interface. Our system will not need this, so we will remove it. This bridge is configured in the /etc/libvirt/qemu/networks/default.xml file, so to remove this bridge, simply delete the contents of the file.

<source lang="bash">
cat /dev/null >/etc/libvirt/qemu/networks/default.xml
</source>

The next time you reboot, that bridge will be gone.

'''Madi''': Put in the command to delete the bridge before a reboot.

=== Create /etc/xen/scripts/an-network-script ===

This script will be used by Xen to turn the dom0 ethX interfaces into bridges. All traffic to the bridge, be it from dom0 or domU VMs, will be routeable out of the corresponding pethX device. As domU VMs come online, a hotplug script will create virtual interfaces between this new bridge and the domU's interface(s). Think of the vifX.Y devices as being the network cables you'd normally run between a server and a switch.

Before we proceed, please note three things;
# You don't need to use the file name an-network-script. I suggest this name mainly to keep in line with the rest of the 'AN!x' naming used here.
# If you install convirt or other hypervisor tools, they will likely create their own bridge script.
# Adding eth1 is optional, as we know ahead of time that eth1 will not be made available to any virtual machines as it is dedicated to [[DRBD]]. I'm adding it here because I like having things consistent; Do whichever makes more sense to you.

First, touch the file and then chmod it to be executable.

<source lang="bash">
touch /etc/xen/scripts/an-network-script
chmod 755 /etc/xen/scripts/an-network-script
</source>

Now edit it to contain the following:
<source lang="bash">
vim /etc/xen/scripts/an-network-script
</source>
<source lang="text">
#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=eth0
"$dir/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=eth2
</source>

Now tell Xen to execute that script by editing /etc/xen/xend-config.sxp file and changing the network-script argument to point to this new script (this is line 158 in the default xend-config.sxp script):

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>
<source lang="text">
#(network-script network-bridge)
#(network-script /bin/true)
(network-script an-network-script)
</source>

'''Warning''': The next step may trigger fencing of the nodes! As such, be sure that you're not running anything critical. If unsure, please stop cman or reboot the nodes.

<source lang="bash">
/etc/init.d/cman stop
</source>

Now restart xend.

<source lang="bash">
/etc/init.d/xend restart
</source>

If everything worked, you should now be able to run ifconfig and see that all the ethX devices have matching pethX, virtual and bridge devices.

<source lang="bash">
ifconfig
</source>
<source lang="text">
eth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet addr:192.168.1.74 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:78 errors:0 dropped:0 overruns:0 frame:0
TX packets:57 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9796 (9.5 KiB) TX bytes:12574 (12.2 KiB)

eth1 Link encap:Ethernet HWaddr 00:1B:21:72:9B:5A
inet addr:192.168.2.74 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:9b5a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:832 (832.0 b) TX bytes:6234 (6.0 KiB)
Memory:feae0000-feb00000

eth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet addr:192.168.3.74 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:32 errors:0 dropped:0 overruns:0 frame:0
TX packets:29 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5471 (5.3 KiB) TX bytes:5867 (5.7 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

peth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:224486 errors:0 dropped:0 overruns:0 frame:0
TX packets:55349 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:319406626 (304.6 MiB) TX bytes:27384681 (26.1 MiB)
Interrupt:225 Base address:0x8000

peth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:35 errors:0 dropped:0 overruns:0 frame:0
TX packets:70 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6827 (6.6 KiB) TX bytes:12470 (12.1 KiB)
Memory:fe9e0000-fea00000
</source>

'''Note''': The virbr0 may remain until you reboot your nodes.

If you see something like this, then you are ready to proceed! Now start your cluster back up.

<source lang="bash">
/etc/init.d/cman start
</source>

We're done for now. There is more to do in Xen, but this was all we needed to do in order to proceed with the next several steps. Onces we have the clustered storage online, we'll come back to Xen for the domU setup.

= Building the DRBD Array =

Building the DRBD array requires a few steps. First, raw space on either node must be prepared. Next, DRBD must be told that it is to create a resource using this newly configured raw space. Finally, the new array must be initialized.

== A Map of the Cluster's Storage ==

The layout of the storage in the cluster can quickly become difficult to follow. Below is an [[ASCII]] drawing which should help you see how DRBD will tie in to the rest of the cluster's storage. This map assumes a simple [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]] array underlying each node. If your node has a single hard drive, simply collapse the first two layers into one. Similarly, if your underlying storage is a more complex RAID array, simply expand the number of physical devices at the top level.

<source lang="text">
Node1 Node2
_____ _____ _____ _____
| sda | | sdb | | sda | | sdb |
|_____| |_____| |_____| |_____|
|_______| |_______|
_______ ____|___ _______ _______ ____|___ _______
__|__ __|__ __|__ __|__ __|__ __|__ __|__ __|__
| md0 | | md1 | | md2 | | md3 | | md3 | | md2 | | md1 | | md0 |
|_____| |_____| |_____| |_____| |_____| |_____| |_____| |_____|
| | | | | | | |
___|___ _|_ ____|____ |___________| ____|____ _|_ ___|___
| /boot | | / | | <swap> | | | <swap> | | / | | /boot |
|_______| |___| |_________| ______|______ |_________| |___| |_______|
| /dev/drbd0 |
|_____________|
|
____|______
| clvm PV |
|___________|
|
_____|_____
| drbd0_vg0 |
|___________|
|
_____|_____ ___...____
| | |
___|___ ___|___ ___|___
| lv_X | | lv_Y | | lv_N |
|_______| |_______| |_______|
</source>

== Install The DRBD Tools ==

DRBD has two components; The actual application and tools and the kernel module.

There are two options for installing the DRBD user-land tools at this point; AN!Cluster-built RPMs or using the ones shipped with Fedora. Regardless of which method you choose, you will need to either install the AN!Cluster DRBD kernel module RPMs or else rebuild the source RPMs referenced.

=== Install The AN!Cluster DRBD User-Land Tools ===

I am currently experimenting with ways to solve a DRBD triggered kernel oops in the Xen pvops 2.6.32 kernel. For this reason, I've recompiled the following user-land RPMs under the AN!Cluster variant dom0 kernel RPMs referenced earlier in this paper. If you used the AN! RPMs, then I suggest giving these RPMs a try. However, if you are using myoung's dom0, I recommend sticking to the Fedora-provided user-land DRBD tools.

<source lang="bash">
yum -y install bash-completion heartbeat pacemaker
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-utils-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-xen-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-bash-completion-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-udev-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-heartbeat-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-pacemaker-8.3.7-2.fc13.x86_64.rpm
rpm -ivh drbd*8.3.7-2.fc13.x86_64.rpm
</source>

=== Install The Stock Fedora DRBD User-Land Tools ===

You will need to install the following tools:

<source lang="bash">
yum install drbd.x86_64 drbd-xen.x86_64 drbd-utils.x86_64
</source>

=== Disable heartbeat ===

These packages require that the heartbeat packages be installed. This is for a different cluster platform which we are not using here, so we will disable it from starting with the system.

<source lang="bash">
chkconfig heartbeat off
</source>

== Install The DRBD Kernel Module ==

The kernel module '''must''' match the [[dom0]] kernel that is running. If you update the kernel and neglect to update the DRBD kernel module, the DRBD array '''will not start'''.

To help simplify things, links to pre-compiled DRBD kernel modules are provided. If the kernel version you have installed doesn't match your kernel, instructions on recompiling the DRBD kernel module from source RPM is provided as well.

=== Install Pre-Compiled DRBD Kernel Module RPMs ===

'''Warning''': The RPM provided here '''''will only work''''' with the kernel-2.6.32.21-168.xendom0_an1.fc13.x86_64.rpm kernel. If you are using Michael Young's dom0 kernel, please skip to [[#Building DRBD Kernel Module RPMs From Source|the next section]].

This RPM provides the DRBD kernel module. Note that these RPMs are compiled against the AN!Cluster variant of myoung's 2.6.32.21_168 dom0 kernel.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-km-2.6.32.23_170.dom0_an1.fc13.x86_64-8.3.7-12.fc13.x86_64.rpm
rpm -ivh drbd-km-2.6.32.23_170.dom0_an1.fc13.x86_64-8.3.7-12.fc13.x86_64.rpm
</source>

If you would like to install the debuginfo

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm
rpm -ivh drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm
</source>

=== Building DRBD Kernel Module RPMs From Source ===

If the above RPMs don't work or if the dom0 kernel you are using in any way differs, please follow the steps here to create a DRBD kernel module matched to your running dom0.

First, install the build environment.

<source lang="bash">
yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
</source>

Install the kernel headers and development library for the dom0 kernel:

'''Note''': The following commands use --force to get past the fact that the headers for the 2.6.33 are already installed, thus making RPM think that these are too old and will conflict. Please proceed with caution.

* If you are using Michael Young's kernel:

<source lang="bash">
cd ~
wget -c http://fedorapeople.org/~myoung/dom0/x86_64/kernel-headers-2.6.32.21-170.xendom0.fc12.x86_64.rpm
wget -c http://fedorapeople.org/~myoung/dom0/x86_64/kernel-devel-2.6.32.21-170.xendom0.fc12.x86_64.rpm
rpm -ivh --force kernel*2.6.32.21-170.xendom0.fc12.x86_64.rpm
</source>

* If you are using the AN!Cluster dom0 kernel:

<source lang="bash">
wget -c https://alteeve.com/files/an-cluster/kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm</source>

Now you need to download, prepare, build and install the source RPM.

<source lang="bash">
rpm -ivh http://fedora.mirror.iweb.ca/releases/13/Everything/source/SRPMS/drbd-8.3.7-2.fc13.src.rpm
cd /root/rpmbuild/SPECS/
rpmbuild -bp drbd.spec
cd /root/rpmbuild/BUILD/drbd-8.3.7/
./configure --enable-spec --with-km
cp /root/rpmbuild/BUILD/drbd-8.3.7/drbd-km.spec /root/rpmbuild/SPECS/
cd /root/rpmbuild/SPECS/
rpmbuild -ba drbd-km.spec
cd /root/rpmbuild/RPMS/x86_64
rpm -Uvh drbd-km-*
</source>

This may be needed if drbd-utils, the user-land DRBD tools, is listed as a requirement when trying to install the drbd-km* RPMs. This step will build all of the DRBD tools RPMs.

<source lang="bash">
yum install bash-completion heartbeat pacemaker
cd /root/rpmbuild/SPECS/
rpmbuild -ba drbd.spec
cd ~/rpmbuild/RPMS/x86_64/
rpm -Uvh drbd-*
chkconfig off heartbeat
</source>

You should be good to go now!

== Allocating Raw Space For DRBD On Each Node ==

If you followed the setup steps provided for in "[[Two Node Fedora 13 Cluster]]", you will have a set amount of unconfigured hard drive space. This is what we will use for the DRBD space on either node. If you've got a different setup, you will need to allocate some raw space before proceeding.

=== Create a Simple Partition ===

If you do not have two drives, please follow the next section's steps, but pay attention to the "'''note'''"s. In short, you will need to create one partition, leave the default type of the partition as 83, write the changes to disk and the proceed to the [[#DRBD Configuration Files|DRBD Configuration Files]] section.

=== Creating a RAID level 1 'md' Device ===

This assumes that you have two raw drives, /dev/sda and /dev/sdb. It further assumes that you've created three partitions which have been assigned to three existing /dev/mdX devices. With these assumptions, we will create /dev/sda4 and /dev/sdb4 and, using them, create a new /dev/md3 device that will host the DRBD partition.

If you have multiple drives and plan to use a different [[TLUG_Talk:_Storage_Technologies_and_Theory#RAID_Levels|RAID levels]], please adjust the follow commands accordingly.

==== Creating The New Partitions ====

'''Warning''': The next steps will have you directly accessing your server's hard drive configuration. Please do not proceed on a live server until you've had a chance to work through these steps on a test server. One mistake can '''''blow away all your data'''''.

Start the fdisk shell for the first hard drive; /dev/sda.

<source lang="bash">
fdisk /dev/sda
</source>
<source lang="text">
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help):
</source>

'''Note''': Depending on your configuration, you may not see the above warning or you may see a different warning. Note it, but it is likely nothing to worry about it.

View the current configuration with the '''p'''rint option

<source lang="bash">
p
</source>
<source lang="text">
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6fe1

Device Boot Start End Blocks Id System
/dev/sda1 1 5100 40960000 fd Linux raid autodetect
/dev/sda2 5100 5622 4194304 fd Linux raid autodetect
/dev/sda3 * 5622 5654 256000 fd Linux raid autodetect

Command (m for help):
</source>

Now we know for sure that the next free partition number is 4. We will now create the '''n'''ew partition.

<source lang="bash">
n
</source>
<source lang="text">
Command action
e extended
p primary partition (1-4)
</source>

We will make it a '''p'''rimary partition

<source lang="bash">
p
</source>
<source lang="text">
Selected partition 4
First cylinder (5654-60801, default 5654):
</source>

Then we simply hit <enter> to select the default starting block.

<source lang="bash">
<enter>
</source>
<source lang="text">
Using default value 5654
Last cylinder, +cylinders or +size{K,M,G} (5654-60801, default 60801):
</source>

Once again we will press <enter> to select the default ending block.

<source lang="bash">
<enter>
</source>
<source lang="text">
Using default value 60801

Command (m for help):
</source>

'''Note''': If you only have one drive and are not creating a [[RAID]] array, you do not to change the type of the partition so you can skip the next few steps. Continue at the step where you write the changes.

Now we need to change the type of partition that it is.

<source lang="bash">
t
</source>
<source lang="text">
Partition number (1-4):
</source>

We know that we are modifying partition number 4.

<source lang="bash">
4
</source>
<source lang="text">
Hex code (type L to list codes):
</source>

Now we need to set the [[hex]] code for the [[Filesystem_List#List_of_Linux_Partition_Types|partition type]] to set. We want to set fd, which defines Linux raid autodetect.

<source lang="bash">
fd
</source>
<source lang="text">
Changed system type of partition 4 to fd (Linux raid autodetect)
</source>

Now check that everything went as expected by once again '''p'''rinting the partition table.

<source lang="bash">
p
</source>
<source lang="text">
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6fe1

Device Boot Start End Blocks Id System
/dev/sda1 1 5100 40960000 fd Linux raid autodetect
/dev/sda2 5100 5622 4194304 fd Linux raid autodetect
/dev/sda3 * 5622 5654 256000 fd Linux raid autodetect
/dev/sda4 5654 60801 442972704+ fd Linux raid autodetect

Command (m for help):
</source>

'''Note''': If you only have one drive, your partitions will be 83 Linux or 82 Linux swap / Solaris, instead of fd Linux raid autodetect.

There it is. So finally, we need to '''w'''rite the changes to the disk.

<source lang="bash">
w
</source>
<source lang="text">
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
</source>

'''Note''': If you only have one drive, reboot now if you got the message above and then skip forward to the "[[#DRBD Configuration Files|DRBD Configuration Files]]" section.

If you see the above message, '''do not''' reboot yet. repeat these steps for the second drive, /dev/sdb, and then reboot.

==== Creating The New /dev/mdX Device ====

''If you only have one drive, skip this step.''

Now we need to use mdadm to create the new [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]] device. This will be used as the device that DRBD will directly access.

<source lang="bash">
mdadm --create /dev/md3 --homehost=localhost.localdomain --raid-devices=2 --level=1 /dev/sda4 /dev/sdb4
</source>
<source lang="text">
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
</source>

Seeing as /boot doesn't exist on this device, we can safely ignore this warning.

<source lang="bash">
y
</source>
<source lang="text">
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/md4 started.
</source>

You can now cat /proc/mdstat to verify that it indeed built. If you're interested, you could open a new terminal window and use watch cat /proc/mdstat and watch the array build.

<source lang="bash">
cat /proc/mdstat
</source>
<source lang="text">
md3 : active raid1 sdb4[1] sda4[0]
442971544 blocks super 1.2 [2/2] [UU]
[>....................] resync = 0.8% (3678976/442971544) finish=111.0min speed=65920K/sec

md2 : active raid1 sda2[0] sdb2[1]
4193272 blocks super 1.1 [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]
40958908 blocks super 1.1 [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk

md0 : active raid1 sda3[0] sdb3[1]
255988 blocks super 1.0 [2/2] [UU]

unused devices: <none>
</source>

Finally, we need to make sure that the new array will start when the system boots. To do this, we'll again use mdadm, but with different options that will have it output data in a format suitable for the /etc/mdadm.conf file. We'll redirect this output to that config file, thus updating it.

<source lang="bash">
mdadm --detail --scan | grep md3 >> /etc/mdadm.conf
cat /etc/mdadm.conf
</source>
<source lang="text">
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=b58df6d0:d925e7bb:c156168d:47c01718
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=ac2cf39c:77cd0314:fedb8407:9b945bb5
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=4e513936:4a966f4e:0dd8402e:6403d10d
ARRAY /dev/md3 metadata=1.2 name=localhost.localdomain:3 UUID=f0b6d0c1:490d47e7:91c7e63a:f8dacc21
</source>

You'll note that the last line, which we just added, is different from the previous lines. This isn't a concern, but you are welcome to re-write it to match the existing format if you wish.

Before you proceed, it is strongly advised that you reboot each node and then verify that the new array did in fact start with the system. You ''do not'' need to wait for the sync to finish before rebooting. It will pick up where you left off once rebooted.

== DRBD Configuration Files ==

DRBD uses a global configuration file, /etc/drbd.d/global_common.conf, and one or more resource files. The resource files need to be created in the /etc/drbd.d/ directory and must have the suffix .res. For this example, we will create a single resource called r0 which we will configure in /etc/drbd.d/r0.res.

=== /etc/drbd.d/global_common.conf ===

The stock /etc/drbd.d/global_common.conf is sane, so we won't bother altering it here.

Full details on all the drbd.conf configuration file directives and arguments can be found [http://www.drbd.org/users-guide/re-drbdconf.html here]. '''Note''': That link doesn't show this new configuration format. Please see [http://www.novell.com/documentation/sle_ha/book_sleha/?page=/documentation/sle_ha/book_sleha/data/sec_ha_drbd_configure.html Novell's] link.

=== /etc/drbd.d/r0.res ===

This is the important part. This defines the resource to use, and must reflect the IP addresses and storage devices that DRBD will use for this resource.

<source lang="bash">
vim /etc/drbd.d/r0.res
</source>
<source lang="bash">
# This is the name of the resource and it's settings. Generally, 'r0' is used
# as the name of the first resource. This is by convention only, though.
resource r0
{
# This tells DRBD where to make the new resource available at on each
# node. This is, again, by convention only.
device /dev/drbd0;

# The main argument here tells DRBD that we will have proper locking
# and fencing, and as such, to allow both nodes to set the resource to
# 'primary' simultaneously.
net
{
allow-two-primaries;
}

# This tells DRBD to automatically set both nodes to 'primary' when the
# nodes start.
startup
{
become-primary-on both;
}

# This tells DRBD to look for and store it's meta-data on the resource
# itself.
meta-disk internal;

# The name below must match the output from `uname -n` on each node.
on an-node01.alteeve.com
{
# This must be the IP address of the interface on the storage
# network (an-node01.sn, in this case).
address 192.168.2.71:7789;

# This is the underlying partition to use for this resource on
# this node.
disk /dev/md3;
}

# Repeat as above, but for the other node.
on an-node02.alteeve.com
{
address 192.168.2.72:7789;
disk /dev/md3;
}
}
</source>

This file must be copied to '''BOTH''' nodes and must match before you proceed.

== Starting The DRBD Resource ==

From the rest of this section, pay attention to whether you see
* '''Node1'''
* '''Node2'''
* '''Both'''

These indicate which node to run the following commands on. There is no functional difference between either node, so just randomly choose one to be '''Node1''' and the other will be '''Node2'''. Once you've chosen which is which, be consistent with which node you run the commands on. Of course, if a command block is proceeded by '''Both''', run the following code block on both nodes.

=== Loading the 'drbd' Module ===

'''Both'''

Normally, we'd load the drbd module by simply starting the /etc/init.d/drbd daemon. However, if we did that at this stage, we'd generate errors because there isn't an UpToDate disk in the array. To get around this, we'll manually load the drbd kernel module using modprobe.

<source lang="bash">
modprobe drbd
</source>

This won't return any output, but if you check, you should now see the special /proc/drbd file.

=== Monitoring Progress ===

'''Both'''

I find it very useful to monitor DRBD while running the rest of the setup. To do this, open a second terminal on each node and use watch to keep an eye on /proc/drbd. This way you will be able to monitor the progress of the array in near-real time.

'''Both'''
<source lang="bash">
watch cat /proc/drbd
</source>

At this stage, it should look like this:

<source lang="text">
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:Unconfigured
</source>

=== Initialize The Resource ===

'''Both'''

This step creates the DRBD meta-data on the new DRBD resource's backing devices. It is only needed when creating new DRBD partitions.

<source lang="bash">
drbdadm create-md r0
</source>
<source lang="text">
--== Thank you for participating in the global usage survey ==--
The server's response is:

you are the 9507th user to install this version
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
</source>

The /proc/drbd output should not have changed at this stage.

=== Starting the Resource ===

'''Both'''

This will attach the backing device, /dev/md3 in our case, and then start the new resource r0.

<source lang="bash">
drbdadm up r0
</source>

There will be no output at the command line. If you are watching /proc/drbd though, you should now see something like this:

<source lang="text">
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:442957988
</source>

That it is Secondary/Secondary and Inconsistent/Inconsistent is expected.

=== Setting the First Primary Node ===

'''Node1'''

As this is a totally new resource, DRBD doesn't know which side of the array is "more valid" than the other. In reality, neither is as there was no existing data of note on either node. This means that we now need to choose a node and tell DRBD to treat it as the "source" node. This step will also tell DRBD to make the "source" node primary. Once set, DRBD will begin sync'ing in the background.

<source lang="bash">
drbdadm -- --overwrite-data-of-peer primary r0
</source>

As before, there will be no output at the command line, but /proc/drbd will change to show the following:

<source lang="text">
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:69024 nr:0 dw:0 dr:69232 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:442888964
[>....................] sync'ed: 0.1% (432508/432576)M
finish: 307:33:42 speed: 320 (320) K/sec
</source>

If you're watching the secondary node, the /proc/drbd will show ro:Secondary/Primary ds:Inconsistent/UpToDate. This is, as you can guess, simply a reflection of it being the "over-written" node.

=== Setting the Second Node to Primary ===

'''Node2'''

The last step to complete the array is to tell the second node to also become primary.

<source lang="bash">
drbdadm primary r0
</source>

As with many drbdadm commands, nothing will be printed to the console. If you're watching the /proc/drbd though, you should see something like Primary/Primary ds:UpToDate/Inconsistent. The Inconsistent flag will remain until the sync is complete.

=== A Note On sync Speed ===

You will notice in the previous step that the sync speed seems awfully slow at 320 (320) K/sec.

'''This is not a problem!'''

As actual data is written to either side of the array, that data will be immediately copied to both nodes. As such, both nodes will always contain up to date copies of the real data. Given this, the syncer is intentionally set low so as to not put too much load on the underlying disks that could cause slow downs. If you still wish to increase the sync speed, you can do so with the following command.

'''Warning''': If you set the DRBD sync speed too high and saturate your disks' maximum write speed, clvmd daemon will likely fail to start in some cases, leading to fences. For this reason, keep your sync rate to about 2/3rds of the underlying disk maximum write speed and hold off on bumping up the sync speed until you know you will have a period of low activity on your cluster.

<source lang="bash">
drbdsetup /dev/drbd0 syncer -r 35M
</source>

The speed-up will not be instant. It will take a little while for the speed to pick up. Once the sync is finished, it is a good idea to revert to the default sync rate.

<source lang="text">
drbdadm syncer r0
</source>

= Setting Up CLVM =

The goal of DRBD in the cluster is to provide clustered [[LVM]], referred to as [[CLVM]] to the nodes. This is done by turning the DRBD partition into an CLVM physical volume.

So now we will create a [[PV]] on top of the new [[DRBD]] partition, /dev/drbd0, that we created in the previous step. Since this new LVM [[PV]] will exist on top of the shared DRBD partition, whatever get written to it's logical volumes will be immediately available on either node, regardless of which node actually initiated the write.

This capability is the underlying reason for creating this cluster; Neither machine is truly needed so if one machine dies, anything on top of the DRBD partition will still be available. When the failed machine returns, the surviving node will have a list of what blocks changed while the other node was gone and can use this list to quickly re-sync the other server.

== Making LVM Cluster-Aware ==

Normally, LVM is run on a single server. This means that at any time, the LVM can write data to the underlying drive and not need to worry if any other device might change anything. In clusters, this isn't the case. The other node could try to write to the shared storage, so then nodes need to enable "locking" to prevent the two nodes from trying to work on the same bit of data at the same time.

The process of enabling this locking is known as making LVM "cluster-aware".

LVM has tool called lvmconf that can be used to enable LVM locking. This is provided as part of the lvm2-cluster package.

<source lang="bash">
yum -y install lvm2-cluster.x86_64
</source>

Now to enable cluster awareness in LVM, run to following command.

<source lang="bash">
lvmconf --enable-cluster
</source>

By default, clvmd, the cluster lvm daemon, is stopped and not set to run on boot. Now that we've enabled LVM locking, we need to start it:

<source lang="bash">
/etc/init.d/clvmd status
</source>
<source lang="text">
clvmd is stopped
active volumes: (none)
</source>

As expected, it is stopped, so lets start it:

<source lang="bash">
/etc/init.d/clvmd start
</source>
<source lang="text">
Activating VGs: No volume groups found
[ OK ]
</source>

'''Note''': I've seen on a few occasions where starting clvmd will time out and, on occasion, fences will be issued. I've not sorted out why, but I have usually been able to resolve this by stopping clvmd and cman, then restarting cman and, finally, restarting clvmd. If I can sort out a way to reliably trigger this problem, I will submit a bug report.

== Filtering Out Devices ==

'''ToDo''': Find a less-aggressive filter.

With the stock /etc/lvm/lvm.conf configuration, all devices on the system will be checked for LVM volumes. This can cause a problem as LVM will give preference to the LVM data on the RAID device over the DRBD device. It sees a duplicate as both are, effectively, one and the same.

To work around this, we need to alter the filter = [] entry. At the time of writing, simply rejecting the underlying /dev/md3 device as a candidate wasn't enough. So for now, we will tell LVM to accept DRBD devices and reject all other devices. To do this, we'll insert "a|/dev/drbd*|" as the first array entry and change the existing entry to "r/.*/".

'''Note''': I would love feedback on a filter argument that successfully ignored just /dev/md3, if anyone can suggest one.

<source lang="bash">
vim /etc/lvm/lvm.conf
</source>
<source lang="text">
# By default we accept every block device:
#filter = [ "a/.*/" ]
filter = [ "a|/dev/drbd*|", "r/.*/" ]
</source>

Now delete the existing cache file so that LVM is forced to rescan the system.

<source lang="bash">
rm -f /etc/lvm/cache/.cache
</source>

The changes take effect immediately.

== Creating a new PV using the DRBD Partition ==

'''Node1'''

We can now proceed with setting up the new DRBD-based LVM physical volume. Once the PV is created, we can create a new volume group and start allocating space to logical volumes.

'''Note''': As we will be using our DRBD device, and as it is a shared block device, most of the following commands only need to be run on one node. Once the block device changes in any way, those changes will near-instantly appear on the other node. For this reason, unless explicitly stated to do so, only run the following commands on one node.

To setup the DRBD partition as an LVM PV, run pvcreate:

<source lang="bash">
pvcreate /dev/drbd0
</source>
<source lang="text">
Physical volume "/dev/drbd0" successfully created
</source>

'''Both'''

Now, on both nodes, check that the new physical volume is visible by using pvdisplay:

<source lang="bash">
pvdisplay
</source>
<source lang="text">
"/dev/drbd0" is a new physical volume of "422.44 GiB"
--- NEW Physical volume ---
PV Name /dev/drbd0
VG Name
PV Size 422.44 GiB
Allocatable NO
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID YHmdip-SuJN-KIEv-2tbK-BT9Q-wfOo-OuQuaW
</source>

If you see PV Name /dev/drbd0 (or your underlying partition) on both nodes, then your DRBD setup and LVM configuration changes are working perfectly!

== Creating a VG on the new PV ==

'''Node1'''

Now we need to create the volume group using the vgcreate command:

<source lang="bash">
vgcreate -c y drbd0_vg0 /dev/drbd0
</source>
<source lang="text">
Clustered volume group "drbd0_vg0" successfully created
</source>

'''Both'''

Now we'll check that the new VG is visible on both nodes using vgdisplay:

<source lang="bash">
vgdisplay
</source>
<source lang="text">
--- Volume group ---
VG Name drbd0_vg0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
Clustered yes
Shared no
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 422.43 GiB
PE Size 4.00 MiB
Total PE 108143
Alloc PE / Size 0 / 0
Free PE / Size 108143 / 422.43 GiB
VG UUID Bb8l9e-es2z-PhaF-Gg3o-2is2-DZ1S-V2RsBF
</source>

If the new VG is visible on both nodes, we are ready to create our first logical volume using the lvcreate tool.

== Creating the First LV on the new VG ==

'''Node1'''

Now we'll create a simple 20 GiB logical volumes. We will use it as a shared GFS2 store for shared files and to store our Xen domU config files later on.

<source lang="bash">
lvcreate -L 20G -n xen_shared drbd0_vg0
</source>
<source lang="text">
Logical volume "xen_shared" created
</source>

'''Both'''

As before, we will check that the new logical volume is visible from both nodes by using the lvdisplay command:

<source lang="bash">
lvdisplay
</source>
<source lang="text">
--- Logical volume ---
LV Name /dev/drbd0_vg0/xen_shared
VG Name drbd0_vg0
LV UUID AqQizc-KBpX-2scN-WFLb-jIeF-QDcM-PlQW84
LV Write Access read/write
LV Status available
# open 0
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
</source>

Again, if this is visible from both nodes, we're set! Repeat this process for all future LVs you will want to create. We will do this a little later to create LVs for Xen VMs.

= Creating A Shared GFS FileSystem =

GFS is a cluster-aware file system that can be simultaneously mounted on two or more nodes at once. We will use it as a place to store ISOs that we'll use to provision our virtual machines.

== Install The GFS2 Utilities ==

Start by installing the [[GFS2]] tools:

<source lang="bash">
yum -y install gfs2-utils.x86_64
</source>

== Format Our CLVM LV With The GFS2 File System ==

'''Node1'''

'''Note''': The following example is designed for the cluster used in the prerequisite HowTo.
* If you have more than 2 nodes, increase the -j 2 to the number of nodes you want to mount this file system on.
* If your cluster is named something other than an-cluster (as set in the cluster.conf file), change -t an-cluster:xen_shared to match you cluster's name. The xen_shared can be whatever you like, but it must be unique in the cluster. I tend to use a name that matches the LV name, but this is my own preference and is not required.

To format the partition run:
<source lang="bash">
mkfs.gfs2 -p lock_dlm -j 2 -t an-cluster:xen_shared /dev/drbd0_vg0/xen_shared
</source>
<source lang="text">
This will destroy any data on /dev/drbd0_vg0/xen_shared.
It appears to contain: symbolic link to `../dm-0'

Are you sure you want to proceed? [y/n]
</source>

Acknowledge the warning, if any, and then press y if you are ready to proceed.

<source lang="bash">
y
</source>
<source lang="text">
Device: /dev/drbd0_vg0/xen_shared
Blocksize: 4096
Device Size 20.00 GB (5242880 blocks)
Filesystem Size: 20.00 GB (5242878 blocks)
Journals: 2
Resource Groups: 80
Locking Protocol: "lock_dlm"
Lock Table: "an-cluster:xen_shared"
UUID: A1487063-2A3F-43B1-3A36-44936B0B4D1E
</source>

Once the format completes, you can mount /dev/drbd0_vg0/xen_shared as you would a normal file system.

'''Both''':

To complete the example, lets mount the GFS2 partition we made just now on /shared and then use df -h to verify.

<source lang="bash">
mkdir /xen_shared
mount /dev/drbd0_vg0/xen_shared /xen_shared
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.8G 34G 8% /
tmpfs 466M 29M 438M 7% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
/dev/dm-0 20G 259M 20G 2% /xen_shared
</source>

You may have noticed that it shows /dev/dm-0 instead of /dev/drbd0_vg0/xen_shared. If you look at the later, you will see that it is simply a [[symlink]] to the former.

<source lang="bash">
ls -lah /dev/drbd0_vg0/xen_shared
</source>
<source lang="text">
lrwxrwxrwx. 1 root root 7 Sep 9 13:24 /dev/drbd0_vg0/xen_shared -> ../dm-0
</source>

== Add An Entry To /etc/fstab ==

The last step is to add an entry for this new partition to each node's /etc/fstab file.

=== Reference The GFS2 Partition By Device Path ===

This is the more traditional method of referencing the GFS2 partition by using it's device path directly.

'''Warning''': An incorrect edit of the /etc/fstab file can leave your system unable to boot! Please review the line generated above to make sure it is accurate and compatible with your setup before proceeding.

<source lang="bash">
vim /etc/fstab
</source>
<source lang="text">
#
# /etc/fstab
# Created by anaconda on Tue Sep 7 06:29:51 2010
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=865690db-85d0-44c5-9f32-ffb6fdf47060 / ext3 defaults 1 1
UUID=8b9822b6-a92e-48c9-96b5-f8943142319e /boot ext3 defaults 1 2
UUID=94b03547-a7e3-45bb-b2d5-837498b370f4 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
xenfs /proc/xen xenfs defaults 0 0
/dev/drbd0_vg0/xen_shared /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

=== Reference The GFS2 Partition By UUID ===

It is sometimes preferable to create an fstab entry that locates the device path via it's [[UUID]]. To do this, you can run the following command which, though a bit cryptic, will print out an /etc/fstab compatible string.

'''Warning''': The same warnings apply here as above

<source lang="bash">
echo `gfs2_edit -p sb /dev/drbd0_vg0/xen_shared | grep sb_uuid | sed -e "s/.*sb_uuid *$.*$/UUID=\L\1\E \/xen_shared\t\tgfs2\trw,suid,dev,exec,nouser,async\t0 0/"`
</source>
<source lang="text">
UUID=a1487063-2a3f-43b1-3a36-44936b0b4d1e /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

You may have noticed that defaults isn't used. Rather, all but the auto option are manually set. This is because the system will drop to single-user mode at boot if it can't mount an auto partition at boot time (auto being implied by defaults). Given that our GFS2 partition sits on top of DRBD and the cluster, there is no way to make it available that early in the boot process.

Further, the gfs2 init script specifically excludes entries in /etc/fstab that have the 'noauto option set. For this reason, we can't simply specify that as we need the init script to see the partition so that it is mounted when GFS2 starts and unmounted when it stops.

Now add this string to /etc/fstab.

<source lang="bash">
vim /etc/fstab
</source>
<source lang="text">
#
# /etc/fstab
# Created by anaconda on Tue Sep 7 06:29:51 2010
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=865690db-85d0-44c5-9f32-ffb6fdf47060 / ext3 defaults 1 1
UUID=8b9822b6-a92e-48c9-96b5-f8943142319e /boot ext3 defaults 1 2
UUID=94b03547-a7e3-45bb-b2d5-837498b370f4 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
xenfs /proc/xen xenfs defaults 0 0
UUID=a1487063-2a3f-43b1-3a36-44936b0b4d1e /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

Please note; At the time of writing this HowTo, there is a bug in findfs and mount. According to [http://www.ietf.org/rfc/rfc4122.txt RFC 4122], programs should accept a [[UUID]] in either upper or lower case. However, this is not currently the case, so you '''must''' pass the UUID in lower-case. Please see bugs [https://bugzilla.redhat.com/show_bug.cgi?id=632373 632373] and [https://bugzilla.redhat.com/show_bug.cgi?id=632385 632385].

== Testing The gfs2 Initialization Script ==

To verify that the new entry is valid, check gfs2's status.

<source lang="bash">
/etc/init.d/gfs2 status
</source>
<source lang="text">
Configured GFS2 mountpoints:
/xen_shared
Active GFS2 mountpoints:
/xen_shared
</source>

Now test stopping and restarting to ensure that the GFS2 partition unmounts and mounts properly.

Stop gfs2.

<source lang="bash">
/etc/init.d/gfs2 stop
</source>
<source lang="text">
Unmounting GFS2 filesystem (/xen_shared): [ OK ]
</source>

Check with df -h to ensure that the mount is gone.

<source lang="bash">
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.0G 35G 6% /
tmpfs 466M 23M 444M 5% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
</source>

Start gfs2 again.

<source lang="bash">
/etc/init.d/gfs2 start
</source>
<source lang="text">
Mounting GFS2 filesystem (/xen_shared): [ OK ]
</source>

Again, check with df -h that it has been remounted.

<source lang="bash">
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.0G 35G 6% /
tmpfs 466M 29M 438M 7% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
/dev/dm-0 20G 259M 20G 2% /xen_shared
</source>

Done!

== Further Reading ==

* [[Grow a GFS2 Partition]]
* [[Hard drive has gone bad in DRBD]]

= Altering Daemon Start Order =

It is important that the various daemons in use by our cluster start in the right order. Most daemons will rely on services provided by another daemon to be running, and will not start or will not operate reliably otherwise.

We need to make sure that xend starts so that the network is stable. Then cman needs to start so that [[fencing]] and [[dlm]] are available. Next, drbd starts so that the clustered storage is available. Then clvmd must start so that the data on the DRBD resource is accessible. Now gfs2 needs to start so that the Xen domU configuration files can be found and finally xendomains must start to boot up the actual domU virtual machines.

To restate as a list, the start order must be:
* xend
* cman
* drbd
* clvmd
* gfs2
* xendomains

To make sure the start order is sane then, we'll edit each of the six daemon's init scripts and alter their Required-Start lines. To make the changes take effect, we will use chkconfig to remove and re-add them to the various start levels.

== Altering xend ==

This should already be done. If it isn't, please see "[[#Make xend play nice with clustering|Making xend play nice with clustering]]" above. If you are revisiting that section, you can skip the cman edit as we will need to make another change in the next step.

== Altering cman ==

This should already be done. If it isn't, please see "[[#Make xend play nice with clustering|Making xend play nice with clustering]]" above. If you are revisiting that section, you can skip the cman edit as we will need to make another change in the next step.

== Altering drbd ==

Now we will tell drbd to start after cman.

This requires the additional step of altering the chkconfig: - 70 08 line to instead read chkconfig: - 20 08. This isn't strictly needed, but will give more room for chkconfig to order the dependent daemons by allowing DRBD to be started as low as position 20, rather than waiting until position 70. This is somewhat more compatible with cman and clvmd which normally start at positions 21 and 24, respectively

<source lang="bash">
vim /etc/init.d/drbd
</source>
<source lang="text">
#!/bin/bash
#
# chkconfig: - 20 08
# description: Loads and unloads the drbd module
#
# Copright 2001-2008 LINBIT Information Technologies
# Philipp Reisner, Lars Ellenberg
#
### BEGIN INIT INFO
# Provides: drbd
# Required-Start: $local_fs $network $syslog cman
# Required-Stop: $local_fs $network $syslog
# Should-Start: sshd multipathd
# Should-Stop: sshd multipathd
# Default-Start:
# Default-Stop:
# Short-Description: Control drbd resources.
### END INIT INFO
</source>

== Altering clvmd ==

Now we will now tell clvmd to start after drbd.

'''Note''': There is currently a minor bug with lvm2-cluster version 2.02.73-2 in that /etc/init.d/clvmd is set by default to mode 0555. This is easily corrected by running the following command. Please check bug [https://bugzilla.redhat.com/show_bug.cgi?id=636066 636066] to see if this has been resolved.

<source lang="bash">
chmod u+w /etc/init.d/clvmd
</source>

Once you've got write access, edit the file.

<source lang="bash">
vim /etc/init.d/clvmd
</source>
<source lang="text">
#!/bin/bash
#
# chkconfig: - 24 76
# description: Starts and stops clvmd
#
# For Red-Hat-based distributions such as Fedora, RHEL, CentOS.
#
### BEGIN INIT INFO
# Provides: clvmd
# Required-Start: $local_fs drbd
# Required-Stop: $local_fs
# Default-Start:
# Default-Stop: 0 1 6
# Short-Description: Clustered LVM Daemon
### END INIT INFO
</source>

== Altering gfs2 ==

Now we will now tell gfs2 to start after clvmd. You will notice that cman is already listed under Required-Start and Required-Stop. It's true that cman must be started, but we've created a chain here so we can safely replace it with clvmd in the start line.

<source lang="bash">
vim /etc/init.d/gfs2
</source>
<source lang="text">
#!/bin/bash
#
# gfs2 mount/unmount helper
#
# chkconfig: - 26 74
# description: mount/unmount gfs2 filesystems configured in /etc/fstab

### BEGIN INIT INFO
# Provides: gfs2
# Required-Start: $network clvmd
# Required-Stop: $network
# Default-Start:
# Default-Stop:
# Short-Description: mount/unmount gfs2 filesystems configured in /etc/fstab
# Description: mount/unmount gfs2 filesystems configured in /etc/fstab
### END INIT INFO
</source>

== Altering xendomains ==

Finally, we will alter xendomains so that it starts last, after gfs2.

<source lang="bash">
vim /etc/init.d/xendomains
</source>
<source lang="text">
#!/bin/bash
#
# /etc/init.d/xendomains
# Start / stop domains automatically when domain 0 boots / shuts down.
#
# chkconfig: 345 99 00
# description: Start / stop Xen domains.
#
# This script offers fairly basic functionality. It should work on Redhat
# but also on LSB-compliant SuSE releases and on Debian with the LSB package
# installed. (LSB is the Linux Standard Base)
#
# Based on the example in the "Designing High Quality Integrated Linux
# Applications HOWTO" by Avi Alkalay
# <http://www.tldp.org/HOWTO/HighQuality-Apps-HOWTO/>
#
### BEGIN INIT INFO
# Provides: xendomains
# Required-Start: $syslog $remote_fs xend gfs2
# Should-Start:
# Required-Stop: $syslog $remote_fs xend
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop secondary xen domains
# Description: Start / stop domains automatically when domain 0
# boots / shuts down.
### END INIT INFO
</source>

== Applying The Changes ==

Change the start order by removing and re-adding all cluster-related daemons using chkconfig.

<source lang="bash">
chkconfig xenstored off; chkconfig xenconsoled off; chkconfig xend off; chkconfig cman off; chkconfig drbd off; chkconfig clvmd off; chkconfig gfs2 off; chkconfig xendomains off
chkconfig xendomains on; chkconfig gfs2 on; chkconfig clvmd on; chkconfig drbd on; chkconfig cman on; chkconfig xend on; chkconfig xenconsoled on; chkconfig xenstored on
</source>

Now verify that the start order is as we want it.

<source lang="bash">
ls -lah /etc/rc3.d/
</source>
<source lang="text">
lrwxrwxrwx. 1 root root 19 Sep 20 13:37 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 21 Sep 20 13:37 S27xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S28xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S29cman -> ../init.d/cman
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S70drbd -> ../init.d/drbd
lrwxrwxrwx. 1 root root 15 Sep 20 13:37 S71clvmd -> ../init.d/clvmd
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S72gfs2 -> ../init.d/gfs2
lrwxrwxrwx. 1 root root 20 Sep 20 13:37 S99xendomains -> ../init.d/xendomains
</source>

= Setting Up Xen =

'''''WARNING''''': Everything below here is pretty '''seriously screwed up'''.

'''Note''': This is not meant to be an extensive tutorial on Xen itself. It covers enough to get domU VMs provisioned in a manner that will take advantage of the cluster. As such, there is minimal explanation of configuration file options. If you need further help, please drop by the ##xen (yes, two ##) [[IRC]] channel on freenode.org.

== Install The Hypervisor Tools ==

These tools are very useful in provisioning and managing domU VMs.

<source lang="bash">
yum -y install virt-install virt-viewer
</source>

== Install The HVM/KVM Tools ==

For hvm (Hardware Virtual Machines), which is required for [http://www.virtuatopia.com/index.php/Virtualizing_Windows_Server_2008_with_Xen paravirtualized Microsoft] VMs, you must install the following packages as well.

<source lang="bash">
yum -y install qemu-kvm.x86_64 qemu-kvm-tools.x86_64
</source>

== Ensure That Virtualization Is Enabled ==

Many motherboards disable hvm by default in their [[BIOS]]. Assuming that you've got a dom0 kernel running at this stage, you can check if this is the case by checking the xm info output.

<source lang="bash">
xm info
</source>
<source lang="text">
host : an-node04.alteeve.com
release : 2.6.32.23-170.dom0_an1.fc13.x86_64
version : #1 SMP Sun Oct 10 20:39:19 EDT 2010
machine : x86_64
nr_cpus : 4
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 2209
hw_caps : 178bf3ff:efd3fbff:00000000:00001310:00802001:00000000:000037ff:00000000
virt_caps : hvm
total_memory : 4063
free_memory : 2987
node_to_cpu : node0:0-3
node_to_memory : node0:2987
node_to_dma32_mem : node0:2928
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline : dom0_mem=1024M
cc_compiler : gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC)
cc_compile_by : root
cc_compile_domain :
cc_compile_date : Mon Oct 11 01:10:38 EDT 2010
xend_config_format : 4
</source>

Look at the virt_caps and xen_caps lines. Notice the hvm entries? This shows that HVM, also known as "secure virtualization", has been enabled. If you do not see this, please check your mainboard manual for information on enabling this on your system.

'''Note''': The next paragraph applies only when running a vanilla kernel.

If you are running a vanilla kernel, you can check to see if your CPU has support for HVM guests but checking /proc/cpuinfo. What you're looking for depends on your CPU manufacturer. If you have an Intel CPU, you need to look for the vmx flag. Likewise, with AMD CPUs, you need to look for the svm flag.

For a more complete, if somewhat dated paper on this topic, please [http://fedoraproject.org/wiki/FedoraXenQuickstartFC6#Fully-virtualized_guests_.28HVM.2FIntel-VT.2FAMD-V.29 Fedora 6 Xen Quickstart Guide, System Requirements].

== Enabling Migration ==

By default, xend will not allow domU VMs from being migrated onto or off of a given dom0 host. Given that we've got a cluster though, we very much want this behaviour, so now we will enable it. This is done by making edits to /etc/xen/xend-config.sxp. Below is a concise list of options that must be set. Some exist already in the file and need to be commented out or altered.

'''Warning''': The values below are '''very''' permissive. Please review each option and improve the security to fit your network before going into production!

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>
<source lang="text">
(xend-http-server yes)
(xend-unix-server yes)
(xend-tcp-xmlrpc-server yes)
(xend-relocation-server yes)
(xend-udev-event-server yes)
(xend-port 8000)
(xend-relocation-port 8002)
(xend-address '')
(xend-relocation-address '')
(xend-relocation-hosts-allow '')
</source>

Once done, restart xend. It is usually safest to stop the cluster before hand to avoid accidental fencing caused by the underlying network being reconfigured.

<source lang="bash">
/etc/init.d/gfs2 stop
/etc/init.d/clvmd stop
/etc/init.d/cman stop
/etc/init.d/xend restart
/etc/init.d/cman start
/etc/init.d/clvmd start
/etc/init.d/gfs2 start
</source>

== Provisioning domU VMs ==

'''Note''': Long and ugly, but it worked. Needs more testing though.

Web-based install:

<source lang="bash">
# Fedora 13 x86_64 RPM builder VM
lvcreate -L 40G -n f13_builder_01 drbd0_vg0
virt-install --connect xen \
--name f13_builder_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--location http://192.168.1.10/f13/x86_64/img/ \
--os-type linux \
--os-variant fedora13 \
--disk path=/dev/drbd0_vg0/f13_builder_01 \
--network bridge=eth0,mac=00:16:3e:00:10:01 \
--vnc \
--paravirt

# A CentOS test server
lvcreate -L 40G -n c5_test_01 drbd0_vg0
virt-install --connect xen \
--name c5_test_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1-3 \
--location http://192.168.1.10/c5/x86_64/img/ \
--os-type linux \
--os-variant rhel5.4 \
--disk path=/dev/drbd0_vg0/c5_test_01 \
--network bridge=eth0,mac=00:16:3e:00:10:02 \
--vnc \
--paravirt

# Red Hat Enterprise Linux 6 beta 2 test server
lvcreate -L 40G -n rh6b2_test_01 drbd0_vg0
virt-install --connect xen \
--name rh6b2_test_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--location http://192.168.1.10/rhel6/x86_64/img/ \
--os-type linux \
--os-variant rhel6 \
--disk path=/dev/drbd0_vg0/rh6b2_test_01 \
--network bridge=eth0,mac=00:16:3e:00:10:03 \
--vnc \
--paravirt
</source>

ISO-based install:

<source lang="bash">
### Still sorting this out - doesn't seem to find 'qemu-dm' (wants to use 'lib64' but it's in 'lib').
# Windows 2008 Business Server
lvcreate -L 50G -n win2008_01 drbd0_vg0
virt-install --connect xen \
--name win2008_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--cdrom /xen_shared/iso/Win_Server_2008_Bis_x86_64.iso \
--os-type windows \
--os-variant win2k8 \
--disk path=/dev/drbd_vg0/win2008_01 \
--network bridge=eth0,mac=00:16:3e:00:10:04 \
--vnc \
--hvm
</source>

== Enabling Migration ==

We need to make some changes to the xend-config.sxp in order to enable migrating domU VMs between the nodes.

So start by editing the /etc/xen/xend-config.sxp file.

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>

Note that this is not meant to be an extensive list, nor a set of changes that maximize security. This will get you up and running, then from there you can explore the other options one at a time to improve security.

=== xen-api-server ===

This option allows for one or more access method definitions to your domU VMs on a given node. By default, it is set to (xen-api-server ((unix))) which only allows access from the local machine. We'll add a new access definition that will tell Xen to listen on localhost and on our back-channel on [[TCP]] port 9001.

<source lang="text">
# Default:
# (xen-api-server ((unix)))
(xen-api-server ((9001 pam '^localhost$ ^10\\.0\\.1') (unix none)))
</source>

== Provision A Virtual Machine ==

Let's walk through the creation of a simple [[CentOS]] 5.x [[domU]]. We'll call it c5_test_01. Personally, I like to name my virtual machines with the format "os_role_seq" (<Operating System ID>_<Role of the VM>_<Sequence Integer>). There are no (known) restrictions on virtual machine names, so feel free to use names that made sense for you.

=== Install virt-manager ===

We will now start using a graphical tool called virt-manager. It simplifies the creation of domU VMs quite a bit.

<source lang="bash">
yum install virt-manager
</source>

Be sure that you've enabled X-Forwarding if you've ssh'ed into your nodes. This may require modifying your /etc/sshd_config file and changing X11Forwarding to yes. Then when you connect to the node, you will need to use the ssh -X user@node switch.

We'll come back to this program shortly.

=== Create The CLVM LV ===

We will create a new [[CLVM]] [[LV]] for this [[VM]]. It will act as the VM's hard drive, so it needs to have enough capacity for the OS and any data you expect the VM to need. As we're creating a hypothetical router/firewall, we will not need much space. We'll use 20 [[GiB]], which should be more than enough.

As before, this will only need to be done on one node. Being that it is on top of the DRBD resource, it will be visible to both nodes.

'''Node1'''

<source lang="bash">
lvcreate -L 20G -n c5_test_01 drbd0_vg0
</source>
<source lang="text">
Logical volume "c5_test_01" created
</source>

'''Both'''

Verify that the new LV is indeed visible from both nodes now.

<source lang="bash">
lvdisplay
</source>
<source lang="text">
--- Logical volume ---
LV Name /dev/drbd0_vg0/xen_shared
VG Name drbd0_vg0
LV UUID yjt41B-cdc4-LGV4-Ds0m-eRVv-FIY7-3FK4Kb
LV Write Access read/write
LV Status available
# open 1
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0

--- Logical volume ---
LV Name /dev/drbd0_vg0/c5_test_01
VG Name drbd0_vg0
LV UUID eOcBZz-nJzz-wHwg-4cBL-ZZHm-1cm6-Sf82xC
LV Write Access read/write
LV Status available
# open 0
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1
</source>

Now we've got the space, we're ready to provision!

=== Connecting To The Hypervisor From virt-manager ===

At this point, we'll be using the graphical virt-manager tool.

==== Running virt-manager Directly On a Node ====

If you are working on the nodes directly, you can run virt-manager but you will need to have Gnome or KDE started first. The easiest way to do this is to switch to run level 5. As an unprivileged user, run the following command.

<source lang="bash">
startx
</source>

From there, click on Application -> System Tools -> Virtual machine Manager

[[Image:virt-manager_local-node_01.png|thumb|500px|center|Where to find virt-manager in Fedora 13]]

==== Running virt-manager Remotely Over X-Forwarding ====

Some features in virt-manager don't work well unless they are run off of the local machine. The best way to accomplish this from a remote system is to use X-Forwarding, which was discussed in the [[#Install virt-manager|Install virt-manager]] above.

Generally speaking, I prefer to provision VMs using virt-manager directly on a cluster node. This avoids some problems, like accessing underlying LVM volumes. So to do this when you can't directly sit at a node, ssh into your node using the -X switch.

<source lang="bash">
ssh -X root@an-node01
</source>
<source lang="text">
Last login: Sun Sep 19 22:02:38 2010
[root@an-node01 ~]#
</source>

Now start virt-manager at the command line and wait for it to open on your local workstation.

<source lang="bash">
virt-manager
</source>

'''Note''': If your local workstation has a different version of X server or Gnome, the virt-manager window could look fairly primitive. It will work just the same though, and all the icons will be in the same place.

==== Running virt-manager Locally On a Workstation ====

The virt-manager tool can be installed on a computer outside of the cluster and then used to connect to the hypervisor on either node. This is very useful for daily usage of your virtual machines, but isn't recommended during the provisioning stage. In some cases, running virt-manager remotely will cause it to not see the LVM volumes properly, preventing you from choosing the LV for your new VM.

To connect to the hypervisor on the cluster nodes, click on File -> Add Connection. Change the Hypervison to Xen and then change the Connection to Remote tunnel over SSH.

[[Image:virt-manager_remote-workstation_01.png|thumb|500px|center|Adding a remote connection in virt-manager from a workstation]]

In some cases, not all of the nodes on the network will be seen. In the screen shot above, you can see that an-node02 can be seen, but an-node01 wasn't. To add the missing one, just type node-name.local (an-node01.local above). Once selected, hit Connect.

Once the two nodes have been connected, you should see them on the virt-manager main page.

[[Image:virt-manager_remote-workstation_02.png|thumb|500px|center|Remote connection in virt-manager to both nodes in the cluster]]

Once you seen your nodes, you can proceed to use virt-manager the same as if you were running it on one of the nodes directly.

=== Using virt-manager To Provision Our VM ===

How you use virt-manager will be the same regardless of how or where you started it. The following steps will walk you through the program's use.

With the Virtual Machine Manager main menu connected to localhost (QEMU), we can now begin the process of provisioning a new Xen domU virtual machine! The top-left icon, which looks like a monitor with the "play" button on it's screen, will be used to provision our virtual machine.

[[Image:virt-manager_local-node_02.png|thumb|500px|center|The virt-manager main menu]]

'''Madi''': You left off here.

Once you press the Create a new virtual machine button you will be asked to name the VM and to choose the install type. Even if you have a [[PXE]] server, choose Network Install.

[[Image:virt-manager_provision_02.png|thumb|500px|center|Provisioning a Xen domU - Naming the VM and choosing the install source]

=== Storing The domU Configuration Files ===

'''ToDo'''

'''Node1'''

As this is our first [[domU]], we'll start by creating our shared configuration directory on the /xen_shared [[GFS2]] partition we just created.

<source lang="bash">
mkdir /xen_shared/config
</source>

To confirm again that the GFS2 partition is working, verify that this directory is visible from both nodes.

<source lang="bash">
ls -lah /xen_shared/
</source>
<source lang="text">
total 20K
drwxr-xr-x. 3 root root 3.8K Sep 13 12:57 .
dr-xr-xr-x. 27 root root 4.0K Sep 13 11:32 ..
drwxr-xr-x. 2 root root 3.8K Sep 13 12:57 config
</source>


<source lang="bash">
</source>
<source lang="text">
</source>

{{footer}}

Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM

2010-10-13T18:44:44Z

SRSullivan: /* Install The Hypervisor */ "-y" on "yum install" should not be part of defualt recommended command.

{{howto_header}}

'''Warning''': This is currently a dumping ground for notes. '''''DO NOT FOLLOW THIS DOCUMENT'S INSTRUCTIONS'''''. Seriously, it could blow up your computer or cause winter to come early.

----

This HowTo will walk you through setting up [[Xen]] [[VM]]s using [[DRBD]] and [[CLVM]] for high availability.

= Prerequisite =

This talk is an extension of the [[Two Node Fedora 13 Cluster]] HowTo. As such, you will be expected to have a freshly built two-node cluster with spare disk space on either node.

Please do not proceed until you have completed the first tutorial.

= Overview =

This tutorial will cover several topics; [[DRBD]], [[CLVM]], [[GFS2]], [[Xen]] [[dom0]] and [[domU]] [[VM]]s and [[rgmanager]]. Their relationship is thus:

* DRBD provides a mechanism to replicate data across both nodes in real time and guarantees a consistent view of that data from either node. Think of it like [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]], but across machines.
* CLVM sits on the DRBD partition and provides the underlying mechanism for allowing both nodes to access shared data in a clustered environment. It will host a shared filesystem by way of GFS2 as well as [[LV]]s that Xen's domU VMs will use as their disk space.
* GFS2 will be the clustered file system used on one of the DBRD-backed, CLVM-managed partitions. Files that need to be shared between nodes, like the Xen VM configuration files, will exist on this partition.
* Xen will be the hypervisor in use that will manage the various virtual machines. Each virtual machine will exist in an LVM LV.
** Xen's dom0 is the special "host" virtual machine. In this case, dom0 will be the OS installed in the first HowTo.
** Xen's domU virtual machines will be the "floating", highly available servers.
* Lastly, rgmanager will be the component of [[cman]] that will be configured to manage the automatic migration of the virtual machines when failures occur and when nodes recover.

= Setting Up Xen =

It may seem odd to start with [[Xen]] at this stage, but it is going to rather fundamentally alter each node's "host" operating system.

At this point, each node's host OS is a traditional operating system operating on the bare metal. When we install a dom0 kernel though, we tell Xen to boot a mini operating system first, and then to boot our "host" operating system. In effect, this converts the host node's operating system into just another virtual machine, albeit with a special view of the underlying hardware and Xen hypervisor.

This conversion is somewhat disruptive, so I like to get it out of the way right away. We will then do the rest of the setup before returning to Xen later on to create the floating virtual machines.

== A Note On The State Of Xen dom0 Support In Fedora ==

As of Fedora 8, support for Xen [[dom0]] has been removed, but support for the hypervisor and [[domU]] virtual machines remains. Red Hat's position is that KVM will be the supported platform going forward. That said, [http://fedoraproject.org/wiki/Features/XenPvopsDom0 this page] seems to indicate that PV Ops dom0 kernels will be supported in the future. Specifically, when dom0 support is merged into the mainline Linux kernel. When this will be is open to speculation, though "by Fedora 16" seems to be a reasonable educated guess.

What this means for us is that we need to use a non-standard dom0 kernel. Specifically, we will use a kernel created by [http://myoung.fedorapeople.org/ myoung] (Micheal Young) for Fedora 12. This kernel does not directly support DRBD, so be aware that we will need to build new DRBD kernel modules for his kernel and then rebuild the DRBD modules each time his kernel is updated.

== A Note On Rolling Your Own RPMs ==

If you want to roll the source RPMs for both the hypervisor and the [[dom0]] kernel, you will need to make both before you can boot into your dom0 for the first time. This is because the dom0 kernel needs the Xen microkernel provided by the xen hypervisor package to boot.

== Install The Hypervisor ==

We will use [[Xen]] 4.0.1, as provided by [http://pasik.reaktio.net/fedora/ Pasik]. We'll use the [http://pasik.reaktio.net/fedora/xen-4.0.1-0.2.fc13.src.rpm source RPM] to build our own RPM.

Regardless of whether you install from source RPMs or the pre-compiled ones, you will need to install the libvirt, qemu, SDL and PyXML packages from the standard repositories before you can proceed.

<source lang="bash">
yum install libvirt PyXML.x86_64 qemu.x86_64 SDL.x86_64
</source>

Please don't remove existing xen utilities and libraries prior to installing the newer version. If you do, core clustering components may be removed. Instead, be sure to use the rpm -Uvh switches to upgrade the existing packages that may be installed already.

=== Installing Prebuilt RPMs ===

These are locally stored copies of the Xen RPMs built for [[x86_64]] on Fedora 13. This requires some dependent packages be installed first.

'''Warning''': These are all recompiled for this website against Fedora 13, x86_64. If you feel more comfortable, please use [[RPM]]s from a source you are familiar with.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/xen-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-doc-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-hypervisor-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-libs-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/xen-runtime-4.0.1-0.2.fc13.x86_64.rpm
rpm -Uvh xen*4.0.1-0.2.fc13.x86_64.rpm
</source>

Optionally, you may wish to install debug and/or devel RPMs as well.

<source lang="bash">
wget -c https://alteeve.com/files/xen-debuginfo-4.0.1-0.2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/xen-devel-4.0.1-0.2.fc13.x86_64.rpm
rpm -Uvh xen-d*4.0.1-0.2.fc13.x86_64.rpm
</source>

=== Building RPMs From Source ===

To build the RPMs from the source, you need to make sure that you have the build environment installed.

'''Note''': If you are following these instruction after having installed a prior dom0 kernel, you will need to comment out exclude=kernel* in /etc/yum.conf so that the kernel-header package can be installed.

<source lang="bash">
vim /etc/yum.conf
</source>
<source lang="text">
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
#exclude=kernel*
</source>

Now install the development environment.

<source lang="bash">
yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
yum install transfig texi2html SDL-devel libX11-devel tetex-latex gtk2-devel libaio-devel dev86 iasl xz-devel e2fsprogs-devel glibc-devel.i686 xmlto asciidoc elfutils-libelf-devel
</source>

'''Note''': If you had to uncomment the exclude=kernel* line earlier, comment it back out now before you forget.

<source lang="bash">
vim /etc/yum.conf
</source>
<source lang="text">
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
exclude=kernel*
</source>

Now we can now build the RPMs from source.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/xen-4.0.1-0.2.fc13.src.rpm
rpm -ivh xen-4.0.1-0.2.fc13.src.rpm
cd /root/rpmbuild/SPECS/
rpmbuild -ba xen.spec
</source>

Once this is done, you will have seven RPMs built. Two of them are not really needed (xen-debuginfo-4.0.1-0.2.fc13.x86_64.rpm and xen-devel-4.0.1-0.2.fc13.x86_64.rpm) and I will leave it up to you may wish to install them or not. If you don't, modify the next command or move the debug and devel RPMs out of the way first.

<source lang="bash">
cd ~/rpmbuild/RPMS/x86_64/
rpm -Uvh xen*4.0.1-0.2.fc13.x86_64.rpm
</source>

== Installing The AN!Cluster dom0 Kernel ==

'''Notice'''!

The kernel provided here was recompiled on Fedora 13 and is a slightly modified version of [[#Installing Micheal Young's dom0 Kernel|Micheal Young's]] kernel available below. I was originally driven to recompile in an effort to solve a DRBD-related kernel oops. For now, unless you have the same DRBD kernel oops, I'd strongly recommend against using the AN!Cluster dom0 kernel until it has been tested much more thoroughly.

With that warning out of the way...

This kernel was compiled on a Fedora 13, x86_64. The DRBD RPMs available a little later where compiled against this dom0 kernel.

'''Note''': The --force is required because the current kernel is newer than 2.6.32 used here. Without this switch, the RPM would not install.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
</source>

If you would like to install the debug, devel and/or header RPMs for this kernel, they are available below.

'''Note''': The kernel-debuginfo-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm RPM is 213 [[MiB]].

<source lang="bash">
wget -c https://alteeve.com/files/an-cluster/kernel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-debuginfo-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-de*-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
</source>

=== Post AN!Cluster dom0 Install Configuration ===

The entry in grub's /boot/grub/menu.lst won't work. You will need to edit it so that it calls the existing installed operating system as a module.

'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your root= is likely different and your rd_MD_UUID= will definitely be different, even on the same machine across installs. Generally speaking, what follows the kernel /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ... entry made by the dom0 kernel can be copied after the module /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ... entry in the example below.

<source lang="bash">
vim /boot/grub/menu.lst
</source>
<source lang="bash">
title Xen 4.0.x, Linux 2.6.32.23-170.dom0_an1.fc13.x86_64
root (hd0,0)
kernel /xen.gz dom0_mem=1024M
module /vmlinuz-2.6.32.23-170.dom0_an1.fc13.x86_64 ...
module /initramfs-2.6.32.23-170.dom0_an1.fc13.x86_64.img

</source>

== Installing Micheal Young's dom0 Kernel ==

This uses a kernel built for Fedora 12, but it works on Fedora 13. This step involves either installing it over HTML or adding and enabling his repository and then installing it from there.

=== Installing Via myoung's Repository ===

This is almost always the preferred method. However, do note that when myoung updates his kernel, there will be a lag where the dom0 dependent RPMs provided here will no longer be compatible.

To add the repository, download the myoung.dom0.repo into the /etc/yum.repos.d/ directory.

<source lang="bash">
cd /etc/yum.repos.d/
wget -c http://myoung.fedorapeople.org/dom0/myoung.dom0.repo
</source>

To enable his repository, edit the repository file and change the two enabled=0 entries to enabled=1.

<source lang="bash">
vim /etc/yum.repos.d/myoung.dom0.repo
</source>
<source lang="text">
[myoung-dom0]
name=myoung's repository of Fedora based dom0 kernels - $basearch
baseurl=http://fedorapeople.org/~myoung/dom0/$basearch/
enabled=1
gpgcheck=0

[myoung-dom0-source]
name=myoung's repository of Fedora based dom0 kernels - Source
baseurl=http://fedorapeople.org/~myoung/dom0/src/
enabled=1
gpgcheck=0
</source>

Install the [[Xen]] [[dom0]] kernel (edit the version number with the updated version if it has changed).

<source lang="bash">
yum install kernel-2.6.32.21-170.xendom0.fc12.x86_64
</source>

=== Post Michael Young's dom0 Install Configuration ===

The entry in grub's /boot/grub/menu.lst won't work. You will need to edit it so that it calls the existing installed operating system as a module.

'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your root= is likely different and your rd_MD_UUID= will definitely be different, even on the same machine across installs. Generally speaking, what follows the kernel /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ... entry made by the dom0 kernel can be copied after the module /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ... entry in the example below.

<source lang="bash">
vim /boot/grub/menu.lst
</source>
<source lang="bash">
title Xen 4.0.x, Linux kernel 2.6.32.21-170.xendom0.fc12.x86_64
root (hd0,0)
kernel /xen.gz dom0_mem=1024M
module /vmlinuz-2.6.32.21-170.xendom0.fc12.x86_64 ...
module /initramfs-2.6.32.21-170.xendom0.fc12.x86_64.img
</source>

=== Disabling Automatic Kernel Updates ===

Seeing as we're using an older kernel, yum will want to replace it whenever there is an updated kernel* package available. Likewise if myoung updates his kernel. In the latter case, the updated kernel from Mr. Young would break compatibility with our DRBD module. So to be safe, we want to tell yum to never update the kernel.

To do this, we need to add exclude=kernel* to the /etc/yum.conf file.

<source lang="bash">
echo "exclude=kernel*" >> /etc/yum.conf
cat /etc/yum.conf
</source>
<source lang="text">
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=3
color=never

# This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
# It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m

# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d

exclude=kernel*
</source>

=== Make xend play nice with clustering ===

By default under Fedora 13, cman will start before xend. This is a problem because xend takes the network down as part of it's setup. This causes [[totem]] communication to fail which leads to fencing.

'''Note''': Move xenconsoled and xenstore to 09 and 10 start positions and then make xend depend on the before starting.

To avoid this, edit the initialization scripts for /etc/init.d/xend and it's dependents xenconsoled and xenstore to have a lower minimum start position. We need to maintain the start order of xenstore first, xenconsoled second and lastly xend. By default, their minimum start positions are 96, 97 and 98 respectively. We will change these to 10, 11 and 12, again, respectively.

Note that we are '''not''' altering the start position of xendomains! This is intentional as this daemon will start the [[domU]] VMs. This can not happen until all other cluster related daemons have started.

To change the start order we will change the line chkconfig: 2345 9x 01 lines to chkconfig: 2345 1x 01, where x is the given daemon's start number. Further, we'll make sure that xenstored begins first by add it to xenconsoled's Required-Start line. We'll then make sure that xenconsoled starts before xend by adding it to xend's Required-Start line.

To recap the changes;
* xenstored will start first.
** We'll change it's start position from 96 to 10.
** We will not add anything to it's Required-Start as it must be the first daemon to come up.
* xenconsoled will start second.
** We'll change it's start position from 97 to 11.
** We will add xenstored to it's Required-Start line.
* xend will start third.
** We'll change it's start position from 98 to 12.
** We will add xenconsoled to it's Required-Start line.

When done, the three initialization scripts should look like the examples below.

<source lang="bash">
vim /etc/init.d/xenstored
</source>
<source lang="bash">
#!/bin/bash
#
# xenstored Script to start and stop the Xen control daemon.
#
# Author: Daniel Berrange <berrange@redhat.com
#
# chkconfig: 2345 10 01
# description: Starts and stops the Xen xenstored daemon.
### BEGIN INIT INFO
# Provides: xenstored
# Required-Start: $syslog $remote_fs
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xenstored
# Description: Starts and stops the Xen xenstored daemon.
### END INIT INFO
</source>

<source lang="bash">
vim /etc/init.d/xenconsoled
</source>
<source lang="bash">
#!/bin/bash
#
# xenconsoled Script to start and stop the Xen xenconsoled daemon
#
# Author: Daniel P. Berrange <berrange@redhat.com>
#
# chkconfig: 2345 11 01
# description: Starts and stops the Xen control daemon.
### BEGIN INIT INFO
# Provides: xenconsoled
# Required-Start: $syslog $remote_fs xenstored
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xenconsoled
# Description: Starts and stops the Xen xenconsoled daemon.
### END INIT INFO
</source>

<source lang="bash">
vim /etc/init.d/xend
</source>
<source lang="bash">
#!/bin/bash
#
# xend Script to start and stop the Xen control daemon.
#
# Author: Keir Fraser <keir.fraser@cl.cam.ac.uk>
#
# chkconfig: 2345 12 98
# description: Starts and stops the Xen control daemon.
### BEGIN INIT INFO
# Provides: xend
# Required-Start: $syslog $remote_fs xenconsoled
# Should-Start:
# Required-Stop: $syslog $remote_fs
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop xend
# Description: Starts and stops the Xen control daemon.
### END INIT INFO
</source>

With xend set to start at a position lower than 98, we now have room for chkconfig to put other daemons after it in the start order, which will be needed a little later. First and foremost, we now need to tell cman to not start until after xend is up.

As above, we will now edit cman's /etc/init.d/cman script. This time though, we will not edit it's chkconfig line. Instead, we will simply add xend to the Required-Start line.

<source lang="bash">
vim /etc/init.d/cman
</source>
<source lang="bash">
#!/bin/bash
#
# cman - Cluster Manager init script
#
# chkconfig: - 21 79
# description: Starts and stops cman
#
#
### BEGIN INIT INFO
# Provides: cman
# Required-Start: $network $time xend
# Required-Stop: $network $time
# Default-Start:
# Default-Stop:
# Short-Description: Starts and stops cman
# Description: Starts and stops the Cluster Manager set of daemons
### END INIT INFO
</source>

Finally, remove and re-add the xend and cman daemons to re-order them in the start list:

<source lang="bash">
chkconfig xenstored off; chkconfig xenconsoled off; chkconfig xend off; chkconfig cman off;
chkconfig xenstored on; chkconfig xenconsoled on; chkconfig xend on; chkconfig cman on
</source>

Confirm that the order has changed so that xend is earlier in the boot sequence than cman. Assuming you've switched to run-level 3, run:

<source lang="bash">
ls -lah /etc/rc3.d/
</source>

Your start sequence should now look like:

<source lang="text">
lrwxrwxrwx. 1 root root 19 Sep 15 19:29 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 21 Sep 15 19:29 S27xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 14 Sep 15 19:29 S28xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Sep 15 19:29 S29cman -> ../init.d/cman
</source>

=== Booting Into The New dom0 ===

If everything went well, you should be able to boot the new dom0 operating system. If you watch the boot process closely, you will see that the boot process is different. You should now see the Xen hypervisor boot prior to handing off to the "host" operating system. This can be confirmed once the dom0 operating system has booted by checking that the file /proc/xen/capabilities exists. What it contains doesn't matter at this stage, only that it exists at all.

== Configure Networking ==

Networking in Xen, particularly in a cluster, can be confusing. If you are not familiar with networking in Xen, please review to following article before proceeding.

A note of a major change from previous layouts. In Xen 3.x, ethX would be copied to a virtual interface called vethX. Then the real ethX would be renamed to pethX and the virtual interface vethX would be renamed to ethX to take it's place. Finally, a bridge called xenbrX would be created and the real pethX and virtual ethX would be connected to it.

This has been changed somewhat it that now, by default, ethX is left alone and a simple bridge called virbrX would be created. We'll be changing this to be somewhat similar to the old style.

Specifically, the real ethX will be renamed to pethX. Then a bridge will be created called ethX, which plays the role of dom0's interface '''and''' bridges connections from VMs through pethX and out into the real world.

This is explained in more detail, and with diagrams, in the article below.

* [[Networking in Xen]]

=== Adding New NICs to Xen ===

By default, xend manages eth0 only. We need to add eth2. Personally, I don't like to put the storage network ethernet devices under Xen's control as this potentially can cause DRBD problems on xend restart. Whether you add it or not I will leave to your preferences.

You can see which, if any, network devices are under Xen's control by running ifconfig and checking to see if there is a virbrX corresponding to a given ethX device.

<source lang="bash">
ifconfig
</source>
<source lang="text">
eth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet addr:192.168.1.74 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:224261 errors:0 dropped:0 overruns:0 frame:0
TX packets:55174 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:319384110 (304.5 MiB) TX bytes:27348739 (26.0 MiB)
Interrupt:225 Base address:0x8000

eth1 Link encap:Ethernet HWaddr 00:1B:21:72:9B:5A
inet addr:192.168.2.74 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:9b5a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:832 (832.0 b) TX bytes:6234 (6.0 KiB)
Memory:feae0000-feb00000

eth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet addr:192.168.3.74 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:818 (818.0 b) TX bytes:6081 (5.9 KiB)
Memory:fe9e0000-fea00000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

virbr0 Link encap:Ethernet HWaddr 02:23:C8:98:31:17
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:4013 (3.9 KiB)
</source>

In the above example, eth0 has a corresponding virbr0 bridge having it's own subnet. In non-clustered systems, this is fine. For our purposes though, it will not do.

=== Removing The qemu virbr0 Bridge ===

By default, [[QEMU]] creates a bridge called virbr0 designed to connect virtual machines to the first eth0 interface. Our system will not need this, so we will remove it. This bridge is configured in the /etc/libvirt/qemu/networks/default.xml file, so to remove this bridge, simply delete the contents of the file.

<source lang="bash">
cat /dev/null >/etc/libvirt/qemu/networks/default.xml
</source>

The next time you reboot, that bridge will be gone.

'''Madi''': Put in the command to delete the bridge before a reboot.

=== Create /etc/xen/scripts/an-network-script ===

This script will be used by Xen to turn the dom0 ethX interfaces into bridges. All traffic to the bridge, be it from dom0 or domU VMs, will be routeable out of the corresponding pethX device. As domU VMs come online, a hotplug script will create virtual interfaces between this new bridge and the domU's interface(s). Think of the vifX.Y devices as being the network cables you'd normally run between a server and a switch.

Before we proceed, please note three things;
# You don't need to use the file name an-network-script. I suggest this name mainly to keep in line with the rest of the 'AN!x' naming used here.
# If you install convirt or other hypervisor tools, they will likely create their own bridge script.
# Adding eth1 is optional, as we know ahead of time that eth1 will not be made available to any virtual machines as it is dedicated to [[DRBD]]. I'm adding it here because I like having things consistent; Do whichever makes more sense to you.

First, touch the file and then chmod it to be executable.

<source lang="bash">
touch /etc/xen/scripts/an-network-script
chmod 755 /etc/xen/scripts/an-network-script
</source>

Now edit it to contain the following:
<source lang="bash">
vim /etc/xen/scripts/an-network-script
</source>
<source lang="text">
#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=eth0
"$dir/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=eth2
</source>

Now tell Xen to execute that script by editing /etc/xen/xend-config.sxp file and changing the network-script argument to point to this new script (this is line 158 in the default xend-config.sxp script):

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>
<source lang="text">
#(network-script network-bridge)
#(network-script /bin/true)
(network-script an-network-script)
</source>

'''Warning''': The next step may trigger fencing of the nodes! As such, be sure that you're not running anything critical. If unsure, please stop cman or reboot the nodes.

<source lang="bash">
/etc/init.d/cman stop
</source>

Now restart xend.

<source lang="bash">
/etc/init.d/xend restart
</source>

If everything worked, you should now be able to run ifconfig and see that all the ethX devices have matching pethX, virtual and bridge devices.

<source lang="bash">
ifconfig
</source>
<source lang="text">
eth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet addr:192.168.1.74 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:78 errors:0 dropped:0 overruns:0 frame:0
TX packets:57 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9796 (9.5 KiB) TX bytes:12574 (12.2 KiB)

eth1 Link encap:Ethernet HWaddr 00:1B:21:72:9B:5A
inet addr:192.168.2.74 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:9b5a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:832 (832.0 b) TX bytes:6234 (6.0 KiB)
Memory:feae0000-feb00000

eth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet addr:192.168.3.74 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:32 errors:0 dropped:0 overruns:0 frame:0
TX packets:29 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5471 (5.3 KiB) TX bytes:5867 (5.7 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

peth0 Link encap:Ethernet HWaddr 48:5B:39:3C:53:14
inet6 addr: fe80::4a5b:39ff:fe3c:5314/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:224486 errors:0 dropped:0 overruns:0 frame:0
TX packets:55349 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:319406626 (304.6 MiB) TX bytes:27384681 (26.1 MiB)
Interrupt:225 Base address:0x8000

peth2 Link encap:Ethernet HWaddr 00:1B:21:72:96:EA
inet6 addr: fe80::21b:21ff:fe72:96ea/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:35 errors:0 dropped:0 overruns:0 frame:0
TX packets:70 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6827 (6.6 KiB) TX bytes:12470 (12.1 KiB)
Memory:fe9e0000-fea00000
</source>

'''Note''': The virbr0 may remain until you reboot your nodes.

If you see something like this, then you are ready to proceed! Now start your cluster back up.

<source lang="bash">
/etc/init.d/cman start
</source>

We're done for now. There is more to do in Xen, but this was all we needed to do in order to proceed with the next several steps. Onces we have the clustered storage online, we'll come back to Xen for the domU setup.

= Building the DRBD Array =

Building the DRBD array requires a few steps. First, raw space on either node must be prepared. Next, DRBD must be told that it is to create a resource using this newly configured raw space. Finally, the new array must be initialized.

== A Map of the Cluster's Storage ==

The layout of the storage in the cluster can quickly become difficult to follow. Below is an [[ASCII]] drawing which should help you see how DRBD will tie in to the rest of the cluster's storage. This map assumes a simple [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]] array underlying each node. If your node has a single hard drive, simply collapse the first two layers into one. Similarly, if your underlying storage is a more complex RAID array, simply expand the number of physical devices at the top level.

<source lang="text">
Node1 Node2
_____ _____ _____ _____
| sda | | sdb | | sda | | sdb |
|_____| |_____| |_____| |_____|
|_______| |_______|
_______ ____|___ _______ _______ ____|___ _______
__|__ __|__ __|__ __|__ __|__ __|__ __|__ __|__
| md0 | | md1 | | md2 | | md3 | | md3 | | md2 | | md1 | | md0 |
|_____| |_____| |_____| |_____| |_____| |_____| |_____| |_____|
| | | | | | | |
___|___ _|_ ____|____ |___________| ____|____ _|_ ___|___
| /boot | | / | | <swap> | | | <swap> | | / | | /boot |
|_______| |___| |_________| ______|______ |_________| |___| |_______|
| /dev/drbd0 |
|_____________|
|
____|______
| clvm PV |
|___________|
|
_____|_____
| drbd0_vg0 |
|___________|
|
_____|_____ ___...____
| | |
___|___ ___|___ ___|___
| lv_X | | lv_Y | | lv_N |
|_______| |_______| |_______|
</source>

== Install The DRBD Tools ==

DRBD has two components; The actual application and tools and the kernel module.

There are two options for installing the DRBD user-land tools at this point; AN!Cluster-built RPMs or using the ones shipped with Fedora. Regardless of which method you choose, you will need to either install the AN!Cluster DRBD kernel module RPMs or else rebuild the source RPMs referenced.

=== Install The AN!Cluster DRBD User-Land Tools ===

I am currently experimenting with ways to solve a DRBD triggered kernel oops in the Xen pvops 2.6.32 kernel. For this reason, I've recompiled the following user-land RPMs under the AN!Cluster variant dom0 kernel RPMs referenced earlier in this paper. If you used the AN! RPMs, then I suggest giving these RPMs a try. However, if you are using myoung's dom0, I recommend sticking to the Fedora-provided user-land DRBD tools.

<source lang="bash">
yum -y install bash-completion heartbeat pacemaker
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-utils-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-xen-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-bash-completion-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-udev-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-heartbeat-8.3.7-2.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/drbd-pacemaker-8.3.7-2.fc13.x86_64.rpm
rpm -ivh drbd*8.3.7-2.fc13.x86_64.rpm
</source>

=== Install The Stock Fedora DRBD User-Land Tools ===

You will need to install the following tools:

<source lang="bash">
yum install drbd.x86_64 drbd-xen.x86_64 drbd-utils.x86_64
</source>

=== Disable heartbeat ===

These packages require that the heartbeat packages be installed. This is for a different cluster platform which we are not using here, so we will disable it from starting with the system.

<source lang="bash">
chkconfig heartbeat off
</source>

== Install The DRBD Kernel Module ==

The kernel module '''must''' match the [[dom0]] kernel that is running. If you update the kernel and neglect to update the DRBD kernel module, the DRBD array '''will not start'''.

To help simplify things, links to pre-compiled DRBD kernel modules are provided. If the kernel version you have installed doesn't match your kernel, instructions on recompiling the DRBD kernel module from source RPM is provided as well.

=== Install Pre-Compiled DRBD Kernel Module RPMs ===

'''Warning''': The RPM provided here '''''will only work''''' with the kernel-2.6.32.21-168.xendom0_an1.fc13.x86_64.rpm kernel. If you are using Michael Young's dom0 kernel, please skip to [[#Building DRBD Kernel Module RPMs From Source|the next section]].

This RPM provides the DRBD kernel module. Note that these RPMs are compiled against the AN!Cluster variant of myoung's 2.6.32.21_168 dom0 kernel.

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-km-2.6.32.23_170.dom0_an1.fc13.x86_64-8.3.7-12.fc13.x86_64.rpm
rpm -ivh drbd-km-2.6.32.23_170.dom0_an1.fc13.x86_64-8.3.7-12.fc13.x86_64.rpm
</source>

If you would like to install the debuginfo

<source lang="bash">
cd ~
wget -c https://alteeve.com/files/an-cluster/drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm
rpm -ivh drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm
</source>

=== Building DRBD Kernel Module RPMs From Source ===

If the above RPMs don't work or if the dom0 kernel you are using in any way differs, please follow the steps here to create a DRBD kernel module matched to your running dom0.

First, install the build environment.

<source lang="bash">
yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"
</source>

Install the kernel headers and development library for the dom0 kernel:

'''Note''': The following commands use --force to get past the fact that the headers for the 2.6.33 are already installed, thus making RPM think that these are too old and will conflict. Please proceed with caution.

* If you are using Michael Young's kernel:

<source lang="bash">
cd ~
wget -c http://fedorapeople.org/~myoung/dom0/x86_64/kernel-headers-2.6.32.21-170.xendom0.fc12.x86_64.rpm
wget -c http://fedorapeople.org/~myoung/dom0/x86_64/kernel-devel-2.6.32.21-170.xendom0.fc12.x86_64.rpm
rpm -ivh --force kernel*2.6.32.21-170.xendom0.fc12.x86_64.rpm
</source>

* If you are using the AN!Cluster dom0 kernel:

<source lang="bash">
wget -c https://alteeve.com/files/an-cluster/kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
wget -c https://alteeve.com/files/an-cluster/kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm
rpm -ivh --force kernel-devel-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm kernel-headers-2.6.32.23-170.dom0_an1.fc13.x86_64.rpm</source>

Now you need to download, prepare, build and install the source RPM.

<source lang="bash">
rpm -ivh http://fedora.mirror.iweb.ca/releases/13/Everything/source/SRPMS/drbd-8.3.7-2.fc13.src.rpm
cd /root/rpmbuild/SPECS/
rpmbuild -bp drbd.spec
cd /root/rpmbuild/BUILD/drbd-8.3.7/
./configure --enable-spec --with-km
cp /root/rpmbuild/BUILD/drbd-8.3.7/drbd-km.spec /root/rpmbuild/SPECS/
cd /root/rpmbuild/SPECS/
rpmbuild -ba drbd-km.spec
cd /root/rpmbuild/RPMS/x86_64
rpm -Uvh drbd-km-*
</source>

This may be needed if drbd-utils, the user-land DRBD tools, is listed as a requirement when trying to install the drbd-km* RPMs. This step will build all of the DRBD tools RPMs.

<source lang="bash">
yum install bash-completion heartbeat pacemaker
cd /root/rpmbuild/SPECS/
rpmbuild -ba drbd.spec
cd ~/rpmbuild/RPMS/x86_64/
rpm -Uvh drbd-*
chkconfig off heartbeat
</source>

You should be good to go now!

== Allocating Raw Space For DRBD On Each Node ==

If you followed the setup steps provided for in "[[Two Node Fedora 13 Cluster]]", you will have a set amount of unconfigured hard drive space. This is what we will use for the DRBD space on either node. If you've got a different setup, you will need to allocate some raw space before proceeding.

=== Create a Simple Partition ===

If you do not have two drives, please follow the next section's steps, but pay attention to the "'''note'''"s. In short, you will need to create one partition, leave the default type of the partition as 83, write the changes to disk and the proceed to the [[#DRBD Configuration Files|DRBD Configuration Files]] section.

=== Creating a RAID level 1 'md' Device ===

This assumes that you have two raw drives, /dev/sda and /dev/sdb. It further assumes that you've created three partitions which have been assigned to three existing /dev/mdX devices. With these assumptions, we will create /dev/sda4 and /dev/sdb4 and, using them, create a new /dev/md3 device that will host the DRBD partition.

If you have multiple drives and plan to use a different [[TLUG_Talk:_Storage_Technologies_and_Theory#RAID_Levels|RAID levels]], please adjust the follow commands accordingly.

==== Creating The New Partitions ====

'''Warning''': The next steps will have you directly accessing your server's hard drive configuration. Please do not proceed on a live server until you've had a chance to work through these steps on a test server. One mistake can '''''blow away all your data'''''.

Start the fdisk shell for the first hard drive; /dev/sda.

<source lang="bash">
fdisk /dev/sda
</source>
<source lang="text">
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help):
</source>

'''Note''': Depending on your configuration, you may not see the above warning or you may see a different warning. Note it, but it is likely nothing to worry about it.

View the current configuration with the '''p'''rint option

<source lang="bash">
p
</source>
<source lang="text">
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6fe1

Device Boot Start End Blocks Id System
/dev/sda1 1 5100 40960000 fd Linux raid autodetect
/dev/sda2 5100 5622 4194304 fd Linux raid autodetect
/dev/sda3 * 5622 5654 256000 fd Linux raid autodetect

Command (m for help):
</source>

Now we know for sure that the next free partition number is 4. We will now create the '''n'''ew partition.

<source lang="bash">
n
</source>
<source lang="text">
Command action
e extended
p primary partition (1-4)
</source>

We will make it a '''p'''rimary partition

<source lang="bash">
p
</source>
<source lang="text">
Selected partition 4
First cylinder (5654-60801, default 5654):
</source>

Then we simply hit <enter> to select the default starting block.

<source lang="bash">
<enter>
</source>
<source lang="text">
Using default value 5654
Last cylinder, +cylinders or +size{K,M,G} (5654-60801, default 60801):
</source>

Once again we will press <enter> to select the default ending block.

<source lang="bash">
<enter>
</source>
<source lang="text">
Using default value 60801

Command (m for help):
</source>

'''Note''': If you only have one drive and are not creating a [[RAID]] array, you do not to change the type of the partition so you can skip the next few steps. Continue at the step where you write the changes.

Now we need to change the type of partition that it is.

<source lang="bash">
t
</source>
<source lang="text">
Partition number (1-4):
</source>

We know that we are modifying partition number 4.

<source lang="bash">
4
</source>
<source lang="text">
Hex code (type L to list codes):
</source>

Now we need to set the [[hex]] code for the [[Filesystem_List#List_of_Linux_Partition_Types|partition type]] to set. We want to set fd, which defines Linux raid autodetect.

<source lang="bash">
fd
</source>
<source lang="text">
Changed system type of partition 4 to fd (Linux raid autodetect)
</source>

Now check that everything went as expected by once again '''p'''rinting the partition table.

<source lang="bash">
p
</source>
<source lang="text">
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6fe1

Device Boot Start End Blocks Id System
/dev/sda1 1 5100 40960000 fd Linux raid autodetect
/dev/sda2 5100 5622 4194304 fd Linux raid autodetect
/dev/sda3 * 5622 5654 256000 fd Linux raid autodetect
/dev/sda4 5654 60801 442972704+ fd Linux raid autodetect

Command (m for help):
</source>

'''Note''': If you only have one drive, your partitions will be 83 Linux or 82 Linux swap / Solaris, instead of fd Linux raid autodetect.

There it is. So finally, we need to '''w'''rite the changes to the disk.

<source lang="bash">
w
</source>
<source lang="text">
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
</source>

'''Note''': If you only have one drive, reboot now if you got the message above and then skip forward to the "[[#DRBD Configuration Files|DRBD Configuration Files]]" section.

If you see the above message, '''do not''' reboot yet. repeat these steps for the second drive, /dev/sdb, and then reboot.

==== Creating The New /dev/mdX Device ====

''If you only have one drive, skip this step.''

Now we need to use mdadm to create the new [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]] device. This will be used as the device that DRBD will directly access.

<source lang="bash">
mdadm --create /dev/md3 --homehost=localhost.localdomain --raid-devices=2 --level=1 /dev/sda4 /dev/sdb4
</source>
<source lang="text">
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
</source>

Seeing as /boot doesn't exist on this device, we can safely ignore this warning.

<source lang="bash">
y
</source>
<source lang="text">
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/md4 started.
</source>

You can now cat /proc/mdstat to verify that it indeed built. If you're interested, you could open a new terminal window and use watch cat /proc/mdstat and watch the array build.

<source lang="bash">
cat /proc/mdstat
</source>
<source lang="text">
md3 : active raid1 sdb4[1] sda4[0]
442971544 blocks super 1.2 [2/2] [UU]
[>....................] resync = 0.8% (3678976/442971544) finish=111.0min speed=65920K/sec

md2 : active raid1 sda2[0] sdb2[1]
4193272 blocks super 1.1 [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]
40958908 blocks super 1.1 [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk

md0 : active raid1 sda3[0] sdb3[1]
255988 blocks super 1.0 [2/2] [UU]

unused devices: <none>
</source>

Finally, we need to make sure that the new array will start when the system boots. To do this, we'll again use mdadm, but with different options that will have it output data in a format suitable for the /etc/mdadm.conf file. We'll redirect this output to that config file, thus updating it.

<source lang="bash">
mdadm --detail --scan | grep md3 >> /etc/mdadm.conf
cat /etc/mdadm.conf
</source>
<source lang="text">
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=b58df6d0:d925e7bb:c156168d:47c01718
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=ac2cf39c:77cd0314:fedb8407:9b945bb5
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=4e513936:4a966f4e:0dd8402e:6403d10d
ARRAY /dev/md3 metadata=1.2 name=localhost.localdomain:3 UUID=f0b6d0c1:490d47e7:91c7e63a:f8dacc21
</source>

You'll note that the last line, which we just added, is different from the previous lines. This isn't a concern, but you are welcome to re-write it to match the existing format if you wish.

Before you proceed, it is strongly advised that you reboot each node and then verify that the new array did in fact start with the system. You ''do not'' need to wait for the sync to finish before rebooting. It will pick up where you left off once rebooted.

== DRBD Configuration Files ==

DRBD uses a global configuration file, /etc/drbd.d/global_common.conf, and one or more resource files. The resource files need to be created in the /etc/drbd.d/ directory and must have the suffix .res. For this example, we will create a single resource called r0 which we will configure in /etc/drbd.d/r0.res.

=== /etc/drbd.d/global_common.conf ===

The stock /etc/drbd.d/global_common.conf is sane, so we won't bother altering it here.

Full details on all the drbd.conf configuration file directives and arguments can be found [http://www.drbd.org/users-guide/re-drbdconf.html here]. '''Note''': That link doesn't show this new configuration format. Please see [http://www.novell.com/documentation/sle_ha/book_sleha/?page=/documentation/sle_ha/book_sleha/data/sec_ha_drbd_configure.html Novell's] link.

=== /etc/drbd.d/r0.res ===

This is the important part. This defines the resource to use, and must reflect the IP addresses and storage devices that DRBD will use for this resource.

<source lang="bash">
vim /etc/drbd.d/r0.res
</source>
<source lang="bash">
# This is the name of the resource and it's settings. Generally, 'r0' is used
# as the name of the first resource. This is by convention only, though.
resource r0
{
# This tells DRBD where to make the new resource available at on each
# node. This is, again, by convention only.
device /dev/drbd0;

# The main argument here tells DRBD that we will have proper locking
# and fencing, and as such, to allow both nodes to set the resource to
# 'primary' simultaneously.
net
{
allow-two-primaries;
}

# This tells DRBD to automatically set both nodes to 'primary' when the
# nodes start.
startup
{
become-primary-on both;
}

# This tells DRBD to look for and store it's meta-data on the resource
# itself.
meta-disk internal;

# The name below must match the output from `uname -n` on each node.
on an-node01.alteeve.com
{
# This must be the IP address of the interface on the storage
# network (an-node01.sn, in this case).
address 192.168.2.71:7789;

# This is the underlying partition to use for this resource on
# this node.
disk /dev/md3;
}

# Repeat as above, but for the other node.
on an-node02.alteeve.com
{
address 192.168.2.72:7789;
disk /dev/md3;
}
}
</source>

This file must be copied to '''BOTH''' nodes and must match before you proceed.

== Starting The DRBD Resource ==

From the rest of this section, pay attention to whether you see
* '''Node1'''
* '''Node2'''
* '''Both'''

These indicate which node to run the following commands on. There is no functional difference between either node, so just randomly choose one to be '''Node1''' and the other will be '''Node2'''. Once you've chosen which is which, be consistent with which node you run the commands on. Of course, if a command block is proceeded by '''Both''', run the following code block on both nodes.

=== Loading the 'drbd' Module ===

'''Both'''

Normally, we'd load the drbd module by simply starting the /etc/init.d/drbd daemon. However, if we did that at this stage, we'd generate errors because there isn't an UpToDate disk in the array. To get around this, we'll manually load the drbd kernel module using modprobe.

<source lang="bash">
modprobe drbd
</source>

This won't return any output, but if you check, you should now see the special /proc/drbd file.

=== Monitoring Progress ===

'''Both'''

I find it very useful to monitor DRBD while running the rest of the setup. To do this, open a second terminal on each node and use watch to keep an eye on /proc/drbd. This way you will be able to monitor the progress of the array in near-real time.

'''Both'''
<source lang="bash">
watch cat /proc/drbd
</source>

At this stage, it should look like this:

<source lang="text">
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:Unconfigured
</source>

=== Initialize The Resource ===

'''Both'''

This step creates the DRBD meta-data on the new DRBD resource's backing devices. It is only needed when creating new DRBD partitions.

<source lang="bash">
drbdadm create-md r0
</source>
<source lang="text">
--== Thank you for participating in the global usage survey ==--
The server's response is:

you are the 9507th user to install this version
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
</source>

The /proc/drbd output should not have changed at this stage.

=== Starting the Resource ===

'''Both'''

This will attach the backing device, /dev/md3 in our case, and then start the new resource r0.

<source lang="bash">
drbdadm up r0
</source>

There will be no output at the command line. If you are watching /proc/drbd though, you should now see something like this:

<source lang="text">
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:442957988
</source>

That it is Secondary/Secondary and Inconsistent/Inconsistent is expected.

=== Setting the First Primary Node ===

'''Node1'''

As this is a totally new resource, DRBD doesn't know which side of the array is "more valid" than the other. In reality, neither is as there was no existing data of note on either node. This means that we now need to choose a node and tell DRBD to treat it as the "source" node. This step will also tell DRBD to make the "source" node primary. Once set, DRBD will begin sync'ing in the background.

<source lang="bash">
drbdadm -- --overwrite-data-of-peer primary r0
</source>

As before, there will be no output at the command line, but /proc/drbd will change to show the following:

<source lang="text">
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@xenmaster002.iplink.net, 2010-09-07 16:02:46
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:69024 nr:0 dw:0 dr:69232 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:442888964
[>....................] sync'ed: 0.1% (432508/432576)M
finish: 307:33:42 speed: 320 (320) K/sec
</source>

If you're watching the secondary node, the /proc/drbd will show ro:Secondary/Primary ds:Inconsistent/UpToDate. This is, as you can guess, simply a reflection of it being the "over-written" node.

=== Setting the Second Node to Primary ===

'''Node2'''

The last step to complete the array is to tell the second node to also become primary.

<source lang="bash">
drbdadm primary r0
</source>

As with many drbdadm commands, nothing will be printed to the console. If you're watching the /proc/drbd though, you should see something like Primary/Primary ds:UpToDate/Inconsistent. The Inconsistent flag will remain until the sync is complete.

=== A Note On sync Speed ===

You will notice in the previous step that the sync speed seems awfully slow at 320 (320) K/sec.

'''This is not a problem!'''

As actual data is written to either side of the array, that data will be immediately copied to both nodes. As such, both nodes will always contain up to date copies of the real data. Given this, the syncer is intentionally set low so as to not put too much load on the underlying disks that could cause slow downs. If you still wish to increase the sync speed, you can do so with the following command.

'''Warning''': If you set the DRBD sync speed too high and saturate your disks' maximum write speed, clvmd daemon will likely fail to start in some cases, leading to fences. For this reason, keep your sync rate to about 2/3rds of the underlying disk maximum write speed and hold off on bumping up the sync speed until you know you will have a period of low activity on your cluster.

<source lang="bash">
drbdsetup /dev/drbd0 syncer -r 35M
</source>

The speed-up will not be instant. It will take a little while for the speed to pick up. Once the sync is finished, it is a good idea to revert to the default sync rate.

<source lang="text">
drbdadm syncer r0
</source>

= Setting Up CLVM =

The goal of DRBD in the cluster is to provide clustered [[LVM]], referred to as [[CLVM]] to the nodes. This is done by turning the DRBD partition into an CLVM physical volume.

So now we will create a [[PV]] on top of the new [[DRBD]] partition, /dev/drbd0, that we created in the previous step. Since this new LVM [[PV]] will exist on top of the shared DRBD partition, whatever get written to it's logical volumes will be immediately available on either node, regardless of which node actually initiated the write.

This capability is the underlying reason for creating this cluster; Neither machine is truly needed so if one machine dies, anything on top of the DRBD partition will still be available. When the failed machine returns, the surviving node will have a list of what blocks changed while the other node was gone and can use this list to quickly re-sync the other server.

== Making LVM Cluster-Aware ==

Normally, LVM is run on a single server. This means that at any time, the LVM can write data to the underlying drive and not need to worry if any other device might change anything. In clusters, this isn't the case. The other node could try to write to the shared storage, so then nodes need to enable "locking" to prevent the two nodes from trying to work on the same bit of data at the same time.

The process of enabling this locking is known as making LVM "cluster-aware".

LVM has tool called lvmconf that can be used to enable LVM locking. This is provided as part of the lvm2-cluster package.

<source lang="bash">
yum -y install lvm2-cluster.x86_64
</source>

Now to enable cluster awareness in LVM, run to following command.

<source lang="bash">
lvmconf --enable-cluster
</source>

By default, clvmd, the cluster lvm daemon, is stopped and not set to run on boot. Now that we've enabled LVM locking, we need to start it:

<source lang="bash">
/etc/init.d/clvmd status
</source>
<source lang="text">
clvmd is stopped
active volumes: (none)
</source>

As expected, it is stopped, so lets start it:

<source lang="bash">
/etc/init.d/clvmd start
</source>
<source lang="text">
Activating VGs: No volume groups found
[ OK ]
</source>

'''Note''': I've seen on a few occasions where starting clvmd will time out and, on occasion, fences will be issued. I've not sorted out why, but I have usually been able to resolve this by stopping clvmd and cman, then restarting cman and, finally, restarting clvmd. If I can sort out a way to reliably trigger this problem, I will submit a bug report.

== Filtering Out Devices ==

'''ToDo''': Find a less-aggressive filter.

With the stock /etc/lvm/lvm.conf configuration, all devices on the system will be checked for LVM volumes. This can cause a problem as LVM will give preference to the LVM data on the RAID device over the DRBD device. It sees a duplicate as both are, effectively, one and the same.

To work around this, we need to alter the filter = [] entry. At the time of writing, simply rejecting the underlying /dev/md3 device as a candidate wasn't enough. So for now, we will tell LVM to accept DRBD devices and reject all other devices. To do this, we'll insert "a|/dev/drbd*|" as the first array entry and change the existing entry to "r/.*/".

'''Note''': I would love feedback on a filter argument that successfully ignored just /dev/md3, if anyone can suggest one.

<source lang="bash">
vim /etc/lvm/lvm.conf
</source>
<source lang="text">
# By default we accept every block device:
#filter = [ "a/.*/" ]
filter = [ "a|/dev/drbd*|", "r/.*/" ]
</source>

Now delete the existing cache file so that LVM is forced to rescan the system.

<source lang="bash">
rm -f /etc/lvm/cache/.cache
</source>

The changes take effect immediately.

== Creating a new PV using the DRBD Partition ==

'''Node1'''

We can now proceed with setting up the new DRBD-based LVM physical volume. Once the PV is created, we can create a new volume group and start allocating space to logical volumes.

'''Note''': As we will be using our DRBD device, and as it is a shared block device, most of the following commands only need to be run on one node. Once the block device changes in any way, those changes will near-instantly appear on the other node. For this reason, unless explicitly stated to do so, only run the following commands on one node.

To setup the DRBD partition as an LVM PV, run pvcreate:

<source lang="bash">
pvcreate /dev/drbd0
</source>
<source lang="text">
Physical volume "/dev/drbd0" successfully created
</source>

'''Both'''

Now, on both nodes, check that the new physical volume is visible by using pvdisplay:

<source lang="bash">
pvdisplay
</source>
<source lang="text">
"/dev/drbd0" is a new physical volume of "422.44 GiB"
--- NEW Physical volume ---
PV Name /dev/drbd0
VG Name
PV Size 422.44 GiB
Allocatable NO
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID YHmdip-SuJN-KIEv-2tbK-BT9Q-wfOo-OuQuaW
</source>

If you see PV Name /dev/drbd0 (or your underlying partition) on both nodes, then your DRBD setup and LVM configuration changes are working perfectly!

== Creating a VG on the new PV ==

'''Node1'''

Now we need to create the volume group using the vgcreate command:

<source lang="bash">
vgcreate -c y drbd0_vg0 /dev/drbd0
</source>
<source lang="text">
Clustered volume group "drbd0_vg0" successfully created
</source>

'''Both'''

Now we'll check that the new VG is visible on both nodes using vgdisplay:

<source lang="bash">
vgdisplay
</source>
<source lang="text">
--- Volume group ---
VG Name drbd0_vg0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
Clustered yes
Shared no
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 422.43 GiB
PE Size 4.00 MiB
Total PE 108143
Alloc PE / Size 0 / 0
Free PE / Size 108143 / 422.43 GiB
VG UUID Bb8l9e-es2z-PhaF-Gg3o-2is2-DZ1S-V2RsBF
</source>

If the new VG is visible on both nodes, we are ready to create our first logical volume using the lvcreate tool.

== Creating the First LV on the new VG ==

'''Node1'''

Now we'll create a simple 20 GiB logical volumes. We will use it as a shared GFS2 store for shared files and to store our Xen domU config files later on.

<source lang="bash">
lvcreate -L 20G -n xen_shared drbd0_vg0
</source>
<source lang="text">
Logical volume "xen_shared" created
</source>

'''Both'''

As before, we will check that the new logical volume is visible from both nodes by using the lvdisplay command:

<source lang="bash">
lvdisplay
</source>
<source lang="text">
--- Logical volume ---
LV Name /dev/drbd0_vg0/xen_shared
VG Name drbd0_vg0
LV UUID AqQizc-KBpX-2scN-WFLb-jIeF-QDcM-PlQW84
LV Write Access read/write
LV Status available
# open 0
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
</source>

Again, if this is visible from both nodes, we're set! Repeat this process for all future LVs you will want to create. We will do this a little later to create LVs for Xen VMs.

= Creating A Shared GFS FileSystem =

GFS is a cluster-aware file system that can be simultaneously mounted on two or more nodes at once. We will use it as a place to store ISOs that we'll use to provision our virtual machines.

== Install The GFS2 Utilities ==

Start by installing the [[GFS2]] tools:

<source lang="bash">
yum -y install gfs2-utils.x86_64
</source>

== Format Our CLVM LV With The GFS2 File System ==

'''Node1'''

'''Note''': The following example is designed for the cluster used in the prerequisite HowTo.
* If you have more than 2 nodes, increase the -j 2 to the number of nodes you want to mount this file system on.
* If your cluster is named something other than an-cluster (as set in the cluster.conf file), change -t an-cluster:xen_shared to match you cluster's name. The xen_shared can be whatever you like, but it must be unique in the cluster. I tend to use a name that matches the LV name, but this is my own preference and is not required.

To format the partition run:
<source lang="bash">
mkfs.gfs2 -p lock_dlm -j 2 -t an-cluster:xen_shared /dev/drbd0_vg0/xen_shared
</source>
<source lang="text">
This will destroy any data on /dev/drbd0_vg0/xen_shared.
It appears to contain: symbolic link to `../dm-0'

Are you sure you want to proceed? [y/n]
</source>

Acknowledge the warning, if any, and then press y if you are ready to proceed.

<source lang="bash">
y
</source>
<source lang="text">
Device: /dev/drbd0_vg0/xen_shared
Blocksize: 4096
Device Size 20.00 GB (5242880 blocks)
Filesystem Size: 20.00 GB (5242878 blocks)
Journals: 2
Resource Groups: 80
Locking Protocol: "lock_dlm"
Lock Table: "an-cluster:xen_shared"
UUID: A1487063-2A3F-43B1-3A36-44936B0B4D1E
</source>

Once the format completes, you can mount /dev/drbd0_vg0/xen_shared as you would a normal file system.

'''Both''':

To complete the example, lets mount the GFS2 partition we made just now on /shared and then use df -h to verify.

<source lang="bash">
mkdir /xen_shared
mount /dev/drbd0_vg0/xen_shared /xen_shared
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.8G 34G 8% /
tmpfs 466M 29M 438M 7% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
/dev/dm-0 20G 259M 20G 2% /xen_shared
</source>

You may have noticed that it shows /dev/dm-0 instead of /dev/drbd0_vg0/xen_shared. If you look at the later, you will see that it is simply a [[symlink]] to the former.

<source lang="bash">
ls -lah /dev/drbd0_vg0/xen_shared
</source>
<source lang="text">
lrwxrwxrwx. 1 root root 7 Sep 9 13:24 /dev/drbd0_vg0/xen_shared -> ../dm-0
</source>

== Add An Entry To /etc/fstab ==

The last step is to add an entry for this new partition to each node's /etc/fstab file.

=== Reference The GFS2 Partition By Device Path ===

This is the more traditional method of referencing the GFS2 partition by using it's device path directly.

'''Warning''': An incorrect edit of the /etc/fstab file can leave your system unable to boot! Please review the line generated above to make sure it is accurate and compatible with your setup before proceeding.

<source lang="bash">
vim /etc/fstab
</source>
<source lang="text">
#
# /etc/fstab
# Created by anaconda on Tue Sep 7 06:29:51 2010
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=865690db-85d0-44c5-9f32-ffb6fdf47060 / ext3 defaults 1 1
UUID=8b9822b6-a92e-48c9-96b5-f8943142319e /boot ext3 defaults 1 2
UUID=94b03547-a7e3-45bb-b2d5-837498b370f4 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
xenfs /proc/xen xenfs defaults 0 0
/dev/drbd0_vg0/xen_shared /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

=== Reference The GFS2 Partition By UUID ===

It is sometimes preferable to create an fstab entry that locates the device path via it's [[UUID]]. To do this, you can run the following command which, though a bit cryptic, will print out an /etc/fstab compatible string.

'''Warning''': The same warnings apply here as above

<source lang="bash">
echo `gfs2_edit -p sb /dev/drbd0_vg0/xen_shared | grep sb_uuid | sed -e "s/.*sb_uuid *$.*$/UUID=\L\1\E \/xen_shared\t\tgfs2\trw,suid,dev,exec,nouser,async\t0 0/"`
</source>
<source lang="text">
UUID=a1487063-2a3f-43b1-3a36-44936b0b4d1e /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

You may have noticed that defaults isn't used. Rather, all but the auto option are manually set. This is because the system will drop to single-user mode at boot if it can't mount an auto partition at boot time (auto being implied by defaults). Given that our GFS2 partition sits on top of DRBD and the cluster, there is no way to make it available that early in the boot process.

Further, the gfs2 init script specifically excludes entries in /etc/fstab that have the 'noauto option set. For this reason, we can't simply specify that as we need the init script to see the partition so that it is mounted when GFS2 starts and unmounted when it stops.

Now add this string to /etc/fstab.

<source lang="bash">
vim /etc/fstab
</source>
<source lang="text">
#
# /etc/fstab
# Created by anaconda on Tue Sep 7 06:29:51 2010
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=865690db-85d0-44c5-9f32-ffb6fdf47060 / ext3 defaults 1 1
UUID=8b9822b6-a92e-48c9-96b5-f8943142319e /boot ext3 defaults 1 2
UUID=94b03547-a7e3-45bb-b2d5-837498b370f4 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
xenfs /proc/xen xenfs defaults 0 0
UUID=a1487063-2a3f-43b1-3a36-44936b0b4d1e /xen_shared gfs2 rw,suid,dev,exec,nouser,async 0 0
</source>

Please note; At the time of writing this HowTo, there is a bug in findfs and mount. According to [http://www.ietf.org/rfc/rfc4122.txt RFC 4122], programs should accept a [[UUID]] in either upper or lower case. However, this is not currently the case, so you '''must''' pass the UUID in lower-case. Please see bugs [https://bugzilla.redhat.com/show_bug.cgi?id=632373 632373] and [https://bugzilla.redhat.com/show_bug.cgi?id=632385 632385].

== Testing The gfs2 Initialization Script ==

To verify that the new entry is valid, check gfs2's status.

<source lang="bash">
/etc/init.d/gfs2 status
</source>
<source lang="text">
Configured GFS2 mountpoints:
/xen_shared
Active GFS2 mountpoints:
/xen_shared
</source>

Now test stopping and restarting to ensure that the GFS2 partition unmounts and mounts properly.

Stop gfs2.

<source lang="bash">
/etc/init.d/gfs2 stop
</source>
<source lang="text">
Unmounting GFS2 filesystem (/xen_shared): [ OK ]
</source>

Check with df -h to ensure that the mount is gone.

<source lang="bash">
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.0G 35G 6% /
tmpfs 466M 23M 444M 5% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
</source>

Start gfs2 again.

<source lang="bash">
/etc/init.d/gfs2 start
</source>
<source lang="text">
Mounting GFS2 filesystem (/xen_shared): [ OK ]
</source>

Again, check with df -h that it has been remounted.

<source lang="bash">
df -h
</source>
<source lang="text">
Filesystem Size Used Avail Use% Mounted on
/dev/md1 39G 2.0G 35G 6% /
tmpfs 466M 29M 438M 7% /dev/shm
/dev/md0 243M 70M 161M 31% /boot
xenstore 466M 32K 466M 1% /var/lib/xenstored
/dev/dm-0 20G 259M 20G 2% /xen_shared
</source>

Done!

== Further Reading ==

* [[Grow a GFS2 Partition]]
* [[Hard drive has gone bad in DRBD]]

= Altering Daemon Start Order =

It is important that the various daemons in use by our cluster start in the right order. Most daemons will rely on services provided by another daemon to be running, and will not start or will not operate reliably otherwise.

We need to make sure that xend starts so that the network is stable. Then cman needs to start so that [[fencing]] and [[dlm]] are available. Next, drbd starts so that the clustered storage is available. Then clvmd must start so that the data on the DRBD resource is accessible. Now gfs2 needs to start so that the Xen domU configuration files can be found and finally xendomains must start to boot up the actual domU virtual machines.

To restate as a list, the start order must be:
* xend
* cman
* drbd
* clvmd
* gfs2
* xendomains

To make sure the start order is sane then, we'll edit each of the six daemon's init scripts and alter their Required-Start lines. To make the changes take effect, we will use chkconfig to remove and re-add them to the various start levels.

== Altering xend ==

This should already be done. If it isn't, please see "[[#Make xend play nice with clustering|Making xend play nice with clustering]]" above. If you are revisiting that section, you can skip the cman edit as we will need to make another change in the next step.

== Altering cman ==

This should already be done. If it isn't, please see "[[#Make xend play nice with clustering|Making xend play nice with clustering]]" above. If you are revisiting that section, you can skip the cman edit as we will need to make another change in the next step.

== Altering drbd ==

Now we will tell drbd to start after cman.

This requires the additional step of altering the chkconfig: - 70 08 line to instead read chkconfig: - 20 08. This isn't strictly needed, but will give more room for chkconfig to order the dependent daemons by allowing DRBD to be started as low as position 20, rather than waiting until position 70. This is somewhat more compatible with cman and clvmd which normally start at positions 21 and 24, respectively

<source lang="bash">
vim /etc/init.d/drbd
</source>
<source lang="text">
#!/bin/bash
#
# chkconfig: - 20 08
# description: Loads and unloads the drbd module
#
# Copright 2001-2008 LINBIT Information Technologies
# Philipp Reisner, Lars Ellenberg
#
### BEGIN INIT INFO
# Provides: drbd
# Required-Start: $local_fs $network $syslog cman
# Required-Stop: $local_fs $network $syslog
# Should-Start: sshd multipathd
# Should-Stop: sshd multipathd
# Default-Start:
# Default-Stop:
# Short-Description: Control drbd resources.
### END INIT INFO
</source>

== Altering clvmd ==

Now we will now tell clvmd to start after drbd.

'''Note''': There is currently a minor bug with lvm2-cluster version 2.02.73-2 in that /etc/init.d/clvmd is set by default to mode 0555. This is easily corrected by running the following command. Please check bug [https://bugzilla.redhat.com/show_bug.cgi?id=636066 636066] to see if this has been resolved.

<source lang="bash">
chmod u+w /etc/init.d/clvmd
</source>

Once you've got write access, edit the file.

<source lang="bash">
vim /etc/init.d/clvmd
</source>
<source lang="text">
#!/bin/bash
#
# chkconfig: - 24 76
# description: Starts and stops clvmd
#
# For Red-Hat-based distributions such as Fedora, RHEL, CentOS.
#
### BEGIN INIT INFO
# Provides: clvmd
# Required-Start: $local_fs drbd
# Required-Stop: $local_fs
# Default-Start:
# Default-Stop: 0 1 6
# Short-Description: Clustered LVM Daemon
### END INIT INFO
</source>

== Altering gfs2 ==

Now we will now tell gfs2 to start after clvmd. You will notice that cman is already listed under Required-Start and Required-Stop. It's true that cman must be started, but we've created a chain here so we can safely replace it with clvmd in the start line.

<source lang="bash">
vim /etc/init.d/gfs2
</source>
<source lang="text">
#!/bin/bash
#
# gfs2 mount/unmount helper
#
# chkconfig: - 26 74
# description: mount/unmount gfs2 filesystems configured in /etc/fstab

### BEGIN INIT INFO
# Provides: gfs2
# Required-Start: $network clvmd
# Required-Stop: $network
# Default-Start:
# Default-Stop:
# Short-Description: mount/unmount gfs2 filesystems configured in /etc/fstab
# Description: mount/unmount gfs2 filesystems configured in /etc/fstab
### END INIT INFO
</source>

== Altering xendomains ==

Finally, we will alter xendomains so that it starts last, after gfs2.

<source lang="bash">
vim /etc/init.d/xendomains
</source>
<source lang="text">
#!/bin/bash
#
# /etc/init.d/xendomains
# Start / stop domains automatically when domain 0 boots / shuts down.
#
# chkconfig: 345 99 00
# description: Start / stop Xen domains.
#
# This script offers fairly basic functionality. It should work on Redhat
# but also on LSB-compliant SuSE releases and on Debian with the LSB package
# installed. (LSB is the Linux Standard Base)
#
# Based on the example in the "Designing High Quality Integrated Linux
# Applications HOWTO" by Avi Alkalay
# <http://www.tldp.org/HOWTO/HighQuality-Apps-HOWTO/>
#
### BEGIN INIT INFO
# Provides: xendomains
# Required-Start: $syslog $remote_fs xend gfs2
# Should-Start:
# Required-Stop: $syslog $remote_fs xend
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Default-Enabled: yes
# Short-Description: Start/stop secondary xen domains
# Description: Start / stop domains automatically when domain 0
# boots / shuts down.
### END INIT INFO
</source>

== Applying The Changes ==

Change the start order by removing and re-adding all cluster-related daemons using chkconfig.

<source lang="bash">
chkconfig xenstored off; chkconfig xenconsoled off; chkconfig xend off; chkconfig cman off; chkconfig drbd off; chkconfig clvmd off; chkconfig gfs2 off; chkconfig xendomains off
chkconfig xendomains on; chkconfig gfs2 on; chkconfig clvmd on; chkconfig drbd on; chkconfig cman on; chkconfig xend on; chkconfig xenconsoled on; chkconfig xenstored on
</source>

Now verify that the start order is as we want it.

<source lang="bash">
ls -lah /etc/rc3.d/
</source>
<source lang="text">
lrwxrwxrwx. 1 root root 19 Sep 20 13:37 S26xenstored -> ../init.d/xenstored
lrwxrwxrwx. 1 root root 21 Sep 20 13:37 S27xenconsoled -> ../init.d/xenconsoled
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S28xend -> ../init.d/xend
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S29cman -> ../init.d/cman
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S70drbd -> ../init.d/drbd
lrwxrwxrwx. 1 root root 15 Sep 20 13:37 S71clvmd -> ../init.d/clvmd
lrwxrwxrwx. 1 root root 14 Sep 20 13:37 S72gfs2 -> ../init.d/gfs2
lrwxrwxrwx. 1 root root 20 Sep 20 13:37 S99xendomains -> ../init.d/xendomains
</source>

= Setting Up Xen =

'''''WARNING''''': Everything below here is pretty '''seriously screwed up'''.

'''Note''': This is not meant to be an extensive tutorial on Xen itself. It covers enough to get domU VMs provisioned in a manner that will take advantage of the cluster. As such, there is minimal explanation of configuration file options. If you need further help, please drop by the ##xen (yes, two ##) [[IRC]] channel on freenode.org.

== Install The Hypervisor Tools ==

These tools are very useful in provisioning and managing domU VMs.

<source lang="bash">
yum -y install virt-install virt-viewer
</source>

== Install The HVM/KVM Tools ==

For hvm (Hardware Virtual Machines), which is required for [http://www.virtuatopia.com/index.php/Virtualizing_Windows_Server_2008_with_Xen paravirtualized Microsoft] VMs, you must install the following packages as well.

<source lang="bash">
yum -y install qemu-kvm.x86_64 qemu-kvm-tools.x86_64
</source>

== Ensure That Virtualization Is Enabled ==

Many motherboards disable hvm by default in their [[BIOS]]. Assuming that you've got a dom0 kernel running at this stage, you can check if this is the case by checking the xm info output.

<source lang="bash">
xm info
</source>
<source lang="text">
host : an-node04.alteeve.com
release : 2.6.32.23-170.dom0_an1.fc13.x86_64
version : #1 SMP Sun Oct 10 20:39:19 EDT 2010
machine : x86_64
nr_cpus : 4
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 2209
hw_caps : 178bf3ff:efd3fbff:00000000:00001310:00802001:00000000:000037ff:00000000
virt_caps : hvm
total_memory : 4063
free_memory : 2987
node_to_cpu : node0:0-3
node_to_memory : node0:2987
node_to_dma32_mem : node0:2928
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline : dom0_mem=1024M
cc_compiler : gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC)
cc_compile_by : root
cc_compile_domain :
cc_compile_date : Mon Oct 11 01:10:38 EDT 2010
xend_config_format : 4
</source>

Look at the virt_caps and xen_caps lines. Notice the hvm entries? This shows that HVM, also known as "secure virtualization", has been enabled. If you do not see this, please check your mainboard manual for information on enabling this on your system.

'''Note''': The next paragraph applies only when running a vanilla kernel.

If you are running a vanilla kernel, you can check to see if your CPU has support for HVM guests but checking /proc/cpuinfo. What you're looking for depends on your CPU manufacturer. If you have an Intel CPU, you need to look for the vmx flag. Likewise, with AMD CPUs, you need to look for the svm flag.

For a more complete, if somewhat dated paper on this topic, please [http://fedoraproject.org/wiki/FedoraXenQuickstartFC6#Fully-virtualized_guests_.28HVM.2FIntel-VT.2FAMD-V.29 Fedora 6 Xen Quickstart Guide, System Requirements].

== Enabling Migration ==

By default, xend will not allow domU VMs from being migrated onto or off of a given dom0 host. Given that we've got a cluster though, we very much want this behaviour, so now we will enable it. This is done by making edits to /etc/xen/xend-config.sxp. Below is a concise list of options that must be set. Some exist already in the file and need to be commented out or altered.

'''Warning''': The values below are '''very''' permissive. Please review each option and improve the security to fit your network before going into production!

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>
<source lang="text">
(xend-http-server yes)
(xend-unix-server yes)
(xend-tcp-xmlrpc-server yes)
(xend-relocation-server yes)
(xend-udev-event-server yes)
(xend-port 8000)
(xend-relocation-port 8002)
(xend-address '')
(xend-relocation-address '')
(xend-relocation-hosts-allow '')
</source>

Once done, restart xend. It is usually safest to stop the cluster before hand to avoid accidental fencing caused by the underlying network being reconfigured.

<source lang="bash">
/etc/init.d/gfs2 stop
/etc/init.d/clvmd stop
/etc/init.d/cman stop
/etc/init.d/xend restart
/etc/init.d/cman start
/etc/init.d/clvmd start
/etc/init.d/gfs2 start
</source>

== Provisioning domU VMs ==

'''Note''': Long and ugly, but it worked. Needs more testing though.

Web-based install:

<source lang="bash">
# Fedora 13 x86_64 RPM builder VM
lvcreate -L 40G -n f13_builder_01 drbd0_vg0
virt-install --connect xen \
--name f13_builder_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--location http://192.168.1.10/f13/x86_64/img/ \
--os-type linux \
--os-variant fedora13 \
--disk path=/dev/drbd0_vg0/f13_builder_01 \
--network bridge=eth0,mac=00:16:3e:00:10:01 \
--vnc \
--paravirt

# A CentOS test server
lvcreate -L 40G -n c5_test_01 drbd0_vg0
virt-install --connect xen \
--name c5_test_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1-3 \
--location http://192.168.1.10/c5/x86_64/img/ \
--os-type linux \
--os-variant rhel5.4 \
--disk path=/dev/drbd0_vg0/c5_test_01 \
--network bridge=eth0,mac=00:16:3e:00:10:02 \
--vnc \
--paravirt

# Red Hat Enterprise Linux 6 beta 2 test server
lvcreate -L 40G -n rh6b2_test_01 drbd0_vg0
virt-install --connect xen \
--name rh6b2_test_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--location http://192.168.1.10/rhel6/x86_64/img/ \
--os-type linux \
--os-variant rhel6 \
--disk path=/dev/drbd0_vg0/rh6b2_test_01 \
--network bridge=eth0,mac=00:16:3e:00:10:03 \
--vnc \
--paravirt
</source>

ISO-based install:

<source lang="bash">
### Still sorting this out - doesn't seem to find 'qemu-dm' (wants to use 'lib64' but it's in 'lib').
# Windows 2008 Business Server
lvcreate -L 50G -n win2008_01 drbd0_vg0
virt-install --connect xen \
--name win2008_01 \
--ram 1024 \
--arch x86_64 \
--vcpus 1 \
--cpuset 1 \
--cdrom /xen_shared/iso/Win_Server_2008_Bis_x86_64.iso \
--os-type windows \
--os-variant win2k8 \
--disk path=/dev/drbd_vg0/win2008_01 \
--network bridge=eth0,mac=00:16:3e:00:10:04 \
--vnc \
--hvm
</source>

== Enabling Migration ==

We need to make some changes to the xend-config.sxp in order to enable migrating domU VMs between the nodes.

So start by editing the /etc/xen/xend-config.sxp file.

<source lang="bash">
vim /etc/xen/xend-config.sxp
</source>

Note that this is not meant to be an extensive list, nor a set of changes that maximize security. This will get you up and running, then from there you can explore the other options one at a time to improve security.

=== xen-api-server ===

This option allows for one or more access method definitions to your domU VMs on a given node. By default, it is set to (xen-api-server ((unix))) which only allows access from the local machine. We'll add a new access definition that will tell Xen to listen on localhost and on our back-channel on [[TCP]] port 9001.

<source lang="text">
# Default:
# (xen-api-server ((unix)))
(xen-api-server ((9001 pam '^localhost$ ^10\\.0\\.1') (unix none)))
</source>

== Provision A Virtual Machine ==

Let's walk through the creation of a simple [[CentOS]] 5.x [[domU]]. We'll call it c5_test_01. Personally, I like to name my virtual machines with the format "os_role_seq" (<Operating System ID>_<Role of the VM>_<Sequence Integer>). There are no (known) restrictions on virtual machine names, so feel free to use names that made sense for you.

=== Install virt-manager ===

We will now start using a graphical tool called virt-manager. It simplifies the creation of domU VMs quite a bit.

<source lang="bash">
yum install virt-manager
</source>

Be sure that you've enabled X-Forwarding if you've ssh'ed into your nodes. This may require modifying your /etc/sshd_config file and changing X11Forwarding to yes. Then when you connect to the node, you will need to use the ssh -X user@node switch.

We'll come back to this program shortly.

=== Create The CLVM LV ===

We will create a new [[CLVM]] [[LV]] for this [[VM]]. It will act as the VM's hard drive, so it needs to have enough capacity for the OS and any data you expect the VM to need. As we're creating a hypothetical router/firewall, we will not need much space. We'll use 20 [[GiB]], which should be more than enough.

As before, this will only need to be done on one node. Being that it is on top of the DRBD resource, it will be visible to both nodes.

'''Node1'''

<source lang="bash">
lvcreate -L 20G -n c5_test_01 drbd0_vg0
</source>
<source lang="text">
Logical volume "c5_test_01" created
</source>

'''Both'''

Verify that the new LV is indeed visible from both nodes now.

<source lang="bash">
lvdisplay
</source>
<source lang="text">
--- Logical volume ---
LV Name /dev/drbd0_vg0/xen_shared
VG Name drbd0_vg0
LV UUID yjt41B-cdc4-LGV4-Ds0m-eRVv-FIY7-3FK4Kb
LV Write Access read/write
LV Status available
# open 1
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0

--- Logical volume ---
LV Name /dev/drbd0_vg0/c5_test_01
VG Name drbd0_vg0
LV UUID eOcBZz-nJzz-wHwg-4cBL-ZZHm-1cm6-Sf82xC
LV Write Access read/write
LV Status available
# open 0
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1
</source>

Now we've got the space, we're ready to provision!

=== Connecting To The Hypervisor From virt-manager ===

At this point, we'll be using the graphical virt-manager tool.

==== Running virt-manager Directly On a Node ====

If you are working on the nodes directly, you can run virt-manager but you will need to have Gnome or KDE started first. The easiest way to do this is to switch to run level 5. As an unprivileged user, run the following command.

<source lang="bash">
startx
</source>

From there, click on Application -> System Tools -> Virtual machine Manager

[[Image:virt-manager_local-node_01.png|thumb|500px|center|Where to find virt-manager in Fedora 13]]

==== Running virt-manager Remotely Over X-Forwarding ====

Some features in virt-manager don't work well unless they are run off of the local machine. The best way to accomplish this from a remote system is to use X-Forwarding, which was discussed in the [[#Install virt-manager|Install virt-manager]] above.

Generally speaking, I prefer to provision VMs using virt-manager directly on a cluster node. This avoids some problems, like accessing underlying LVM volumes. So to do this when you can't directly sit at a node, ssh into your node using the -X switch.

<source lang="bash">
ssh -X root@an-node01
</source>
<source lang="text">
Last login: Sun Sep 19 22:02:38 2010
[root@an-node01 ~]#
</source>

Now start virt-manager at the command line and wait for it to open on your local workstation.

<source lang="bash">
virt-manager
</source>

'''Note''': If your local workstation has a different version of X server or Gnome, the virt-manager window could look fairly primitive. It will work just the same though, and all the icons will be in the same place.

==== Running virt-manager Locally On a Workstation ====

The virt-manager tool can be installed on a computer outside of the cluster and then used to connect to the hypervisor on either node. This is very useful for daily usage of your virtual machines, but isn't recommended during the provisioning stage. In some cases, running virt-manager remotely will cause it to not see the LVM volumes properly, preventing you from choosing the LV for your new VM.

To connect to the hypervisor on the cluster nodes, click on File -> Add Connection. Change the Hypervison to Xen and then change the Connection to Remote tunnel over SSH.

[[Image:virt-manager_remote-workstation_01.png|thumb|500px|center|Adding a remote connection in virt-manager from a workstation]]

In some cases, not all of the nodes on the network will be seen. In the screen shot above, you can see that an-node02 can be seen, but an-node01 wasn't. To add the missing one, just type node-name.local (an-node01.local above). Once selected, hit Connect.

Once the two nodes have been connected, you should see them on the virt-manager main page.

[[Image:virt-manager_remote-workstation_02.png|thumb|500px|center|Remote connection in virt-manager to both nodes in the cluster]]

Once you seen your nodes, you can proceed to use virt-manager the same as if you were running it on one of the nodes directly.

=== Using virt-manager To Provision Our VM ===

How you use virt-manager will be the same regardless of how or where you started it. The following steps will walk you through the program's use.

With the Virtual Machine Manager main menu connected to localhost (QEMU), we can now begin the process of provisioning a new Xen domU virtual machine! The top-left icon, which looks like a monitor with the "play" button on it's screen, will be used to provision our virtual machine.

[[Image:virt-manager_local-node_02.png|thumb|500px|center|The virt-manager main menu]]

'''Madi''': You left off here.

Once you press the Create a new virtual machine button you will be asked to name the VM and to choose the install type. Even if you have a [[PXE]] server, choose Network Install.

[[Image:virt-manager_provision_02.png|thumb|500px|center|Provisioning a Xen domU - Naming the VM and choosing the install source]

=== Storing The domU Configuration Files ===

'''ToDo'''

'''Node1'''

As this is our first [[domU]], we'll start by creating our shared configuration directory on the /xen_shared [[GFS2]] partition we just created.

<source lang="bash">
mkdir /xen_shared/config
</source>

To confirm again that the GFS2 partition is working, verify that this directory is visible from both nodes.

<source lang="bash">
ls -lah /xen_shared/
</source>
<source lang="text">
total 20K
drwxr-xr-x. 3 root root 3.8K Sep 13 12:57 .
dr-xr-xr-x. 27 root root 4.0K Sep 13 11:32 ..
drwxr-xr-x. 2 root root 3.8K Sep 13 12:57 config
</source>


<source lang="bash">
</source>
<source lang="text">
</source>

{{footer}}

Abandoned - Two Node Fedora 13 Cluster

2010-10-13T17:58:58Z

SRSullivan: /* Testing IPMI */ Added line to enable service on boot.

{{howto_header}}

'''''Notice''''', '''Sep. 08, 2010''' - ''Draft 1'': This is the first draft of this HowTo. It has not been peer-reviewed yet, so please proceed with caution and on a test cluster. Any and all feedback is much appreciated.

= Overview =

This paper has one goal;

# How to assemble the simplest cluster possible, a '''2-Node Cluster''', which you can then expand on for your own needs.

With this completed, you can then jump into "step 2" papers that will show various uses of a two node cluster:

# How to create a "floating" virtual machine that can move between the two nodes in the event of a node failure, maximizing up time.
# How to create simple resources that can move between nodes. Examples will be a simple PostgreSQL database, DHCP, DNS and web servers.

== Prerequisites ==

It is expected that you are already comfortable with the Linux command line, specifically [[bash]], and that you are familiar with general administrative tasks in Red Hat based distributions, specifically [[Fedora]]. You will also need to be comfortable using editors like [[vim]], [[nano]] or similar. This paper uses vim in examples. Simply substitute your favourite editor in it's place.

You are also expected to be comfortable with networking concepts. You will be expected to understand TCP/IP, [[multicast]], broadcast, subnets and netmasks, routing and other relatively basic networking concepts. Please take the time to become familiar with these concepts before proceeding.

This said, where feasible, as much detail as is possible will be provided. For example, all configuration file locations will be shown and functioning sample files will be provided.

== Platform ==

This paper will implement the [[Red Hat]] Cluster Suite using the Fedora v13 distribution. This paper uses the [[x86_64]] repositories, however, if you are on an [[i386]] (32 bit) system, you should be able to follow along fine. Simply replace x86_64 with i386 or i686 in package names.

You can either download the stock Fedora 13 DVD ISO, or you can try out the alpha [[#AN!Cluster Install|AN!Cluster Install]] DVD. (4.3GB iso). If you use the latter, please test it out on a development or test cluster. If you have any problems with the AN!Cluster variant Fedora distro, please [[Digimer|contact me]] and let me know what your trouble was.

== Why Fedora 13? ==

Generally speaking, I much prefer to use a server-oriented Linux distribution like [[CentOS]], [[Debian]] or similar. However, there have been many recent changes in the Linux-Clustering world that have made all of the currently available server-class distributions obsolete. With luck, once [[Red Hat]] Enterprise Linux and [[CentOS]] version 6 is released, this will change.

Until then, [[Fedora]] version 13 provides the most up to date binary releases of the new implementation of the clustering stack available. For this reason, FC13 is the best choice in clustering, if you want to be current. To mitigate some of the issues introduced by using a workstation distribution, many packages will be stripped out of the default install.

== Focus ==

Clusters can serve to solve three problems; '''Reliability''', '''Performance''' and '''Scalability'''.

This focus of the cluster described in this paper is primarily '''reliability'''. Second to this, '''scalability''' will be the priority leaving '''performance''' to be addressed only when it does not impact the first two criteria. This is not to indicate that performance is not a valid priority, it simply isn't the priority of this paper.

== Goal ==

At the end of this paper, you should have a fully functioning two-node array capable of hosting a "floating" resources. That is, resources that exist on one node and can be easily moved to the other node with minimal effort and down time. This should conclude with a solid foundation for adding more virtual servers up to the limit of your cluster's resources.

This paper should also serve to show how to build the foundation of any other cluster configuration. This paper has a core focus of introducing the main issues that come with clustering and hopes to serve as a foundation for any cluster configuration outside the scope of this paper.

= Begin =

Let's begin!

== Hardware ==

We will need two physical servers each with the following hardware:
* One or more multi-core [[CPU]]s with Virtualization support.
* Three network cards; At least one should be gigabit or faster.
* One or more hard drives.
* You will need some form of a [[fence|fence device]]. This can be an [[IPMI]]-enabled server, a [http://nodeassassin.org Node Assassin], a fenceable [http://www.apc.com/products/resource/include/techspec_index.cfm?base_sku=AP7900&tab=features PDU] or similar.

This paper uses the following hardware:
* ASUS [http://support.asus.com/search/search.aspx?keyword=m4a78l-m&SLanguage=en-us M4A78L-M]
* AMD Athlon II x2 250
* 2GB Kingston DDR2 KVR800D2N6K2/4G (4GB kit split between the two nodes)
* 1x Intel 82540 PCI NICs
* 1x D-Link DGE-560T
* Node Assassin

This is not an endorsement of the above hardware. I bought what was within my budget that would serve the purposes of creating this document. What you purchase shouldn't matter, so long as the minimum requirements are met.

== Pre-Assembly ==

With multiple NICs, it is quite likely that the mapping of physical devices to logical ethX devices may not be ideal. This is a particular issue if you decide to [[Setting Up a PXE Server in Fedora|network boot]] your install media. In that case, if the wrong NIC is chosen for eth0, you will be presented with a list of MAC addresses to attempt setup with.

Before you assemble your servers, record their network cards' [[MAC]] addresses. I like to keep simple text files like these:

<source lang="bash">
cat an-node01.mac
</source>
<source lang="text">
90:E6:BA:71:82:EA eth0 # Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
00:21:91:19:96:53 eth1 # D-Link System Inc DGE-560T PCI Express Gigabit Ethernet Adapter
00:0E:0C:59:46:E4 eth2 # Intel Corporation 82540EM Gigabit Ethernet Controller
</source>

<source lang="bash">
cat an-node02.mac
</source>
<source lang="text">
90:E6:BA:71:82:D8 eth0 # Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
00:21:91:19:96:5A eth1 # D-Link System Inc DGE-560T PCI Express Gigabit Ethernet Adapter
00:0E:0C:59:45:78 eth2 # Intel Corporation 82540EM Gigabit Ethernet Controller
</source>

Feel free to record the information in any way that suits you the best.

== OS Install ==

Start with a stock Fedora 13 install. This How-To uses Fedora 13 x86_64, however it should be fairly easy to adapt to other recent Fedora versions. This document is also attempting to be easily ported to [[CentOS]]/[[RHEL]] version 6 once it is released. This will not adapt well to CentOS/RHEL version 5 though... Much of the cluster stack has changed dramatically since it was initially released.

These are sample kickstart script used by this paper. Be sure to set your own password string and network settings.

'''''Warning'''''! These kickstart scripts '''''will erase your hard drive'''''! Adapt them, don't blindly use them.

Generic cluster node kickstart scripts.
* [[Fedora13 KS an-node01.ks|an-node01.ks]]
* [[Fedora13 KS an-node02.ks|an-node02.ks]]

== AN!Cluster Install ==

If you are feeling brave, below is a link to a custom install DVD that contains the kickstart scripts to setup nodes and an an-cluster directory with all the configuration files.

* Download the custom '''AN!Cluster v0.2.001''' Install DVD. (4.5[[GiB]] iso). (Currently disabled - Reworking for F13)

== Post OS Install ==

Once the OS is installed, we need to do a couple things;

# Disable selinux
# Setup networking.
# Change the default run-level.

=== Disable 'selinux' ===

To allow this paper to focus on clustering, we will disable selinux. Obviously, this introduces security issues that you may not be comfortable with. If you wish to leave selinux enabled, it will be up to you to sort out issues that crop up.

To disable selinux, edit /etc/selinux/config and change SELINUX=enforcing to SELINUX=permissive. You will need to reboot in order for the changes to take effect.

=== Change the Default Run-Level ===

'''This is an optional step'''. It improves performance only, it is not a required step.

If you don't plan to work on your nodes directly, it makes sense to switch the default run level from 5 to 3. This prevents the window manager, like Gnome or KDE, from starting at boot. This frees up a fair of memory and system resources and reduces the possible attack vectors.

To do this, edit /etc/inittab, change the id:5:initdefault: line to id:3:initdefault: and then switch to run level 3:

<source lang="bash">
vim /etc/inittab
</source>
<source lang="text">
id:3:initdefault:
</source>
<source lang="bash">
init 3
</source>

=== Setup Networking ===

We need to remove NetworkManager, enable network, configure the ifcfg-eth* files and then start the network daemon.

==== Managed Switch Warning ====

'''WARNING''': Please pay attention to this warning. The vast majority of cluster problems end up being network related. The hardest ones to diagnose are usually multicast issues.

If you use a managed switch, be careful about enabling [[Multicast IGMP Snooping]] or [[Spanning Tree Protocol]]! They have been known to cause problems by not allowing multicast packets to reach all nodes. This can cause somewhat random break-downs in communication between your nodes, leading to fences and DLM lock timeouts.

'''Ensure the [[PIM Routing]] is setup and working.

If you have problems with your cluster not forming, or seemingly random fencing, try using an unmanaged switch or use a cross-over cable. If the problem goes away, you are most likely dealing with a managed switch configuration problem.

==== Network Layout ====

This setup expects you to have three physical network cards connected to three independent networks. To have a common vernacular, lets use this table to describe them:

{|
!class="cell_all"|Network Description
!class="cell_tbr"|Short Name
!class="cell_tbr"|Device Name
!class="cell_tbr"|Suggested Subnet
!class="cell_tbr"|NIC Properties
|-
|class="cell_blr"|Back-Channel Network
|class="cell_br"|BCN
|class="cell_br"|eth0
|class="cell_br"|192.168.1.0/24
|class="cell_br"|NICs with [[IPMI]] piggy-back '''must''' be used here. Second-fastest NIC should be used here.
|-
|class="cell_blr"|Storage Network
|class="cell_br"|SN
|class="cell_br"|eth1
|class="cell_br"|192.168.2.0/24
|class="cell_br"|Fastest NIC should be used here.
|-
|class="cell_blr"|Internet-Facing Network
|class="cell_br"|IFN
|class="cell_br"|eth2
|class="cell_br"|192.168.3.0/24
|class="cell_br"|Remaining NIC should be used here. If using a [[Setting Up a PXE Server in Fedora|PXE server]], this should be a bootable NIC.
|}

Take note of these concerns when planning which NIC to use on each subnet. These issues are presented in the order that they must be addressed in:
# If your nodes have [[IPMI]] piggy-backing on a normal NIC, that NIC '''''must''''' be used on BCN subnet. Having your fence device accessible on a subnet that can be remotely accessed can pose a ''major'' security risk.
# The fastest NIC should be used for your SN subnet. Be sure to know which NICs support the largest jumbo frames when considering this.
# If you still have two NICs to choose from, use the fastest remaining NIC for your BCN subnet. This will minimize the time it takes to perform tasks like hot-migration of live virtual machines.
# The final NIC should be used for the IFN subnet.

==== Node IP Addresses ====

Obviously, the IP addresses you give to your nodes should be ones that suit you best. In this example, the following IP addresses are used:

{|
|class="cell_br"| 
|class="cell_tbr"|'''Internet-Facing Network''' (IFN)
|class="cell_tbr"|'''Storage Network''' (SN)
|class="cell_tbr"|'''Back-Channel Network''' (BCN)
|-
|class="cell_blr"|'''an-node01'''
|class="cell_br"|192.168.1.71
|class="cell_br"|192.168.2.71
|class="cell_br"|192.168.3.71
|-
|class="cell_blr"|'''an-node02'''
|class="cell_br"|192.168.1.72
|class="cell_br"|192.168.2.72
|class="cell_br"|192.168.3.72
|}

==== Remove NetworkManager ====

Some cluster software '''will not''' start with NetworkManager installed. This is because it is designed to be a highly-adaptive network system that can accommodate frequent changes in the network. To simplify these network transitions for end-users, a lot of reconfiguration of the network is done behind the scenes.

For workstations, this is wonderful. For clustering, this can be disastrous. Transient network outages are already a risk to a cluster's stability!

So first up, make sure that NetworkManager is completely removed from your system. If you used the [[#OS Install|kickstart]] scripts, then it was not installed. Otherwise, run:

<source lang="bash">
yum remove NetworkManager NetworkManager-gnome NetworkManager-openconnect NetworkManager-openvpn NetworkManager-pptp NetworkManager-vpnc cnetworkmanager knetworkmanager knetworkmanager-openvpn knetworkmanager-pptp knetworkmanager-vpnc libproxy-networkmanager yum-NetworkManager-dispatcher
</source>

==== Setup 'network' ====

Before proceeding with network configuration, check to see if your network cards are aligned to the proper ethX network names. If they need to be adjusted, please follow this How-To before proceeding:

* [[Changing the ethX to Ethernet Device Mapping in Fedora]]

There are a few ways to configure network in Fedora:
* system-config-network (graphical)
* system-config-network-tui (ncurses)
* Directly editing the /etc/sysconfig/network-scripts/ifcfg-eth* files. (See: [http://docs.fedoraproject.org/en-US/Fedora/12/html/Deployment_Guide/s1-networkscripts-interfaces.html here] for a full list of options)

Do not proceed until your node's networking is fully configured.

==== Update the Hosts File ====

Some applications expect to be able to call nodes by their name. To accommodate this, and to ensure that inter-node communication takes place on the back-channel subnet, we remove any existing hostname entries and then add the following to the /etc/hosts file:

'''Note''': Any pre-existing entries matching the name returned by uname -n must be removed from /etc/hosts. There is a good chance there will be an entry that resolves to 127.0.0.1 which would cause problems later.

Obviously, adapt the names and IPs to match your nodes and subnets. The only critical thing is to make sure that the name returned by uname -n is resolvable to the back-channel subnet. I like to add a short-form name for convenience.

The updated /etc/hosts file should look something like this:

<source lang="bash">
vim /etc/hosts
</source>
<source lang="text">
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

# Internet Facing Network
192.168.1.71 an-node01 an-node01.alteeve.com an-node01.bcn
192.168.1.72 an-node02 an-node02.alteeve.com an-node02.bcn

# Storage Network
192.168.2.71 an-node01.sn
192.168.2.72 an-node02.sn

# Back Channel Network
192.168.3.71 an-node01.ifn
192.168.3.72 an-node02.ifn

# Node Assassins
192.168.3.61 batou batou.alteeve.com
192.168.3.62 motoko motoko.alteeve.com
</source>

'''Note''': I use Node Assassins. If you use IPMI or other fence devices, alter the entries as appropriate.

Now to test this, ping both nodes by their name, as returned by uname -n, and make sure the ping packets are sent on the back channel network (192.168.1.0/24).

<source lang="bash">
ping -c 5 an-node01.alteeve.com
</source>
<source lang="text">
PING an-node01 (192.168.1.71) 56(84) bytes of data.
64 bytes from an-node01 (192.168.1.71): icmp_seq=1 ttl=64 time=0.399 ms
64 bytes from an-node01 (192.168.1.71): icmp_seq=2 ttl=64 time=0.403 ms
64 bytes from an-node01 (192.168.1.71): icmp_seq=3 ttl=64 time=0.413 ms
64 bytes from an-node01 (192.168.1.71): icmp_seq=4 ttl=64 time=0.365 ms
64 bytes from an-node01 (192.168.1.71): icmp_seq=5 ttl=64 time=0.428 ms

--- an-node01 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.365/0.401/0.428/0.030 ms
</source>
<source lang="bash">
ping -c 5 an-node02.alteeve.com
</source>
<source lang="text">
PING an-node02 (192.168.1.72) 56(84) bytes of data.
64 bytes from an-node02 (192.168.1.72): icmp_seq=1 ttl=64 time=0.419 ms
64 bytes from an-node02 (192.168.1.72): icmp_seq=2 ttl=64 time=0.405 ms
64 bytes from an-node02 (192.168.1.72): icmp_seq=3 ttl=64 time=0.416 ms
64 bytes from an-node02 (192.168.1.72): icmp_seq=4 ttl=64 time=0.373 ms
64 bytes from an-node02 (192.168.1.72): icmp_seq=5 ttl=64 time=0.396 ms

--- an-node02 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.373/0.401/0.419/0.030 ms
</source>

Be sure that, if your [[fence]] device uses a name, that you include entries to resolve it as well. You can see how I've done this with the two [[Node Assassin]] devices I use. The same applies to [[IPMI]] or other devices, if you plan to reference them by name.

Fencing will be discussed in more detail later on in this HowTo.

==== Disable Firewalls ====

Be sure to flush netfilter tables and disable iptables and ip6tables from starting on our nodes.

There will be enough potential sources of problem as it is. Disabling firewalls at this stage will minimize the chance of an errant iptables rule messing up our configuration. If, before launch, you wish to implement a firewall, feel free to do so but be sure to thoroughly test your cluster to ensure no problems were introduced.

<source lang="bash">
chkconfig --level 2345 iptables off
/etc/init.d/iptables stop
chkconfig --level 2345 ip6tables off
/etc/init.d/ip6tables stop
</source>

==== Setup SSH Shared Keys ====

'''This is an optional step'''. Setting up shared keys will allow your nodes to pass files between one another and execute commands remotely without needing to enter a password. This is obviously somewhat risky from a security point of view. As such, it is up to you whether you do this or not. This is not meant to be a security-focused How-To, so please independently study the risks.

If you're a little new to [[SSH]], it can be a bit confusing keeping connections straight in you head. When you connect to a remote machine, you start the connection on your machine as the user you are logged in as. This is the source user. When you call the remote machine, you tell the machine what user you want to log in as. This is the remote user.

You will need to create an SSH key for each source user, and then you will need to copy the newly generated public key to each remote machine's user directory that you want to connect to. In this example, we want to connect to either node, from either node, as the root user. So we will create a key for each node's root user and then copy the generated public key to the ''other'' node's root user's directory.

For each user, on each machine you want to connect '''from''', run:

<source lang="bash">
# The '2047' is just to screw with brute-forces a bit. :)
ssh-keygen -t rsa -N "" -b 2047 -f ~/.ssh/id_rsa
</source>
<source lang="text">
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
08:d8:ed:72:38:61:c5:0e:cf:bf:dc:28:e5:3c:a7:88 root@an-node01.alteeve.com
The key's randomart image is:
+--[ RSA 2047]----+
| .. |
| o.o. |
| . ==. |
| . =+. |
| + +.S |
| + o |
| = + |
| ...B o |
| E ...+ |
+-----------------+
</source>

This will create two files: the private key called ~/.ssh/id_dsa and the public key called ~/.ssh/id_dsa.pub. The private '''''must never''''' be group or world readable! That is, it should be set to mode 0600.

The two files should look like:

'''Private key''':
<source lang="bash">
cat ~/.ssh/id_rsa
</source>
<source lang="text">
-----BEGIN RSA PRIVATE KEY-----
MIIEoQIBAAKCAQBlL42DC+NJVpJ0rdrWQ1rxGEbPrDoe8j8+RQx3QYiB014R7jY5
EaTenThxG/cudgbLluFxq6Merfl9Tq2It3k9Koq9nV9ZC/vXBcl4MC7pGSQaUw2h
DVwI7OCSWtnS+awR/1d93tANXRwy7K5ic1pcviJeN66dPuuPqJEF/SKE7yEBapMq
sN28G4IiLdimsV+UYXPQLOiMy5stmyGhFhQGH3kxYzJPOgiwZEFPZyXinGVoV+qa
9ERSjSKAL+g21zbYB/XFK9jLNSJqDIPa//wz0T+73agZ0zNlxygmXcJvapEsFGDG
O6tcy/3XlatSxjEZvvfdOnC310gJVp0bcyWDAgMBAAECggEAMZd0y91vr+n2Laln
r8ujLravPekzMyeXR3Wf/nLn7HkjibYubRnwrApyNz11kBfYjL+ODqAIemjZ9kgx
VOhXS1smVHhk2se8zk3PyFAVLblcsGo0K9LYYKd4CULtrzEe3FNBFje10FbqEytc
7HOMvheR0IuJ0Reda/M54K2H1Y6VemtMbT+aTcgxOSOgflkjCTAeeOajqP5r0TRg
1tY6/k46hLiBka9Oaj+QHHoWp+aQkb+ReHUBcUihnz3jcw2u8HYrQIO4+v4Ud2kr
C9QHPW907ykQTMAzhMvZ3DIOcqTzA0r857ps6FANTM87tqpse5h2KfdIjc0Ok/AY
eKgYAQKBgQDm/P0RygIJl6szVhOb5EsQU0sBUoMT3oZKmPcjHSsyVFPuEDoq1FG7
uZYMESkVVSYKvv5hTkRuVOqNE/EKtk5bwu4mM0S3qJo99cLREKB6zNdBp9z2ACDn
0XIIFIalXAPwYpoFYi1YfG8tFfSDvinLI6JLDT003N47qW1cC5rmgQKBgHAkbfX9
8u3LiT8JqCf1I+xoBTwH64grq/7HQ+PmwRqId+HyyDCm9Y/mkAW1hYQB+cL4y3OO
kGL60CZJ4eFiTYrSfmVa0lTbAlEfcORK/HXZkLRRW03iuwdAbZ7DIMzTvY2HgFlU
L1CfemtmzEC4E6t5/nA4Ytk9kPSlzbzxfXIDAoGAY/WtaqpZ0V7iRpgEal0UIt94
wPy9HrcYtGWX5Yk07VXS8F3zXh99s1hv148BkWrEyLe4i9F8CacTzbOIh1M3e7xS
pRNgtH3xKckV4rVoTVwh9xa2p3qMwuU/jMGdNygnyDpTXusKppVK417x7qU3nuIv
1HzJNPwz6+u5GLEo+oECgYAs++AEKj81dkzytXv3s1UasstOvlqTv/j5dZNdKyZQ
72cvgsUdBwxAEhu5vov1XRmERWrPSuPOYI/4m/B5CYbTZgZ/v8PZeBTg17zgRtgo
qgJq4qu+fXHKweR3KAzTPSivSiiJLMTiEWb5CD5sw6pYQdJ3z5aPUCwChzQVU8Wf
YwKBgQCvoYG7gwx/KGn5zm5tDpeWb3GBJdCeZDaj1ulcnHR0wcuBlxkw/TcIadZ3
kqIHlkjll5qk5EiNGNlnpHjEU9X67OKk211QDiNkg3KAIDMKBltE2AHe8DhFsV8a
Mc/t6vHYZ632hZ7b0WNuudB4GHJShOumXD+NfJgzxqKJyfGkpQ==
-----END RSA PRIVATE KEY-----
</source>

'''Public key''':
<source lang="bash">
cat ~/.ssh/id_rsa.pub
</source>
<source lang="text">
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAGUvjYML40lWknSt2tZDWvEYRs+sOh7yPz5FDHdBiIHTXhHuNjkRpN6dOHEb9y52BsuW4XGrox6t+X1OrYi3eT0qir2dX1kL+9cFyXgwLukZJBpTDaENXAjs4JJa2dL5rBH/V33e0A1dHDLsrmJzWly+Il43rp0+64+okQX9IoTvIQFqkyqw3bwbgiIt2KaxX5Rhc9As6IzLmy2bIaEWFAYfeTFjMk86CLBkQU9nJeKcZWhX6pr0RFKNIoAv6DbXNtgH9cUr2Ms1ImoMg9r//DPRP7vdqBnTM2XHKCZdwm9qkSwUYMY7q1zL/deVq1LGMRm+9906cLfXSAlWnRtzJYM= root@an-node01.alteeve.com
</source>

Copy the public key and then ssh normally into the remote machine as the root user. Create a file called ~/.ssh/authorized_keys and paste in the key.

From '''an-node01''', type:

<source lang="bash">
ssh root@an-node02
</source>
<source lang="text">
The authenticity of host 'an-node02 (192.168.1.72)' can't be established.
RSA key fingerprint is d4:1b:68:5f:fa:ef:0f:0c:16:e7:f9:0a:8d:69:3e:c5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'an-node02,192.168.1.72' (RSA) to the list of known hosts.
root@an-node02's password:
Last login: Fri Oct 1 20:07:01 2010 from 192.168.1.102
</source>

You will now be logged into an-node02 as the root user. Create the ~/.ssh/authorized_keys file and paste into it the public key from an-node01. If the remote machine's user hasn't used ssh yet, their ~/.ssh directory will not exist.

<source lang="bash">
cat ~/.ssh/authorized_keys
</source>
<source lang="text">
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAGUvjYML40lWknSt2tZDWvEYRs+sOh7yPz5FDHdBiIHTXhHuNjkRpN6dOHEb9y52BsuW4XGrox6t+X1OrYi3eT0qir2dX1kL+9cFyXgwLukZJBpTDaENXAjs4JJa2dL5rBH/V33e0A1dHDLsrmJzWly+Il43rp0+64+okQX9IoTvIQFqkyqw3bwbgiIt2KaxX5Rhc9As6IzLmy2bIaEWFAYfeTFjMk86CLBkQU9nJeKcZWhX6pr0RFKNIoAv6DbXNtgH9cUr2Ms1ImoMg9r//DPRP7vdqBnTM2XHKCZdwm9qkSwUYMY7q1zL/deVq1LGMRm+9906cLfXSAlWnRtzJYM= root@an-node01.alteeve.com
</source>

Now log out and then log back into the remote machine. This time, the connection should succeed without having entered a password!

= Initial Cluster Setup =

Before we get into specifics, let's take a minute to talk about the major components used in our cluster.

== Core Program Overviews ==

These are the core programs that may be new to you that we will use to build our cluster. Before we configure them, let's take a minute to understand their roles.

=== Cluster Manager ===

The cluster manager, cman, takes the configuration file [[Two-Node Fedora 13 cluster.conf|/etc/cluster/cluster.conf]] configuration file and uses it to configure and start the various cluster elements. This is where nodes are defined, fence devices are configured and various cluster tolerances are set.

=== Corosync ===

[http://www.corosync.org Corosync] is, essentially, the clustering kernel which provides the essential services for cluster-aware applications and other cluster services. At it's core is the [[totem]] protocol which provides cluster membership and ordered messaging. On top of this, it implements a number of core clustering services like [[#cpg; Closed Process Group|closed process group messaging]], [[quorum]] and [[confdb]] (an object database).

It's goal is to provide a substantially simpler and more flexible set of APIs to facilitate clustering in Linux. It manages which nodes are in the cluster, it triggers error messages when something fails, manages cluster locking and so on. Most other clustered applications rely on corosync to know when something has happened or to announce when a cluster-related action has taken place.

Please note that the corosync_overview man page is considered out of date at the time of this writing.

==== cpg; Closed Process Group ====

'''Note''': Only install the following if you wish to review the man page.

<source lang="bash">
yum install corosynclib-devel
man cpg_overview
</source>

The closed process group is a small process layer on top of the totem protocol provided by corosync. It handles the sending and delivery of messages amongst nodes in a consistent order. It adds [[PID]]s and group names to the membership layer. Only members of the group get the messages, thus it is a "closed" group.

=== OpenAIS ===

'''Note''': We will not use OpenAIS. It is included in here to explain it's role to folks coming from the Cluster stable 2 environment.

OpenAIS is now an extension to Corosync that adds an open-source implementation of the [http://www.saforum.org/ Service Availability (SA) Forum's] 'Application Interface Specification' ([[AIS]]). It is an [[API]] and policy designed to be used by applications concerned with maintaining services during faults. AIS implements the 'Availability Management Framework' ([[AMF]]) which, in turn, provides for application fail over, cluster management ([[CLM]]), Checkpointing ([[CKPT]]), Eventing ([[EVT]]), Messaging ([[MSG]]), and Distributed Locking ([[DLOCK]]).

It does this by implementing pluggable shared libraries that get loaded into corosync at runtime. These libraries then can make use of corosync's internal API. Both corosync and openais services provide [[IPC]] interfaces so that application programmers can make use of their behaviour. In short; applications can use OpenAIS to be cluster-aware. It's libraries are used by some applications, including Pacemaker. In our application, we will only be using it's libraries indirectly.

Please note that the openais_overview man page is considered out of date at the time of this writing.

'''Note''': Only install the following if you wish to review the man page.

<source lang="bash">
yum install openais
</source>

== Setup Core Programs ==

We now need to edit and create the configuration files for our cluster.

=== Setup cman ===

You may only need to configure this section.

First, install the cluster manager:

<source lang="bash">
yum -y install cman
</source>

==== Setting Up /etc/cluster/cluster.conf ====

Once installed, we need to create and fill in the [[Two-Node Fedora 13 cluster.conf|/etc/cluster/cluster.conf]] file. This is the core configuration file of our cluster and is an [[XML]] formatted file.

'''Notice'''

This is, in many ways, the "core" of your cluster. This section is short, but expect to spend some time getting the right setup for your environment. Please pay careful attention to all options and thoroughly test your configuration before going into production!

* Please see: [[Two-Node Fedora 13 cluster.conf]] for a full explanation on this file and it's options.

<source lang="bash">
vim /etc/cluster/cluster.conf
</source>

The example in that article details many of the common options you will want to use. For a more complete list, please run man 5 cluster.conf for a more complete discussion on this file. Also, please be aware that some arguments can overlap between [[#Setup Corosync|corosync]] and this file.

Here is a real-world example configuration used in the development two-node AN!Cluster. This will likely not work for you, but should act as a good base to adapt and expand on.

<source lang="xml">
<?xml version="1.0"?>
<cluster name="an-cluster" config_version="1">
<cman two_node="1" expected_votes="1"/>
<totem secauth="off" rrp_mode="active"/>
<clusternodes>
<clusternode name="an-node01.alteeve.com" nodeid="1">
<fence>
<method name="node_assassin">
<device name="batou" port="01" action="reboot"/>
</method>
</fence>
</clusternode>
<clusternode name="an-node02.alteeve.com" nodeid="2">
<fence>
<method name="node_assassin">
<device name="batou" port="02" action="reboot"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="batou" agent="fence_na" ipaddr="batou.alteeve.com" login="username" passwd="secret" quiet="1"/>
</fencedevices>
</cluster>
</source>

Once your version of the cluster.conf is done, you need to verify it. Be sure to validate it against the cluster.rng validation file.

<source lang="bash">
xmllint --relaxng /usr/share/cluster/cluster.rng /etc/cluster/cluster.conf
</source>

If there are errors, fix them. Once it is formatted properly, the contents of you cluster.conf file will be returned followed by "/etc/cluster/cluster.conf validates". Once this is the case, you can move on.

'''Note''': Normally at this stage, you'd use chkconfig to enable cman to start at boot. However, depending on how you use your cluster, you may need to modify the boot order. For now, let's leave it off until we're got all the components enabled. This is particularly important if you will be using [[DRBD]], [[CLVM]] or [[Xen]].

<source lang="bash">
chkconfig cman off
</source>

To see if it's working though, have a terminal open to both nodes and manually start cman. I suggest have two terminal windows open on each node. In one, start tailing /var/log/messages:

<source lang="bash">
clear; tail -f -n 0 /var/log/messages
</source>

Now, with both log files being watched, start cman on both nodes.

'''Note''': You MUST execute the following on both nodes withing the time limit set by post_join_delay (default is 6 seconds). If you wait longer than this, the first node started will [[fence]] the other node, assuming you have fencing configured properly. If fencing is '''not''' configured properly, your first node will hang.

<source lang="bash">
/etc/init.d/cman start
</source>

Examine the log files to verify there were no errors. If there were, fix them. If there was not, fantastic!

'''Warning''': This last step will forcefully reboot your nodes. Do not do this on a production cluster! If it must be done on a production server, stop all non-essential services and be sure you have a good backup, ''just in case''.

Lastly, test that your fencing is configured and working properly. We do this by using a program called fence_node. With both nodes up and running cman, call a fence against the other node. Be sure to be watching the log files.

'''From''' an-node01.alteeve.com

<source lang="bash">
fence_node an-node02.alteeve.com
</source>
<source lang="bash">
fence an-node02.alteeve.com success
</source>

If it worked, when the node comes back up, repeat the process from an-node02.alteeve.com against an-node01.alteeve.com.

=== Setup Corosync ===

'''Note''': In our cluster, we will not need to configure Corosync.

When using cman, as we are here, Corosync's configuration file is ignored.

Most of the settings available in [[Two-Node Fedora 13 corosync.conf|corosync.conf]] can be found in the [[Two-Node Fedora 13 cluster.conf|cluster.conf]] file. Please see the cluster.conf article for a comprehensive list of what is and is not supported.

If you want to configure corosync and not use cman, please see the following article:

* [[Two-Node Fedora 13 corosync.conf]]

= Fencing =

Before proceeding with the cluster.conf file, you must understand what fencing is, how it is used in Red Hat/CentOS clusters and why it is so important.

* The Cluster Admin's Mantra:
** '''The only thing you don't know is what you don't know'''.

Just because one node loses communication with another node, it '''cannot''' assume that the silent node is dead!

== What is it? ==

"Fencing" is the act of isolating a malfunctioning node. The goal is to prevent a '''split-brain''' condition where two nodes think the other member is dead and continue to use a shared resource. When this happens, file system corruption is almost guaranteed. Another dangerous scenario would be if one node paused while writing to a disk, the other node decides it's dead and starts to replay the journal, then the first node recovers and completes the write. The results would be equally disastrous. If you are lucky enough to not lose the shared file system, you will be faced with the task of determining what data got written to which node, merging that data and/or overwriting the node you trust the least. This 'best case' is still pretty lousy.

Fencing, isolating a node from altering shared disks, can be accomplished in a couple of ways:

* Power
** Power fencing is where a device is used to cut the power to a malfunctioning node. This is probably the most common type.
* Blocking
** Blocking is often implemented at the network level. This type of fencing leaves the node alone, but disconnects it from the storage network. Often this is done by a switch which prevents traffic coming from the fenced node.

Those familiar with heartbeat clusters will know "fencing" by the HA Linux term "STONITH", literally, '''S'''hoot '''T'''he '''O'''ther '''N'''ode '''I'''n '''T'''he '''H'''ead.

== Misconception ==

It is a '''very''' common mistake to ignore fencing when first starting to learn about clustering. Often people think ''"It's just for production systems, I don't need to worry about it yet because I don't care what happens to my test cluster."''.

'''''Wrong!'''''

For the most practical reason; the cluster software will block all I/O transactions when it can't guarantee a fence operation succeeded. The result is that your cluster will essentially "lock up". Likewise, [[cman]] and related daemons will fail if they can't find a fence agent to use.

Secondly: testing our cluster will involve inducing errors. Without proper fencing, there is a high probability that our shared file system will be corrupted. That would force the need to start over, making your learning take a lot longer than it needs to.

== Fence Devices ==

Many major [[OEM]]s have their own remote management devices that can serve as fence devices. Examples are [http://dell.ca Dell]'s 'DRAC' (Dell Remote Access Controller), [http://hp.ca HP]'s iLO (Integrate Lights Out), [http://ibm.ca IBM]'s 'RSA' (Remote Supervisor Adapter), [http://sun.ca Sun]'s 'SSP' (System Service Processor) and so on. Smaller manufacturers implement remote management via [[IPMI]], Intelligent Power Management Interface.

In the above devices, fencing is implemented via a build in or integrated device inside the server. These devices are usually accessible even when the host server is powered off or hard locked. Via these devices, the host server can be powered off, reset and powered on remotely, regardless of the state of the host server.

Block fencing is possible when the device connecting a node to shared resources, like a fiber-channel SAN switch, provides a method of logically "unplugging" a defective node from the shared resource, leaving the node itself alone.

== Implementation ==

How you implement fencing depends largely on what kind of fence device(s) you have. This is where I need help from you, dear reader. I only have access to [[Node Assassin]] fence devices on my lab's cluster and [[IPMI]] at work. I will show examples below using these devices. If you have other fence devices, like [[IPMI]], addressable [[PDU]] or so on, please let me know so that I can expand this section.

=== Using Node Assassin To Demonstrate Fencing ===

In Red Hat's cluster software, the fence device(s) are configured in the main /etc/cluster.conf cluster configuration file. This configuration is then acted on via the fenced fence daemon.

When the cluster determines that a node needs to be fenced, the fenced daemon will consult the cluster.conf file for information on how to access the fence device. Given this cluster.conf snippet:

<source lang="xml">
<cluster name="an-cluster" config_version="1">
<clusternodes>
<clusternode name="an-node02.alteeve.com" nodeid="2">
<fence>
<method name="node_assassin">
<device name="motoko" port="02" action="off"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="motoko" agent="fence_na" quiet="true"
ipaddr="motoko.alteeve.com" login="motoko" passwd="secret">
</fencedevice>
</fencedevices>
</cluster>
</source>

If the cluster manager ([[corosync]], specifically) determines that the node an-node02.alteeve.com needs to be fenced, it looks at the first (and only, in this case) <fence> entry's name which is motoko in this case. It gathers the other variables and then looks in the <fencedevices> section for the device with the matching name. With these two sections, it now has all the variable-value pairs needed to pass to the fence agent script set in the agent variable.

So in this example, fenced looks up the details on the motoko Node Assassin fence device. It calls the fence_na program, called a fence agent, and passes the following arguments as a series of lines to the agent via [[STDIN]].
<source lang="bash">
ipaddr=motoko.alteeve.com
login=motoko
passwd=secret
quiet=true
port=2
action=off
</source>

How the fence agent acts on these arguments varies depending on the fence device itself. To continue using [[Node Assassin]] as an example, the fence_na fence agent will create a connection to the device at the IP address (or resolvable name, as in this case) specified in the ipaddr argument. Once connected, it will authenticate using the login and passwd arguments. Once authenticated, it tells the device what port to act on. Finally, it tells the device what action to take. Here, the action is off which, internally, translates as hitting the reset switch, then press and hold the power switch long enough to force a power down. As a last step, it checks to see if there is power still coming from the node to determine whether the fence succeeded or not. In other devices, the port could be a power jack, a network switch port and so on.

Once the device completes, it returns a success or failed message in the form of a prescribed exit code. If the first attempt fails, the fence agent will try the next <fence> method, if a second exists. It will keep trying fence devices in the order they are found in the cluster.conf file until it runs out of devices. If it fails to fence the node, most daemons will "block", that is, lock up and stop responding until the issue is resolved. The logic for this is that a locked up cluster is better than a corrupted one.

If any of the fence devices succeed though, the cluster will know that it is safe to proceed and will reconfigure the cluster without the defective node.

=== An Example IPMI Configuration ===

[[IPMI]] is perhaps the most common method used for fencing nodes in a cluster. It requires having a system board with an IPMI baseboard management controller (BMC). IPMI is often the underlying technology used by many OEM-branded remote access tools. If you don't see a specific fence agent for your server's remote access application, experiment with generic IPMI tools.

Given the ubiquity of IPMI, I'd like to take the time to walk through configuring and implementing IPMI in a cluster.

IPMI is generally configured using tools provided by the manufacturer of your server. Often there is a web interface with a default IP address, user name and password. In many cases though, you can also configure your IPMI device from the command line using user-space tools. The next section will walk you through an example setup of an IPMI device. If you've already configure yours though, note the IP addresses, user names and passwords that you assigned and skip down one section.

==== Configuring IPMI From The Command Line ====

To start, we need to install the IPMI user software.

<source lang="bash">
yum install ipmitool freeipmi.x86_64 freeipmi-bmc-watchdog.x86_64 freeipmi-ipmidetectd.x86_64 OpenIPMI.x86_64
</source>

Once installed, you should be able to check the local IPMI BMC using ipmitool once you're started the ipmi daemon.

<source lang="bash">
/etc/init.d/ipmi start
</source>
<source lang="text">
Starting ipmi drivers: [ OK ]
</source>
<source lang="bash">
ipmitool chassis status
</source>
<source lang="text">
System Power : on
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event : command
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
Front Panel Control : none
</source>

If you see that, you're doing well. You can now check the current configuration using the following command.

<source lang="bash">
ipmitool -I open lan print 1
</source>
<source lang="text">
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 OEM
Auth Type Enable : Callback : NONE MD2 MD5 OEM
: User : NONE MD2 MD5 OEM
: Operator : NONE MD2 MD5 OEM
: Admin : NONE MD2 MD5 OEM
: OEM :
IP Address Source : Static Address
IP Address : 10.255.128.1
Subnet Mask : 255.255.0.0
MAC Address : 00:e0:81:aa:bb:cc
SNMP Community String : AMI
IP Header : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control : ARP Responses Disabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 0.0 seconds
Default Gateway IP : 192.168.1.1
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12,0,0,0,0,0,0,0,0
Cipher Suite Priv Max : aaaaXXaaaXXaaXX
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
</source>

You can change the MAC address, but this isn't advised without a good reason to do so without good reason. That said, here is an example set of commands that will configure, save and then check that the settings took. Adapt the values to suit your environment and preferences.

<source lang="bash">
# Don't change the MAC without a good reason. If you need to though, this should work.
#ipmitool -I open lan set 1 macaddr 00:e0:81:aa:bb:cd

# Set the IP to be static (instead of DHCP)
ipmitool -I open lan set 1 ipsrc static

# Set the IP, default gateway and subnet mask address of the IPMI interface.
ipmitool -I open lan set 1 ipaddr 192.168.1.171
ipmitool -I open lan set 1 defgw ipaddr 192.168.1.1
ipmitool -I open lan set 1 netmask 255.255.255.0

# Set the password.
ipmitool -I open lan set 1 password secret
ipmitool -I open user set password 2 secret

# Set the snmp community string, if appropriate
ipmitool -I open lan set 1 snmp alteeve

# Enable access
ipmitool -I open lan set 1 access on

# Reset the IPMI BMC to make sure the changes took effect.
ipmitool mc reset cold

# Wait a few seconds and then re-run the call that dumped the setup to ensure
# it is now what we want.
sleep 5
ipmitool -I open lan print 1
</source>

If all went well, you should see the same output as above, but now with your new configuration.

==== Testing IPMI ====

If you skipped the previous step and just want the minimal setup that works, you only need to install one package.

<source lang="bash">
yum install ipmitool
</source>

Regardless of how you configured your IPMI devices, you will want to test now that you can check the power state of your nodes using the IPMI interface.

You can only query the status of the remote node(s) this way. You can't use the following command to check your local node.

In the example below, we will check the state of an-node01 from an-node02. Note that here we use the IP address directly, but in practice I like to use a name that resolves to the IP address of the '''IPMI''' interface (denoted by a .ipmi suffix).

<source lang="bash">
ipmitool -I lan -H 192.168.1.171 -U admin -P secret chassis power status
</source>
<source lang="text">
Chassis Power is on
</source>

You could replace status with on, off and cycle to remotely boot, power off and reboot (power cycle) the node. This is, in fact, what the fence_ipmilan fence agent does. If you can afford to stop your servers, it's a good idea to play with the various power options to see how the work. There are a few more option than what I mentioned here, which you can read about in man ipmitool.

Once you have completely tested your IPMI settings, enable the service to start on boot.

<source lang="bash">
chkconfig ipmi on
</source>

==== Configuring /etc/cluster.conf ====

Lastly, we need to add the IPMI configuration to our [[cluster.conf]] file. Note that, unlike with the [[#Using Node Assassin To Demonstrate Fencing |Node Assassin]] setup, there is no port value in device tag. This is because there is only device controlled by the IPMI BMC; The node itself. The other change is that there are now two <fencedevice ... /> entries; One for each node's IPMI device. Finally, pay attention to the <device name="..." field. This is an example of where it matters.

Here is an example...

<source lang="xml">
<cluster name="an-cluster" config_version="1">
<clusternodes>
<clusternode name="an-node01.alteeve.com" nodeid="1" votes="1">
<fence>
<method name="ipmi">
<device name="fence_an01" action="reboot" />
</method>
</fence>
</clusternode>
<clusternode name="an-node02.alteeve.com" nodeid="2" votes="1">
<fence>
<method name="ipmi">
<device name="fence_an02" action="reboot" />
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="fence_an01" agent="fence_ipmilan" ipaddr="192.168.3.61" login="admin" passwd="secret" />
<fencedevice name="fence_an02" agent="fence_ipmilan" ipaddr="192.168.3.62" login="admin" passwd="secret" />
</fencedevices>
</cluster>
</source>

=== Node Assassin ===

A cheap alternative is the [[Node Assassin]], an open-hardware, open source fence device. It was built to allow the use of commodity system boards that lacked remote management support found on more expensive, server class hardware.

'''Full Disclosure''': Node Assassin was created by me, with much help from others, for this paper.

= Clean Up =

Some daemons may have been installed or dragged in during the setup of the cluster. Most notable is heartbeat which is not compatible with [[RHCS]]. The following command will disable various daemons from starting at boot.

As for the rest of the daemons below, it is safe to run even when the daemons aren't installed or have already been removed. Of course, skip the ones you actually want to use. This assumes that the nodes have been built according to this HowTo.

<source lang="bash">
chkconfig heartbeat off; chkconfig iscsid off; chkconfig iptables off; chkconfig ip6tables off
</source>

= So You Have a Cluster - Now What? =

Building the cluster infrastructure was pretty easy, wasn't it?

But what will you '''''do''''' with it?

{|style="width: 75%; text-align: center; padding: 10px; border-spacing: 15px;"
|-
!style="width: 100%; border: 1px solid #dfdfdf;" colspan="3"|Choose Your Own Adventure Cluster
|-
|style="width: 33%; border: 1px solid #dfdfdf;"|  
|style="width: 34%; border: 1px solid #dfdfdf;"| [[Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM|Xen-Based Virtual Machine Host on DRBD+CLVM]] High Availability VM Host
|style="width: 33%; border: 1px solid #dfdfdf;"|  
|}

= Thanks =

* A '''huge''' thanks to [http://iplink.net Interlink Connectivity]! They hire me as a contractor and have allowed me to extend these docs while working on their clusters. Development of these How-Tos would be much slower if not for them. If you need hosting or colo services, drop them a line. Their website is a bit out of date though, so disregard the prices. :)
* To '''sdake''' of [http://corosync.org corosync] for helping me sort out the '''plock''' component and corosync in general.
* To '''Angus Salkeld''' for helping me nail down the Corosync and OpenAIS differences.
* To [https://lists.linux-foundation.org/pipermail/openais/2010-February/013922.html HJ Lee] from the OpenAIS list for helping me understand the mechanisms controlling the Redundant Ring Protocol's failure detection types.
* To [https://lists.linux-foundation.org/pipermail/openais/2010-February/013925.html Steven Dake] for clarifying the to_x vs. logoutput: x arguments in openais.conf.
* To [http://dk.linkedin.com/in/fabbione Fabio Massimo Di Nitto] for helping me get caught up with clustering and VMs.

{{footer}}

Abandoned - Two Node Fedora 13 Cluster

2010-10-12T20:09:12Z

SRSullivan: /* Platform */ Removed Erronious reference to CentOS 5.4 Disk Version, Does not apply to Fedora 13.

{{howto_header}}

'''''Notice''''', '''Sep. 08, 2010''' - ''Draft 1'': This is the first draft of this HowTo. It has not been peer-reviewed yet, so please proceed with caution and on a test cluster. Any and all feedback is much appreciated.

= Overview =

This paper has one goal;

# How to assemble the simplest cluster possible, a '''2-Node Cluster''', which you can then expand on for your own needs.

With this completed, you can then jump into "step 2" papers that will show various uses of a two node cluster:

# How to create a "floating" virtual machine that can move between the two nodes in the event of a node failure, maximizing up time.
# How to create simple resources that can move between nodes. Examples will be a simple PostgreSQL database, DHCP, DNS and web servers.

== Prerequisites ==

It is expected that you are already comfortable with the Linux command line, specifically [[bash]], and that you are familiar with general administrative tasks in Red Hat based distributions, specifically [[Fedora]]. You will also need to be comfortable using editors like [[vim]], [[nano]] or similar. This paper uses vim in examples. Simply substitute your favourite editor in it's place.

You are also expected to be comfortable with networking concepts. You will be expected to understand TCP/IP, [[multicast]], broadcast, subnets and netmasks, routing and other relatively basic networking concepts. Please take the time to become familiar with these concepts before proceeding.

This said, where feasible, as much detail as is possible will be provided. For example, all configuration file locations will be shown and functioning sample files will be provided.

== Platform ==

This paper will implement the [[Red Hat]] Cluster Suite using the Fedora v13 distribution. This paper uses the [[x86_64]] repositories, however, if you are on an [[i386]] (32 bit) system, you should be able to follow along fine. Simply replace x86_64 with i386 or i686 in package names.

You can either download the stock Fedora 13 DVD ISO, or you can try out the alpha [[#AN!Cluster Install|AN!Cluster Install]] DVD. (4.3GB iso). If you use the latter, please test it out on a development or test cluster. If you have any problems with the AN!Cluster variant Fedora distro, please [[Digimer|contact me]] and let me know what your trouble was.

== Why Fedora 13? ==

Generally speaking, I much prefer to use a server-oriented Linux distribution like [[CentOS]], [[Debian]] or similar. However, there have been many recent changes in the Linux-Clustering world that have made all of the currently available server-class distributions obsolete. With luck, once [[Red Hat]] Enterprise Linux and [[CentOS]] version 6 is released, this will change.

Until then, [[Fedora]] version 13 provides the most up to date binary releases of the new implementation of the clustering stack available. For this reason, FC13 is the best choice in clustering, if you want to be current. To mitigate some of the issues introduced by using a workstation distribution, many packages will be stripped out of the default install.

== Focus ==

Clusters can serve to solve three problems; '''Reliability''', '''Performance''' and '''Scalability'''.

This focus of the cluster described in this paper is primarily '''reliability'''. Second to this, '''scalability''' will be the priority leaving '''performance''' to be addressed only when it does not impact the first two criteria. This is not to indicate that performance is not a valid priority, it simply isn't the priority of this paper.

== Goal ==

At the end of this paper, you should have a fully functioning two-node array capable of hosting a "floating" resources. That is, resources that exist on one node and can be easily moved to the other node with minimal effort and down time. This should conclude with a solid foundation for adding more virtual servers up to the limit of your cluster's resources.

This paper should also serve to show how to build the foundation of any other cluster configuration. This paper has a core focus of introducing the main issues that come with clustering and hopes to serve as a foundation for any cluster configuration outside the scope of this paper.

= Begin =

Let's begin!

== Hardware ==

We will need two physical servers each with the following hardware:
* One or more multi-core [[CPU]]s with Virtualization support.
* Three network cards; At least one should be gigabit or faster.
* One or more hard drives.
* You will need some form of a [[fence|fence device]]. This can be an [[IPMI]]-enabled server, a [http://nodeassassin.org Node Assassin], a fenceable [http://www.apc.com/products/resource/include/techspec_index.cfm?base_sku=AP7900&tab=features PDU] or similar.

This paper uses the following hardware:
* ASUS [http://support.asus.com/search/search.aspx?keyword=m4a78l-m&SLanguage=en-us M4A78L-M]
* AMD Athlon II x2 250
* 2GB Kingston DDR2 KVR800D2N6K2/4G (4GB kit split between the two nodes)
* 1x Intel 82540 PCI NICs
* 1x D-Link DGE-560T
* Node Assassin

This is not an endorsement of the above hardware. I bought what was within my budget that would serve the purposes of creating this document. What you purchase shouldn't matter, so long as the minimum requirements are met.

== Pre-Assembly ==

With multiple NICs, it is quite likely that the mapping of physical devices to logical ethX devices may not be ideal. This is a particular issue if you decide to [[Setting Up a PXE Server in Fedora|network boot]] your install media. In that case, if the wrong NIC is chosen for eth0, you will be presented with a list of MAC addresses to attempt setup with.

Before you assemble your servers, record their network cards' [[MAC]] addresses. I like to keep simple text files like these:

<source lang="bash">
cat an-node01.mac
</source>
<source lang="text">
90:E6:BA:71:82:EA eth0 # Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
00:21:91:19:96:53 eth1 # D-Link System Inc DGE-560T PCI Express Gigabit Ethernet Adapter
00:0E:0C:59:46:E4 eth2 # Intel Corporation 82540EM Gigabit Ethernet Controller
</source>

<source lang="bash">
cat an-node02.mac
</source>
<source lang="text">
90:E6:BA:71:82:D8 eth0 # Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
00:21:91:19:96:5A eth1 # D-Link System Inc DGE-560T PCI Express Gigabit Ethernet Adapter
00:0E:0C:59:45:78 eth2 # Intel Corporation 82540EM Gigabit Ethernet Controller
</source>

Feel free to record the information in any way that suits you the best.

== OS Install ==

Start with a stock Fedora 13 install. This How-To uses Fedora 13 x86_64, however it should be fairly easy to adapt to other recent Fedora versions. This document is also attempting to be easily ported to [[CentOS]]/[[RHEL]] version 6 once it is released. This will not adapt well to CentOS/RHEL version 5 though... Much of the cluster stack has changed dramatically since it was initially released.

These are sample kickstart script used by this paper. Be sure to set your own password string and network settings.

'''''Warning'''''! These kickstart scripts '''''will erase your hard drive'''''! Adapt them, don't blindly use them.

Generic cluster node kickstart scripts.
* [[Fedora13 KS an-node01.ks|an-node01.ks]]
* [[Fedora13 KS an-node02.ks|an-node02.ks]]

== AN!Cluster Install ==

If you are feeling brave, below is a link to a custom install DVD that contains the kickstart scripts to setup nodes and an an-cluster directory with all the configuration files.

* Download the custom '''AN!Cluster v0.2.001''' Install DVD. (4.5[[GiB]] iso). (Currently disabled - Reworking for F13)

== Post OS Install ==

Once the OS is installed, we need to do a couple things;

# Disable selinux
# Setup networking.
# Change the default run-level.

=== Disable 'selinux' ===

To allow this paper to focus on clustering, we will disable selinux. Obviously, this introduces security issues that you may not be comfortable with. If you wish to leave selinux enabled, it will be up to you to sort out issues that crop up.

To disable selinux, edit /etc/selinux/config and change SELINUX=enforcing to SELINUX=permissive. You will need to reboot in order for the changes to take effect.

=== Change the Default Run-Level ===

'''This is an optional step'''. It improves performance only, it is not a required step.

If you don't plan to work on your nodes directly, it makes sense to switch the default run level from 5 to 3. This prevents the window manager, like Gnome or KDE, from starting at boot. This frees up a fair of memory and system resources and reduces the possible attack vectors.

To do this, edit /etc/inittab, change the id:5:initdefault: line to id:3:initdefault: and then switch to run level 3:

<source lang="bash">
vim /etc/inittab
</source>
<source lang="text">
id:3:initdefault:
</source>
<source lang="bash">
init 3
</source>

=== Setup Networking ===

We need to remove NetworkManager, enable network, configure the ifcfg-eth* files and then start the network daemon.

==== Managed Switch Warning ====

'''WARNING''': Please pay attention to this warning. The vast majority of cluster problems end up being network related. The hardest ones to diagnose are usually multicast issues.

If you use a managed switch, be careful about enabling [[Multicast IGMP Snooping]] or [[Spanning Tree Protocol]]! They have been known to cause problems by not allowing multicast packets to reach all nodes. This can cause somewhat random break-downs in communication between your nodes, leading to fences and DLM lock timeouts.

'''Ensure the [[PIM Routing]] is setup and working.

If you have problems with your cluster not forming, or seemingly random fencing, try using an unmanaged switch or use a cross-over cable. If the problem goes away, you are most likely dealing with a managed switch configuration problem.

==== Network Layout ====

This setup expects you to have three physical network cards connected to three independent networks. To have a common vernacular, lets use this table to describe them:

{|
!class="cell_all"|Network Description
!class="cell_tbr"|Short Name
!class="cell_tbr"|Device Name
!class="cell_tbr"|Suggested Subnet
!class="cell_tbr"|NIC Properties
|-
|class="cell_blr"|Back-Channel Network
|class="cell_br"|BCN
|class="cell_br"|eth0
|class="cell_br"|192.168.1.0/24
|class="cell_br"|NICs with [[IPMI]] piggy-back '''must''' be used here. Second-fastest NIC should be used here.
|-
|class="cell_blr"|Storage Network
|class="cell_br"|SN
|class="cell_br"|eth1
|class="cell_br"|192.168.2.0/24
|class="cell_br"|Fastest NIC should be used here.
|-
|class="cell_blr"|Internet-Facing Network
|class="cell_br"|IFN
|class="cell_br"|eth2
|class="cell_br"|192.168.3.0/24
|class="cell_br"|Remaining NIC should be used here. If using a [[Setting Up a PXE Server in Fedora|PXE server]], this should be a bootable NIC.
|}

Take note of these concerns when planning which NIC to use on each subnet. These issues are presented in the order that they must be addressed in:
# If your nodes have [[IPMI]] piggy-backing on a normal NIC, that NIC '''''must''''' be used on BCN subnet. Having your fence device accessible on a subnet that can be remotely accessed can pose a ''major'' security risk.
# The fastest NIC should be used for your SN subnet. Be sure to know which NICs support the largest jumbo frames when considering this.
# If you still have two NICs to choose from, use the fastest remaining NIC for your BCN subnet. This will minimize the time it takes to perform tasks like hot-migration of live virtual machines.
# The final NIC should be used for the IFN subnet.

==== Node IP Addresses ====

Obviously, the IP addresses you give to your nodes should be ones that suit you best. In this example, the following IP addresses are used:

{|
|class="cell_br"| 
|class="cell_tbr"|'''Internet-Facing Network''' (IFN)
|class="cell_tbr"|'''Storage Network''' (SN)
|class="cell_tbr"|'''Back-Channel Network''' (BCN)
|-
|class="cell_blr"|'''an-node01'''
|class="cell_br"|192.168.1.71
|class="cell_br"|192.168.2.71
|class="cell_br"|192.168.3.71
|-
|class="cell_blr"|'''an-node02'''
|class="cell_br"|192.168.1.72
|class="cell_br"|192.168.2.72
|class="cell_br"|192.168.3.72
|}

==== Remove NetworkManager ====

Some cluster software '''will not''' start with NetworkManager installed. This is because it is designed to be a highly-adaptive network system that can accommodate frequent changes in the network. To simplify these network transitions for end-users, a lot of reconfiguration of the network is done behind the scenes.

For workstations, this is wonderful. For clustering, this can be disastrous. Transient network outages are already a risk to a cluster's stability!

So first up, make sure that NetworkManager is completely removed from your system. If you used the [[#OS Install|kickstart]] scripts, then it was not installed. Otherwise, run:

<source lang="bash">
yum remove NetworkManager NetworkManager-gnome NetworkManager-openconnect NetworkManager-openvpn NetworkManager-pptp NetworkManager-vpnc cnetworkmanager knetworkmanager knetworkmanager-openvpn knetworkmanager-pptp knetworkmanager-vpnc libproxy-networkmanager yum-NetworkManager-dispatcher
</source>

==== Setup 'network' ====

Before proceeding with network configuration, check to see if your network cards are aligned to the proper ethX network names. If they need to be adjusted, please follow this How-To before proceeding:

* [[Changing the ethX to Ethernet Device Mapping in Fedora]]

There are a few ways to configure network in Fedora:
* system-config-network (graphical)
* system-config-network-tui (ncurses)
* Directly editing the /etc/sysconfig/network-scripts/ifcfg-eth* files. (See: [http://docs.fedoraproject.org/en-US/Fedora/12/html/Deployment_Guide/s1-networkscripts-interfaces.html here] for a full list of options)

Do not proceed until your node's networking is fully configured.

==== Update the Hosts File ====

Some applications expect to be able to call nodes by their name. To accommodate this, and to ensure that inter-node communication takes place on the back-channel subnet, we remove any existing hostname entries and then add the following to the /etc/hosts file:

'''Note''': Any pre-existing entries matching the name returned by uname -n must be removed from /etc/hosts. There is a good chance there will be an entry that resolves to 127.0.0.1 which would cause problems later.

Obviously, adapt the names and IPs to match your nodes and subnets. The only critical thing is to make sure that the name returned by uname -n is resolvable to the back-channel subnet. I like to add a short-form name for convenience.

The updated /etc/hosts file should look something like this:

<source lang="bash">
vim /etc/hosts
</source>
<source lang="text">
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

# Internet Facing Network
192.168.1.71 an-node01 an-node01.alteeve.com an-node01.bcn
192.168.1.72 an-node02 an-node02.alteeve.com an-node02.bcn

# Storage Network
192.168.2.71 an-node01.sn
192.168.2.72 an-node02.sn

# Back Channel Network
192.168.3.71 an-node01.ifn
192.168.3.72 an-node02.ifn

# Node Assassins
192.168.3.61 batou batou.alteeve.com
192.168.3.62 motoko motoko.alteeve.com
</source>

'''Note''': I use Node Assassins. If you use IPMI or other fence devices, alter the entries as appropriate.

Now to test this, ping both nodes by their name, as returned by uname -n, and make sure the ping packets are sent on the back channel network (192.168.1.0/24).

<source lang="bash">
ping -c 5 an-node01.alteeve.com
</source>
<source lang="text">
PING an-node01 (192.168.1.71) 56(84) bytes of data.
64 bytes from an-node01 (192.168.1.71): icmp_seq=1 ttl=64 time=0.399 ms
64 bytes from an-node01 (192.168.1.71): icmp_seq=2 ttl=64 time=0.403 ms
64 bytes from an-node01 (192.168.1.71): icmp_seq=3 ttl=64 time=0.413 ms
64 bytes from an-node01 (192.168.1.71): icmp_seq=4 ttl=64 time=0.365 ms
64 bytes from an-node01 (192.168.1.71): icmp_seq=5 ttl=64 time=0.428 ms

--- an-node01 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.365/0.401/0.428/0.030 ms
</source>
<source lang="bash">
ping -c 5 an-node02.alteeve.com
</source>
<source lang="text">
PING an-node02 (192.168.1.72) 56(84) bytes of data.
64 bytes from an-node02 (192.168.1.72): icmp_seq=1 ttl=64 time=0.419 ms
64 bytes from an-node02 (192.168.1.72): icmp_seq=2 ttl=64 time=0.405 ms
64 bytes from an-node02 (192.168.1.72): icmp_seq=3 ttl=64 time=0.416 ms
64 bytes from an-node02 (192.168.1.72): icmp_seq=4 ttl=64 time=0.373 ms
64 bytes from an-node02 (192.168.1.72): icmp_seq=5 ttl=64 time=0.396 ms

--- an-node02 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.373/0.401/0.419/0.030 ms
</source>

Be sure that, if your [[fence]] device uses a name, that you include entries to resolve it as well. You can see how I've done this with the two [[Node Assassin]] devices I use. The same applies to [[IPMI]] or other devices, if you plan to reference them by name.

Fencing will be discussed in more detail later on in this HowTo.

==== Disable Firewalls ====

Be sure to flush netfilter tables and disable iptables and ip6tables from starting on our nodes.

There will be enough potential sources of problem as it is. Disabling firewalls at this stage will minimize the chance of an errant iptables rule messing up our configuration. If, before launch, you wish to implement a firewall, feel free to do so but be sure to thoroughly test your cluster to ensure no problems were introduced.

<source lang="bash">
chkconfig --level 2345 iptables off
/etc/init.d/iptables stop
chkconfig --level 2345 ip6tables off
/etc/init.d/ip6tables stop
</source>

==== Setup SSH Shared Keys ====

'''This is an optional step'''. Setting up shared keys will allow your nodes to pass files between one another and execute commands remotely without needing to enter a password. This is obviously somewhat risky from a security point of view. As such, it is up to you whether you do this or not. This is not meant to be a security-focused How-To, so please independently study the risks.

If you're a little new to [[SSH]], it can be a bit confusing keeping connections straight in you head. When you connect to a remote machine, you start the connection on your machine as the user you are logged in as. This is the source user. When you call the remote machine, you tell the machine what user you want to log in as. This is the remote user.

You will need to create an SSH key for each source user, and then you will need to copy the newly generated public key to each remote machine's user directory that you want to connect to. In this example, we want to connect to either node, from either node, as the root user. So we will create a key for each node's root user and then copy the generated public key to the ''other'' node's root user's directory.

For each user, on each machine you want to connect '''from''', run:

<source lang="bash">
# The '2047' is just to screw with brute-forces a bit. :)
ssh-keygen -t rsa -N "" -b 2047 -f ~/.ssh/id_rsa
</source>
<source lang="text">
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
08:d8:ed:72:38:61:c5:0e:cf:bf:dc:28:e5:3c:a7:88 root@an-node01.alteeve.com
The key's randomart image is:
+--[ RSA 2047]----+
| .. |
| o.o. |
| . ==. |
| . =+. |
| + +.S |
| + o |
| = + |
| ...B o |
| E ...+ |
+-----------------+
</source>

This will create two files: the private key called ~/.ssh/id_dsa and the public key called ~/.ssh/id_dsa.pub. The private '''''must never''''' be group or world readable! That is, it should be set to mode 0600.

The two files should look like:

'''Private key''':
<source lang="bash">
cat ~/.ssh/id_rsa
</source>
<source lang="text">
-----BEGIN RSA PRIVATE KEY-----
MIIEoQIBAAKCAQBlL42DC+NJVpJ0rdrWQ1rxGEbPrDoe8j8+RQx3QYiB014R7jY5
EaTenThxG/cudgbLluFxq6Merfl9Tq2It3k9Koq9nV9ZC/vXBcl4MC7pGSQaUw2h
DVwI7OCSWtnS+awR/1d93tANXRwy7K5ic1pcviJeN66dPuuPqJEF/SKE7yEBapMq
sN28G4IiLdimsV+UYXPQLOiMy5stmyGhFhQGH3kxYzJPOgiwZEFPZyXinGVoV+qa
9ERSjSKAL+g21zbYB/XFK9jLNSJqDIPa//wz0T+73agZ0zNlxygmXcJvapEsFGDG
O6tcy/3XlatSxjEZvvfdOnC310gJVp0bcyWDAgMBAAECggEAMZd0y91vr+n2Laln
r8ujLravPekzMyeXR3Wf/nLn7HkjibYubRnwrApyNz11kBfYjL+ODqAIemjZ9kgx
VOhXS1smVHhk2se8zk3PyFAVLblcsGo0K9LYYKd4CULtrzEe3FNBFje10FbqEytc
7HOMvheR0IuJ0Reda/M54K2H1Y6VemtMbT+aTcgxOSOgflkjCTAeeOajqP5r0TRg
1tY6/k46hLiBka9Oaj+QHHoWp+aQkb+ReHUBcUihnz3jcw2u8HYrQIO4+v4Ud2kr
C9QHPW907ykQTMAzhMvZ3DIOcqTzA0r857ps6FANTM87tqpse5h2KfdIjc0Ok/AY
eKgYAQKBgQDm/P0RygIJl6szVhOb5EsQU0sBUoMT3oZKmPcjHSsyVFPuEDoq1FG7
uZYMESkVVSYKvv5hTkRuVOqNE/EKtk5bwu4mM0S3qJo99cLREKB6zNdBp9z2ACDn
0XIIFIalXAPwYpoFYi1YfG8tFfSDvinLI6JLDT003N47qW1cC5rmgQKBgHAkbfX9
8u3LiT8JqCf1I+xoBTwH64grq/7HQ+PmwRqId+HyyDCm9Y/mkAW1hYQB+cL4y3OO
kGL60CZJ4eFiTYrSfmVa0lTbAlEfcORK/HXZkLRRW03iuwdAbZ7DIMzTvY2HgFlU
L1CfemtmzEC4E6t5/nA4Ytk9kPSlzbzxfXIDAoGAY/WtaqpZ0V7iRpgEal0UIt94
wPy9HrcYtGWX5Yk07VXS8F3zXh99s1hv148BkWrEyLe4i9F8CacTzbOIh1M3e7xS
pRNgtH3xKckV4rVoTVwh9xa2p3qMwuU/jMGdNygnyDpTXusKppVK417x7qU3nuIv
1HzJNPwz6+u5GLEo+oECgYAs++AEKj81dkzytXv3s1UasstOvlqTv/j5dZNdKyZQ
72cvgsUdBwxAEhu5vov1XRmERWrPSuPOYI/4m/B5CYbTZgZ/v8PZeBTg17zgRtgo
qgJq4qu+fXHKweR3KAzTPSivSiiJLMTiEWb5CD5sw6pYQdJ3z5aPUCwChzQVU8Wf
YwKBgQCvoYG7gwx/KGn5zm5tDpeWb3GBJdCeZDaj1ulcnHR0wcuBlxkw/TcIadZ3
kqIHlkjll5qk5EiNGNlnpHjEU9X67OKk211QDiNkg3KAIDMKBltE2AHe8DhFsV8a
Mc/t6vHYZ632hZ7b0WNuudB4GHJShOumXD+NfJgzxqKJyfGkpQ==
-----END RSA PRIVATE KEY-----
</source>

'''Public key''':
<source lang="bash">
cat ~/.ssh/id_rsa.pub
</source>
<source lang="text">
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAGUvjYML40lWknSt2tZDWvEYRs+sOh7yPz5FDHdBiIHTXhHuNjkRpN6dOHEb9y52BsuW4XGrox6t+X1OrYi3eT0qir2dX1kL+9cFyXgwLukZJBpTDaENXAjs4JJa2dL5rBH/V33e0A1dHDLsrmJzWly+Il43rp0+64+okQX9IoTvIQFqkyqw3bwbgiIt2KaxX5Rhc9As6IzLmy2bIaEWFAYfeTFjMk86CLBkQU9nJeKcZWhX6pr0RFKNIoAv6DbXNtgH9cUr2Ms1ImoMg9r//DPRP7vdqBnTM2XHKCZdwm9qkSwUYMY7q1zL/deVq1LGMRm+9906cLfXSAlWnRtzJYM= root@an-node01.alteeve.com
</source>

Copy the public key and then ssh normally into the remote machine as the root user. Create a file called ~/.ssh/authorized_keys and paste in the key.

From '''an-node01''', type:

<source lang="bash">
ssh root@an-node02
</source>
<source lang="text">
The authenticity of host 'an-node02 (192.168.1.72)' can't be established.
RSA key fingerprint is d4:1b:68:5f:fa:ef:0f:0c:16:e7:f9:0a:8d:69:3e:c5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'an-node02,192.168.1.72' (RSA) to the list of known hosts.
root@an-node02's password:
Last login: Fri Oct 1 20:07:01 2010 from 192.168.1.102
</source>

You will now be logged into an-node02 as the root user. Create the ~/.ssh/authorized_keys file and paste into it the public key from an-node01. If the remote machine's user hasn't used ssh yet, their ~/.ssh directory will not exist.

<source lang="bash">
cat ~/.ssh/authorized_keys
</source>
<source lang="text">
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAGUvjYML40lWknSt2tZDWvEYRs+sOh7yPz5FDHdBiIHTXhHuNjkRpN6dOHEb9y52BsuW4XGrox6t+X1OrYi3eT0qir2dX1kL+9cFyXgwLukZJBpTDaENXAjs4JJa2dL5rBH/V33e0A1dHDLsrmJzWly+Il43rp0+64+okQX9IoTvIQFqkyqw3bwbgiIt2KaxX5Rhc9As6IzLmy2bIaEWFAYfeTFjMk86CLBkQU9nJeKcZWhX6pr0RFKNIoAv6DbXNtgH9cUr2Ms1ImoMg9r//DPRP7vdqBnTM2XHKCZdwm9qkSwUYMY7q1zL/deVq1LGMRm+9906cLfXSAlWnRtzJYM= root@an-node01.alteeve.com
</source>

Now log out and then log back into the remote machine. This time, the connection should succeed without having entered a password!

= Initial Cluster Setup =

Before we get into specifics, let's take a minute to talk about the major components used in our cluster.

== Core Program Overviews ==

These are the core programs that may be new to you that we will use to build our cluster. Before we configure them, let's take a minute to understand their roles.

=== Cluster Manager ===

The cluster manager, cman, takes the configuration file [[Two-Node Fedora 13 cluster.conf|/etc/cluster/cluster.conf]] configuration file and uses it to configure and start the various cluster elements. This is where nodes are defined, fence devices are configured and various cluster tolerances are set.

=== Corosync ===

[http://www.corosync.org Corosync] is, essentially, the clustering kernel which provides the essential services for cluster-aware applications and other cluster services. At it's core is the [[totem]] protocol which provides cluster membership and ordered messaging. On top of this, it implements a number of core clustering services like [[#cpg; Closed Process Group|closed process group messaging]], [[quorum]] and [[confdb]] (an object database).

It's goal is to provide a substantially simpler and more flexible set of APIs to facilitate clustering in Linux. It manages which nodes are in the cluster, it triggers error messages when something fails, manages cluster locking and so on. Most other clustered applications rely on corosync to know when something has happened or to announce when a cluster-related action has taken place.

Please note that the corosync_overview man page is considered out of date at the time of this writing.

==== cpg; Closed Process Group ====

'''Note''': Only install the following if you wish to review the man page.

<source lang="bash">
yum install corosynclib-devel
man cpg_overview
</source>

The closed process group is a small process layer on top of the totem protocol provided by corosync. It handles the sending and delivery of messages amongst nodes in a consistent order. It adds [[PID]]s and group names to the membership layer. Only members of the group get the messages, thus it is a "closed" group.

=== OpenAIS ===

'''Note''': We will not use OpenAIS. It is included in here to explain it's role to folks coming from the Cluster stable 2 environment.

OpenAIS is now an extension to Corosync that adds an open-source implementation of the [http://www.saforum.org/ Service Availability (SA) Forum's] 'Application Interface Specification' ([[AIS]]). It is an [[API]] and policy designed to be used by applications concerned with maintaining services during faults. AIS implements the 'Availability Management Framework' ([[AMF]]) which, in turn, provides for application fail over, cluster management ([[CLM]]), Checkpointing ([[CKPT]]), Eventing ([[EVT]]), Messaging ([[MSG]]), and Distributed Locking ([[DLOCK]]).

It does this by implementing pluggable shared libraries that get loaded into corosync at runtime. These libraries then can make use of corosync's internal API. Both corosync and openais services provide [[IPC]] interfaces so that application programmers can make use of their behaviour. In short; applications can use OpenAIS to be cluster-aware. It's libraries are used by some applications, including Pacemaker. In our application, we will only be using it's libraries indirectly.

Please note that the openais_overview man page is considered out of date at the time of this writing.

'''Note''': Only install the following if you wish to review the man page.

<source lang="bash">
yum install openais
</source>

== Setup Core Programs ==

We now need to edit and create the configuration files for our cluster.

=== Setup cman ===

You may only need to configure this section.

First, install the cluster manager:

<source lang="bash">
yum -y install cman
</source>

==== Setting Up /etc/cluster/cluster.conf ====

Once installed, we need to create and fill in the [[Two-Node Fedora 13 cluster.conf|/etc/cluster/cluster.conf]] file. This is the core configuration file of our cluster and is an [[XML]] formatted file.

'''Notice'''

This is, in many ways, the "core" of your cluster. This section is short, but expect to spend some time getting the right setup for your environment. Please pay careful attention to all options and thoroughly test your configuration before going into production!

* Please see: [[Two-Node Fedora 13 cluster.conf]] for a full explanation on this file and it's options.

<source lang="bash">
vim /etc/cluster/cluster.conf
</source>

The example in that article details many of the common options you will want to use. For a more complete list, please run man 5 cluster.conf for a more complete discussion on this file. Also, please be aware that some arguments can overlap between [[#Setup Corosync|corosync]] and this file.

Here is a real-world example configuration used in the development two-node AN!Cluster. This will likely not work for you, but should act as a good base to adapt and expand on.

<source lang="xml">
<?xml version="1.0"?>
<cluster name="an-cluster" config_version="1">
<cman two_node="1" expected_votes="1"/>
<totem secauth="off" rrp_mode="active"/>
<clusternodes>
<clusternode name="an-node01.alteeve.com" nodeid="1">
<fence>
<method name="node_assassin">
<device name="batou" port="01" action="reboot"/>
</method>
</fence>
</clusternode>
<clusternode name="an-node02.alteeve.com" nodeid="2">
<fence>
<method name="node_assassin">
<device name="batou" port="02" action="reboot"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="batou" agent="fence_na" ipaddr="batou.alteeve.com" login="username" passwd="secret" quiet="1"/>
</fencedevices>
</cluster>
</source>

Once your version of the cluster.conf is done, you need to verify it. Be sure to validate it against the cluster.rng validation file.

<source lang="bash">
xmllint --relaxng /usr/share/cluster/cluster.rng /etc/cluster/cluster.conf
</source>

If there are errors, fix them. Once it is formatted properly, the contents of you cluster.conf file will be returned followed by "/etc/cluster/cluster.conf validates". Once this is the case, you can move on.

'''Note''': Normally at this stage, you'd use chkconfig to enable cman to start at boot. However, depending on how you use your cluster, you may need to modify the boot order. For now, let's leave it off until we're got all the components enabled. This is particularly important if you will be using [[DRBD]], [[CLVM]] or [[Xen]].

<source lang="bash">
chkconfig cman off
</source>

To see if it's working though, have a terminal open to both nodes and manually start cman. I suggest have two terminal windows open on each node. In one, start tailing /var/log/messages:

<source lang="bash">
clear; tail -f -n 0 /var/log/messages
</source>

Now, with both log files being watched, start cman on both nodes.

'''Note''': You MUST execute the following on both nodes withing the time limit set by post_join_delay (default is 6 seconds). If you wait longer than this, the first node started will [[fence]] the other node, assuming you have fencing configured properly. If fencing is '''not''' configured properly, your first node will hang.

<source lang="bash">
/etc/init.d/cman start
</source>

Examine the log files to verify there were no errors. If there were, fix them. If there was not, fantastic!

'''Warning''': This last step will forcefully reboot your nodes. Do not do this on a production cluster! If it must be done on a production server, stop all non-essential services and be sure you have a good backup, ''just in case''.

Lastly, test that your fencing is configured and working properly. We do this by using a program called fence_node. With both nodes up and running cman, call a fence against the other node. Be sure to be watching the log files.

'''From''' an-node01.alteeve.com

<source lang="bash">
fence_node an-node02.alteeve.com
</source>
<source lang="bash">
fence an-node02.alteeve.com success
</source>

If it worked, when the node comes back up, repeat the process from an-node02.alteeve.com against an-node01.alteeve.com.

=== Setup Corosync ===

'''Note''': In our cluster, we will not need to configure Corosync.

When using cman, as we are here, Corosync's configuration file is ignored.

Most of the settings available in [[Two-Node Fedora 13 corosync.conf|corosync.conf]] can be found in the [[Two-Node Fedora 13 cluster.conf|cluster.conf]] file. Please see the cluster.conf article for a comprehensive list of what is and is not supported.

If you want to configure corosync and not use cman, please see the following article:

* [[Two-Node Fedora 13 corosync.conf]]

= Fencing =

Before proceeding with the cluster.conf file, you must understand what fencing is, how it is used in Red Hat/CentOS clusters and why it is so important.

* The Cluster Admin's Mantra:
** '''The only thing you don't know is what you don't know'''.

Just because one node loses communication with another node, it '''cannot''' assume that the silent node is dead!

== What is it? ==

"Fencing" is the act of isolating a malfunctioning node. The goal is to prevent a '''split-brain''' condition where two nodes think the other member is dead and continue to use a shared resource. When this happens, file system corruption is almost guaranteed. Another dangerous scenario would be if one node paused while writing to a disk, the other node decides it's dead and starts to replay the journal, then the first node recovers and completes the write. The results would be equally disastrous. If you are lucky enough to not lose the shared file system, you will be faced with the task of determining what data got written to which node, merging that data and/or overwriting the node you trust the least. This 'best case' is still pretty lousy.

Fencing, isolating a node from altering shared disks, can be accomplished in a couple of ways:

* Power
** Power fencing is where a device is used to cut the power to a malfunctioning node. This is probably the most common type.
* Blocking
** Blocking is often implemented at the network level. This type of fencing leaves the node alone, but disconnects it from the storage network. Often this is done by a switch which prevents traffic coming from the fenced node.

Those familiar with heartbeat clusters will know "fencing" by the HA Linux term "STONITH", literally, '''S'''hoot '''T'''he '''O'''ther '''N'''ode '''I'''n '''T'''he '''H'''ead.

== Misconception ==

It is a '''very''' common mistake to ignore fencing when first starting to learn about clustering. Often people think ''"It's just for production systems, I don't need to worry about it yet because I don't care what happens to my test cluster."''.

'''''Wrong!'''''

For the most practical reason; the cluster software will block all I/O transactions when it can't guarantee a fence operation succeeded. The result is that your cluster will essentially "lock up". Likewise, [[cman]] and related daemons will fail if they can't find a fence agent to use.

Secondly: testing our cluster will involve inducing errors. Without proper fencing, there is a high probability that our shared file system will be corrupted. That would force the need to start over, making your learning take a lot longer than it needs to.

== Fence Devices ==

Many major [[OEM]]s have their own remote management devices that can serve as fence devices. Examples are [http://dell.ca Dell]'s 'DRAC' (Dell Remote Access Controller), [http://hp.ca HP]'s iLO (Integrate Lights Out), [http://ibm.ca IBM]'s 'RSA' (Remote Supervisor Adapter), [http://sun.ca Sun]'s 'SSP' (System Service Processor) and so on. Smaller manufacturers implement remote management via [[IPMI]], Intelligent Power Management Interface.

In the above devices, fencing is implemented via a build in or integrated device inside the server. These devices are usually accessible even when the host server is powered off or hard locked. Via these devices, the host server can be powered off, reset and powered on remotely, regardless of the state of the host server.

Block fencing is possible when the device connecting a node to shared resources, like a fiber-channel SAN switch, provides a method of logically "unplugging" a defective node from the shared resource, leaving the node itself alone.

== Implementation ==

How you implement fencing depends largely on what kind of fence device(s) you have. This is where I need help from you, dear reader. I only have access to [[Node Assassin]] fence devices on my lab's cluster and [[IPMI]] at work. I will show examples below using these devices. If you have other fence devices, like [[IPMI]], addressable [[PDU]] or so on, please let me know so that I can expand this section.

=== Using Node Assassin To Demonstrate Fencing ===

In Red Hat's cluster software, the fence device(s) are configured in the main /etc/cluster.conf cluster configuration file. This configuration is then acted on via the fenced fence daemon.

When the cluster determines that a node needs to be fenced, the fenced daemon will consult the cluster.conf file for information on how to access the fence device. Given this cluster.conf snippet:

<source lang="xml">
<cluster name="an-cluster" config_version="1">
<clusternodes>
<clusternode name="an-node02.alteeve.com" nodeid="2">
<fence>
<method name="node_assassin">
<device name="motoko" port="02" action="off"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="motoko" agent="fence_na" quiet="true"
ipaddr="motoko.alteeve.com" login="motoko" passwd="secret">
</fencedevice>
</fencedevices>
</cluster>
</source>

If the cluster manager ([[corosync]], specifically) determines that the node an-node02.alteeve.com needs to be fenced, it looks at the first (and only, in this case) <fence> entry's name which is motoko in this case. It gathers the other variables and then looks in the <fencedevices> section for the device with the matching name. With these two sections, it now has all the variable-value pairs needed to pass to the fence agent script set in the agent variable.

So in this example, fenced looks up the details on the motoko Node Assassin fence device. It calls the fence_na program, called a fence agent, and passes the following arguments as a series of lines to the agent via [[STDIN]].
<source lang="bash">
ipaddr=motoko.alteeve.com
login=motoko
passwd=secret
quiet=true
port=2
action=off
</source>

How the fence agent acts on these arguments varies depending on the fence device itself. To continue using [[Node Assassin]] as an example, the fence_na fence agent will create a connection to the device at the IP address (or resolvable name, as in this case) specified in the ipaddr argument. Once connected, it will authenticate using the login and passwd arguments. Once authenticated, it tells the device what port to act on. Finally, it tells the device what action to take. Here, the action is off which, internally, translates as hitting the reset switch, then press and hold the power switch long enough to force a power down. As a last step, it checks to see if there is power still coming from the node to determine whether the fence succeeded or not. In other devices, the port could be a power jack, a network switch port and so on.

Once the device completes, it returns a success or failed message in the form of a prescribed exit code. If the first attempt fails, the fence agent will try the next <fence> method, if a second exists. It will keep trying fence devices in the order they are found in the cluster.conf file until it runs out of devices. If it fails to fence the node, most daemons will "block", that is, lock up and stop responding until the issue is resolved. The logic for this is that a locked up cluster is better than a corrupted one.

If any of the fence devices succeed though, the cluster will know that it is safe to proceed and will reconfigure the cluster without the defective node.

=== An Example IPMI Configuration ===

[[IPMI]] is perhaps the most common method used for fencing nodes in a cluster. It requires having a system board with an IPMI baseboard management controller (BMC). IPMI is often the underlying technology used by many OEM-branded remote access tools. If you don't see a specific fence agent for your server's remote access application, experiment with generic IPMI tools.

Given the ubiquity of IPMI, I'd like to take the time to walk through configuring and implementing IPMI in a cluster.

IPMI is generally configured using tools provided by the manufacturer of your server. Often there is a web interface with a default IP address, user name and password. In many cases though, you can also configure your IPMI device from the command line using user-space tools. The next section will walk you through an example setup of an IPMI device. If you've already configure yours though, note the IP addresses, user names and passwords that you assigned and skip down one section.

==== Configuring IPMI From The Command Line ====

To start, we need to install the IPMI user software.

<source lang="bash">
yum install ipmitool freeipmi.x86_64 freeipmi-bmc-watchdog.x86_64 freeipmi-ipmidetectd.x86_64 OpenIPMI.x86_64
</source>

Once installed, you should be able to check the local IPMI BMC using ipmitool once you're started the ipmi daemon.

<source lang="bash">
/etc/init.d/ipmi start
</source>
<source lang="text">
Starting ipmi drivers: [ OK ]
</source>
<source lang="bash">
ipmitool chassis status
</source>
<source lang="text">
System Power : on
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event : command
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
Front Panel Control : none
</source>

If you see that, you're doing well. You can now check the current configuration using the following command.

<source lang="bash">
ipmitool -I open lan print 1
</source>
<source lang="text">
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 OEM
Auth Type Enable : Callback : NONE MD2 MD5 OEM
: User : NONE MD2 MD5 OEM
: Operator : NONE MD2 MD5 OEM
: Admin : NONE MD2 MD5 OEM
: OEM :
IP Address Source : Static Address
IP Address : 10.255.128.1
Subnet Mask : 255.255.0.0
MAC Address : 00:e0:81:aa:bb:cc
SNMP Community String : AMI
IP Header : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control : ARP Responses Disabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 0.0 seconds
Default Gateway IP : 192.168.1.1
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12,0,0,0,0,0,0,0,0
Cipher Suite Priv Max : aaaaXXaaaXXaaXX
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
</source>

You can change the MAC address, but this isn't advised without a good reason to do so without good reason. That said, here is an example set of commands that will configure, save and then check that the settings took. Adapt the values to suit your environment and preferences.

<source lang="bash">
# Don't change the MAC without a good reason. If you need to though, this should work.
#ipmitool -I open lan set 1 macaddr 00:e0:81:aa:bb:cd

# Set the IP to be static (instead of DHCP)
ipmitool -I open lan set 1 ipsrc static

# Set the IP, default gateway and subnet mask address of the IPMI interface.
ipmitool -I open lan set 1 ipaddr 192.168.1.171
ipmitool -I open lan set 1 defgw ipaddr 192.168.1.1
ipmitool -I open lan set 1 netmask 255.255.255.0

# Set the password.
ipmitool -I open lan set 1 password secret
ipmitool -I open user set password 2 secret

# Set the snmp community string, if appropriate
ipmitool -I open lan set 1 snmp alteeve

# Enable access
ipmitool -I open lan set 1 access on

# Reset the IPMI BMC to make sure the changes took effect.
ipmitool mc reset cold

# Wait a few seconds and then re-run the call that dumped the setup to ensure
# it is now what we want.
sleep 5
ipmitool -I open lan print 1
</source>

If all went well, you should see the same output as above, but now with your new configuration.

==== Testing IPMI ====

If you skipped the previous step and just want the minimal setup that works, you only need to install one package.

<source lang="bash">
yum install ipmitool
</source>

Regardless of how you configured your IPMI devices, you will want to test now that you can check the power state of your nodes using the IPMI interface.

You can only query the status of the remote node(s) this way. You can't use the following command to check your local node.

In the example below, we will check the state of an-node01 from an-node02. Note that here we use the IP address directly, but in practice I like to use a name that resolves to the IP address of the '''IPMI''' interface (denoted by a .ipmi suffix).

<source lang="bash">
ipmitool -I lan -H 192.168.1.171 -U admin -P secret chassis power status
</source>
<source lang="text">
Chassis Power is on
</source>

You could replace status with on, off and cycle to remotely boot, power off and reboot (power cycle) the node. This is, in fact, what the fence_ipmilan fence agent does. If you can afford to stop your servers, it's a good idea to play with the various power options to see how the work. There are a few more option than what I mentioned here, which you can read about in man ipmitool.

==== Configuring /etc/cluster.conf ====

Lastly, we need to add the IPMI configuration to our [[cluster.conf]] file. Note that, unlike with the [[#Using Node Assassin To Demonstrate Fencing |Node Assassin]] setup, there is no port value in device tag. This is because there is only device controlled by the IPMI BMC; The node itself. The other change is that there are now two <fencedevice ... /> entries; One for each node's IPMI device. Finally, pay attention to the <device name="..." field. This is an example of where it matters.

Here is an example...

<source lang="xml">
<cluster name="an-cluster" config_version="1">
<clusternodes>
<clusternode name="an-node01.alteeve.com" nodeid="1" votes="1">
<fence>
<method name="ipmi">
<device name="fence_an01" action="reboot" />
</method>
</fence>
</clusternode>
<clusternode name="an-node02.alteeve.com" nodeid="2" votes="1">
<fence>
<method name="ipmi">
<device name="fence_an02" action="reboot" />
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="fence_an01" agent="fence_ipmilan" ipaddr="192.168.3.61" login="admin" passwd="secret" />
<fencedevice name="fence_an02" agent="fence_ipmilan" ipaddr="192.168.3.62" login="admin" passwd="secret" />
</fencedevices>
</cluster>
</source>

=== Node Assassin ===

A cheap alternative is the [[Node Assassin]], an open-hardware, open source fence device. It was built to allow the use of commodity system boards that lacked remote management support found on more expensive, server class hardware.

'''Full Disclosure''': Node Assassin was created by me, with much help from others, for this paper.

= Clean Up =

Some daemons may have been installed or dragged in during the setup of the cluster. Most notable is heartbeat which is not compatible with [[RHCS]]. The following command will disable various daemons from starting at boot.

As for the rest of the daemons below, it is safe to run even when the daemons aren't installed or have already been removed. Of course, skip the ones you actually want to use. This assumes that the nodes have been built according to this HowTo.

<source lang="bash">
chkconfig heartbeat off; chkconfig iscsid off; chkconfig iptables off; chkconfig ip6tables off
</source>

= So You Have a Cluster - Now What? =

Building the cluster infrastructure was pretty easy, wasn't it?

But what will you '''''do''''' with it?

{|style="width: 75%; text-align: center; padding: 10px; border-spacing: 15px;"
|-
!style="width: 100%; border: 1px solid #dfdfdf;" colspan="3"|Choose Your Own Adventure Cluster
|-
|style="width: 33%; border: 1px solid #dfdfdf;"|  
|style="width: 34%; border: 1px solid #dfdfdf;"| [[Two Node Fedora 13 Cluster - Xen-Based Virtual Machine Host on DRBD+CLVM|Xen-Based Virtual Machine Host on DRBD+CLVM]] High Availability VM Host
|style="width: 33%; border: 1px solid #dfdfdf;"|  
|}

= Thanks =

* A '''huge''' thanks to [http://iplink.net Interlink Connectivity]! They hire me as a contractor and have allowed me to extend these docs while working on their clusters. Development of these How-Tos would be much slower if not for them. If you need hosting or colo services, drop them a line. Their website is a bit out of date though, so disregard the prices. :)
* To '''sdake''' of [http://corosync.org corosync] for helping me sort out the '''plock''' component and corosync in general.
* To '''Angus Salkeld''' for helping me nail down the Corosync and OpenAIS differences.
* To [https://lists.linux-foundation.org/pipermail/openais/2010-February/013922.html HJ Lee] from the OpenAIS list for helping me understand the mechanisms controlling the Redundant Ring Protocol's failure detection types.
* To [https://lists.linux-foundation.org/pipermail/openais/2010-February/013925.html Steven Dake] for clarifying the to_x vs. logoutput: x arguments in openais.conf.
* To [http://dk.linkedin.com/in/fabbione Fabio Massimo Di Nitto] for helping me get caught up with clustering and VMs.

{{footer}}

User:SRSullivan

2010-08-23T03:59:48Z

SRSullivan: Initial Content

= Scott Sullivan =
[[File:Profile_Photo_Scott_Sullivan.jpg|thumb|left|Profile Photo for Scott Sullivan]]

File:Profile Photo Scott Sullivan.jpg

2010-08-23T03:55:22Z

SRSullivan: Profile Photo for Scott Sullivan

Profile Photo for Scott Sullivan