Fencing KVM Virtual Servers

{(howto_header}}

It's difficult to image a production use-case for clusters built on virtual machines. As a learning and test bed though, clusters of virtual machines can be invaluable.

Clusters of VMs still require fencing, just the same. Generally this is accomplished by using a fence agent that calls the hypervisor and asks it to force the target VM off.

Each hypervisor will have it's own methods for doing this. In this tutorial, we will discuss fencing virtual machines running on the KVM hypervisor using the fence_virtd daemon and the fence_xvm guest fence agent.

Note: This tutorial was based on Clusterlabs.org's "Guest Fencing" tutorial. Thanks to Andrew Beekhof for writing this and helping me sort out some problems I ran into.

Steps

Setting up fencing requires two steps;

1. Configure the host's fence_virtd to listen for fence requests from guests.
2. Configure the guests to talk to the host's daemon using their fence_xvm fence agents.

There is no need for the guest and host operating systems to match. So the specific steps for each of these two steps will vary somewhat depending on which OS you are using.

Host Configuration

This tutorial will use Fedora 18 as the host operating system.

Fedora 18

Configuring the host is a three step process;

1. Install the needed components.
2. Create a key.
3. Configure the fence_virtd daemon.

Fedora 18 Host; Installing Components

The first step is to install the fence_virtd components;

yum install fence-virt fence-virtd fence-virtd-multicast fence-virtd-libvirt

Loaded plugins: langpacks, presto, refresh-packagekit
Resolving Dependencies
--> Running transaction check
---> Package fence-virt.x86_64 0:0.3.0-5.fc18 will be installed
---> Package fence-virtd.x86_64 0:0.3.0-5.fc18 will be installed
---> Package fence-virtd-libvirt.x86_64 0:0.3.0-5.fc18 will be installed
---> Package fence-virtd-multicast.x86_64 0:0.3.0-5.fc18 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                     Arch         Version            Repository    Size
================================================================================
Installing:
 fence-virt                  x86_64       0.3.0-5.fc18       fedora        38 k
 fence-virtd                 x86_64       0.3.0-5.fc18       fedora        30 k
 fence-virtd-libvirt         x86_64       0.3.0-5.fc18       fedora        13 k
 fence-virtd-multicast       x86_64       0.3.0-5.fc18       fedora        19 k

Transaction Summary
================================================================================
Install  4 Packages

Total download size: 101 k
Installed size: 173 k

Is this ok [y/N]: y

Downloading Packages:
(1/4): fence-virt-0.3.0-5.fc18.x86_64.rpm                  |  38 kB   00:00     
(2/4): fence-virtd-0.3.0-5.fc18.x86_64.rpm                 |  30 kB   00:00     
(3/4): fence-virtd-libvirt-0.3.0-5.fc18.x86_64.rpm         |  13 kB   00:00     
(4/4): fence-virtd-multicast-0.3.0-5.fc18.x86_64.rpm       |  19 kB   00:00     
--------------------------------------------------------------------------------
Total                                           187 kB/s | 101 kB     00:00     
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : fence-virtd-0.3.0-5.fc18.x86_64                              1/4 
  Installing : fence-virtd-multicast-0.3.0-5.fc18.x86_64                    2/4 
  Installing : fence-virtd-libvirt-0.3.0-5.fc18.x86_64                      3/4 
  Installing : fence-virt-0.3.0-5.fc18.x86_64                               4/4 
  Verifying  : fence-virtd-multicast-0.3.0-5.fc18.x86_64                    1/4 
  Verifying  : fence-virt-0.3.0-5.fc18.x86_64                               2/4 
  Verifying  : fence-virtd-0.3.0-5.fc18.x86_64                              3/4 
  Verifying  : fence-virtd-libvirt-0.3.0-5.fc18.x86_64                      4/4 

Installed:
  fence-virt.x86_64 0:0.3.0-5.fc18                                              
  fence-virtd.x86_64 0:0.3.0-5.fc18                                             
  fence-virtd-libvirt.x86_64 0:0.3.0-5.fc18                                     
  fence-virtd-multicast.x86_64 0:0.3.0-5.fc18                                   

Complete!

Fedora 18 Host; Create The Key

For security reasons, we want to use a file with a secret key in it. This will ensure that only nodes in our cluster can send requests to terminate VMs to our fence_virtd daemon.

This key is stored in a file called /etc/cluster/fence_xvm.key file. You can use any string you want, including simple words or traditional passwords. In our case though, we will use dd to read in 512 bytes from the kernel's /dev/random (pseudo-)random number generator to create a nice, strong and random key.

dd if=/dev/random of=/etc/cluster/fence_xvm.key bs=512 count=1

0+1 records in
0+1 records out
128 bytes (128 B) copied, 0.000143983 s, 889 kB/s

If you try to look at the contents of this, you will find that it does not render as ASCII text very well. This is expected and not a problem.

Fedora 18 Host; Configure the Daemon

There is a nice configuration mode for fence_virtd that makes configuring it very simple. In this example, we will use all default values.

Note: We will use multicast for communication between the guests and the host. The multicast address does not have to match the multicast address used by your cluster.

fence_virtd -c

Module search path [/usr/lib64/fence-virt]:

Available backends:
    libvirt 0.1
Available listeners:
    multicast 1.2

Listener modules are responsible for accepting requests
from fencing clients.

Listener module [multicast]:

The multicast listener module is designed for use environments
where the guests and hosts may communicate over a network using
multicast.

The multicast address is the address that a client will use to
send fencing requests to fence_virtd.

Multicast IP Address [225.0.0.12]:

Using ipv4 as family.

Multicast IP Port [1229]:

Setting a preferred interface causes fence_virtd to listen only
on that interface.  Normally, it listens on all interfaces.
In environments where the virtual machines are using the host
machine as a gateway, this *must* be set (typically to virbr0).
Set to 'none' for no interface.

Interface [virbr0]:

The key file is the shared key information which is used to
authenticate fencing requests.  The contents of this file must
be distributed to each physical host and virtual machine within
a cluster.

Key File [/etc/cluster/fence_xvm.key]:

Backend modules are responsible for routing requests to
the appropriate hypervisor or management layer.

Backend module [libvirt]:

Configuration complete.

=== Begin Configuration ===
backends {
	libvirt {
		uri = "qemu:///system";
	}

}

listeners {
	multicast {
		port = "1229";
		family = "ipv4";
		interface = "virbr0";
		address = "225.0.0.12";
		key_file = "/etc/cluster/fence_xvm.key";
	}

}

fence_virtd {
	module_path = "/usr/lib64/fence-virt";
	backend = "libvirt";
	listener = "multicast";
}

=== End Configuration ===

Replace /etc/fence_virt.conf with the above [y/N]? y

Note that it will ask if you want to replace the /etc/fence_virt.conf, even though it doesn't exist.

Done!

Fedora 18 Host; Starting the Daemon

There is no init.d or systemd tool for starting or stopping the fence_virtd daemon. So to start it, simple call it at the command line;

fence_virtd

Nothing will be displayed, but we can check that it started well by checking it's exit code and using pidof to check for it's process ID.

echo $?

1

pidof fence_virtd

3244

Note: The process ID you see will almost certainly be different from the one above.

That's it!

Fedora 18 Host; Restarting fence_virtd

If you ever want to stop fence_virtd, in order to load an updated configuration for example, you can use kill to do so.

kill $(pidof fence_virtd)

If you re-run the pidof command, you will see that no process ID is returned any more.

pidof fence_virtd

Now start it back up again;

fence_virtd
pidof fence_virtd

29210

Fedora 18 Host; Testing the Daemon

To test that the daemon is listening, you can run fence_xvm and ask it to list the clients. We have not configured any guests yet, so it should return immediately with no output.

fence_xvm -o list

Had there been a problem, you would see an error like:

fence_xvm -o list

Timed out waiting for response
Operation failed

Guest Configuration

This tutorial will use Fedora 19 (beta TC5) as the guest VMs.

Fedora 19

blah

 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche!	Enterprise Support: Alteeve Support	Community Support
© Alteeve's Niche! Inc. 1997-2024		Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.