Fencing KVM Virtual Servers
{(howto_header}}
It's difficult to image a production use-case for clusters built on virtual machines. As a learning and test bed though, clusters of virtual machines can be invaluable.
Clusters of VMs still require fencing, just the same. Generally this is accomplished by using a fence agent that calls the hypervisor and asks it to force the target VM off.
Each hypervisor will have it's own methods for doing this. In this tutorial, we will discuss fencing virtual machines running on the KVM hypervisor using the fence_virtd daemon and the fence_xvm guest fence agent.
 |
Note: This tutorial was based on Clusterlabs.org's "Guest Fencing" tutorial. Thanks to Andrew Beekhof for writing this and helping me sort out some problems I ran into. |
Steps
Setting up fencing requires two steps;
- Configure the host's fence_virtd to listen for fence requests from guests.
- Configure the guests to talk to the host's daemon using their fence_xvm fence agents.
There is no need for the guest and host operating systems to match. So the specific steps for each of these two steps will vary somewhat depending on which OS you are using.
Host Configuration
This tutorial will use Fedora 18 as the host operating system.
Fedora 18
Configuring the host is a three step process;
- Install the needed components.
- Create a key.
- Configure the fence_virtd daemon.
Fedora 18 Host; Installing Components
The first step is to install the fence_virtd components;
yum install fence-virt fence-virtd fence-virtd-multicast fence-virtd-libvirt
Loaded plugins: langpacks, presto, refresh-packagekit
Resolving Dependencies
--> Running transaction check
---> Package fence-virt.x86_64 0:0.3.0-5.fc18 will be installed
---> Package fence-virtd.x86_64 0:0.3.0-5.fc18 will be installed
---> Package fence-virtd-libvirt.x86_64 0:0.3.0-5.fc18 will be installed
---> Package fence-virtd-multicast.x86_64 0:0.3.0-5.fc18 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
fence-virt x86_64 0.3.0-5.fc18 fedora 38 k
fence-virtd x86_64 0.3.0-5.fc18 fedora 30 k
fence-virtd-libvirt x86_64 0.3.0-5.fc18 fedora 13 k
fence-virtd-multicast x86_64 0.3.0-5.fc18 fedora 19 k
Transaction Summary
================================================================================
Install 4 Packages
Total download size: 101 k
Installed size: 173 k
Is this ok [y/N]: y
Downloading Packages:
(1/4): fence-virt-0.3.0-5.fc18.x86_64.rpm | 38 kB 00:00
(2/4): fence-virtd-0.3.0-5.fc18.x86_64.rpm | 30 kB 00:00
(3/4): fence-virtd-libvirt-0.3.0-5.fc18.x86_64.rpm | 13 kB 00:00
(4/4): fence-virtd-multicast-0.3.0-5.fc18.x86_64.rpm | 19 kB 00:00
--------------------------------------------------------------------------------
Total 187 kB/s | 101 kB 00:00
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : fence-virtd-0.3.0-5.fc18.x86_64 1/4
Installing : fence-virtd-multicast-0.3.0-5.fc18.x86_64 2/4
Installing : fence-virtd-libvirt-0.3.0-5.fc18.x86_64 3/4
Installing : fence-virt-0.3.0-5.fc18.x86_64 4/4
Verifying : fence-virtd-multicast-0.3.0-5.fc18.x86_64 1/4
Verifying : fence-virt-0.3.0-5.fc18.x86_64 2/4
Verifying : fence-virtd-0.3.0-5.fc18.x86_64 3/4
Verifying : fence-virtd-libvirt-0.3.0-5.fc18.x86_64 4/4
Installed:
fence-virt.x86_64 0:0.3.0-5.fc18
fence-virtd.x86_64 0:0.3.0-5.fc18
fence-virtd-libvirt.x86_64 0:0.3.0-5.fc18
fence-virtd-multicast.x86_64 0:0.3.0-5.fc18
Complete!
Fedora 18 Host; Create The Key
For security reasons, we want to use a file with a secret key in it. This will ensure that only nodes in our cluster can send requests to terminate VMs to our fence_virtd daemon.
This key is stored in a file called /etc/cluster/fence_xvm.key file. You can use any string you want, including simple words or traditional passwords. In our case though, we will use dd to read in 512 bytes from the kernel's /dev/random (pseudo-)random number generator to create a nice, strong and random key.
dd if=/dev/random of=/etc/cluster/fence_xvm.key bs=512 count=1
0+1 records in
0+1 records out
128 bytes (128 B) copied, 0.000143983 s, 889 kB/s
If you try to look at the contents of this, you will find that it does not render as ASCII text very well. This is expected and not a problem.
Fedora 18 Host; Configure the Daemon
There is a nice configuration mode for fence_virtd that makes configuring it very simple. In this example, we will use all default values.
|
Note: We will use multicast for communication between the guests and the host. The multicast address does not have to match the multicast address used by your cluster. |
fence_virtd -c
Module search path [/usr/lib64/fence-virt]:
Available backends:
libvirt 0.1
Available listeners:
multicast 1.2
Listener modules are responsible for accepting requests
from fencing clients.
Listener module [multicast]:
The multicast listener module is designed for use environments
where the guests and hosts may communicate over a network using
multicast.
The multicast address is the address that a client will use to
send fencing requests to fence_virtd.
Multicast IP Address [225.0.0.12]:
Using ipv4 as family.
Multicast IP Port [1229]:
Setting a preferred interface causes fence_virtd to listen only
on that interface. Normally, it listens on all interfaces.
In environments where the virtual machines are using the host
machine as a gateway, this *must* be set (typically to virbr0).
Set to 'none' for no interface.
Interface [virbr0]:
The key file is the shared key information which is used to
authenticate fencing requests. The contents of this file must
be distributed to each physical host and virtual machine within
a cluster.
Key File [/etc/cluster/fence_xvm.key]:
Backend modules are responsible for routing requests to
the appropriate hypervisor or management layer.
Backend module [libvirt]:
Configuration complete.
=== Begin Configuration ===
backends {
libvirt {
uri = "qemu:///system";
}
}
listeners {
multicast {
port = "1229";
family = "ipv4";
interface = "virbr0";
address = "225.0.0.12";
key_file = "/etc/cluster/fence_xvm.key";
}
}
fence_virtd {
module_path = "/usr/lib64/fence-virt";
backend = "libvirt";
listener = "multicast";
}
=== End Configuration ===
Replace /etc/fence_virt.conf with the above [y/N]? y
Note that it will ask if you want to replace the /etc/fence_virt.conf, even though it doesn't exist.
Done!
Fedora 18 Host; Starting the Daemon
There is no init.d or systemd tool for starting or stopping the fence_virtd daemon. So to start it, simple call it at the command line;
fence_virtd
Nothing will be displayed, but we can check that it started well by checking it's exit code and using pidof to check for it's process ID.
echo $?
1
pidof fence_virtd
3244
|
Note: The process ID you see will almost certainly be different from the one above. |
That's it!
Fedora 18 Host; Restarting fence_virtd
If you ever want to stop fence_virtd, in order to load an updated configuration for example, you can use kill to do so.
kill $(pidof fence_virtd)
If you re-run the pidof command, you will see that no process ID is returned any more.
pidof fence_virtd
Now start it back up again;
fence_virtd
pidof fence_virtd
29210
Fedora 18 Host; Testing the Daemon
To test that the daemon is listening, you can run fence_xvm and ask it to list the clients. We have not configured any guests yet, so it should return immediately with no output.
fence_xvm -o list
Had there been a problem, you would see an error like:
fence_xvm -o list
Timed out waiting for response
Operation failed
Fedora 18 Host; Bridge Configuration Issue
|
Note: This may not be needed on you system. Please see rhbz#880035, in particular, comment #8. |
In earlier versions of Fedora, fence_xvm "just worked". However, the behaviour of bridging changed and now it is required that the multicast querier for the bridge we use for fencing be enabled manually.
Later, when you configure the clients, you may find that calling fence_xvm -o list times out, despite everything being configured properly. That will indicate that you need to manually enable the multicast querier.
We are using the virbr0 bridge in this tutorial. So to check if we should make the fix, read it's sys file;
cat /sys/class/net/virbr0/bridge/multicast_querier
0
This tells us that it is off. To enable it, write a 1 to that file.
echo 1 > /sys/class/net/virbr0/bridge/multicast_querier
cat /sys/class/net/virbr0/bridge/multicast_querier
1
You should now be able to run fence_xvm -o list on the guests without issue.
If you wish to make this persistent, a contributor who wishes to remain anonymous shared a nice little udev rule and bash script to do this.
First, as the root user, or using sudo, create the script that loops through all bridges named virbrX and enables their multicast querier.
touch /etc/sysconfig/network-scripts/vnet_querier_enable
chmod 755 /etc/sysconfig/network-scripts/vnet_querier_enable
vim /etc/sysconfig/network-scripts/vnet_querier_enable
#!/bin/sh
if [[ $INTERFACE == virbr* ]]; then
/bin/echo 1 > /sys/devices/virtual/net/$INTERFACE/bridge/multicast_querier
fi
Now create the udev rules file.
vim /etc/udev/rules.d/61-virbr-querier.rules
ACTION=="add", SUBSYSTEM=="net", RUN+="/etc/sysconfig/network-scripts/vnet_querier_enable"
Now the querier should be enabled for all virbrX bridges on boot.
Guest Configuration
This tutorial will use Fedora 19 (beta TC5) as the guest VMs.
Fedora 19
blah
| Any questions, feedback, advice, complaints or meanderings are welcome. | |||
| Alteeve's Niche! | Enterprise Support: Alteeve Support |
Community Support | |
| © Alteeve's Niche! Inc. 1997-2024 | Anvil! "Intelligent Availability®" Platform | ||
| legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions. | |||