Fencing KVM Virtual Servers - Revision history

Digimer: /* Guest Configuration */

2013-09-28T22:26:37Z

Guest Configuration

@@ Line 360: / Line 360: @@
 = Guest Configuration =
-This tutorial will use [[Fedora]] 19 (beta TC5) as the guest VMs.
+This tutorial will use [[Fedora]] 19 as the guest VMs.
 == Fedora 19 ==
-If you kept default values, the only step you will need to take is to copy the host's <span class="code">/etc/cluster/fence_xvm.key</span> key to the virtual servers.
+If you kept default values, the only step you will need to take is to install <span class="code">fence-virt</span> and copy the host's <span class="code">/etc/cluster/fence_xvm.key</span> key to the virtual servers.
 === Copy the Host's Key to the VMs ===

Digimer: /* Fedora 18 Host; Starting the Daemon */

2013-09-28T22:09:34Z

Fedora 18 Host; Starting the Daemon

← Older revision		Revision as of 22:09, 28 September 2013
Line 247:		Line 247:
	</syntaxhighlight>		</syntaxhighlight>
	<syntaxhighlight lang="text">		<syntaxhighlight lang="text">
	1		0
	</syntaxhighlight>		</syntaxhighlight>
	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">

Digimer: /* Fedora 18 Host; Configure the Daemon */

2013-09-28T22:08:41Z

Fedora 18 Host; Configure the Daemon

← Older revision		Revision as of 22:08, 28 September 2013
Line 183:		Line 183:
	</syntaxhighlight>		</syntaxhighlight>
	<syntaxhighlight lang="text">		<syntaxhighlight lang="text">
	Key File [/etc/fence_xvm.key]:		Key File [/etc/cluster/fence_xvm.key]: /etc/fence_xvm.key
	</syntaxhighlight>		</syntaxhighlight>
	<syntaxhighlight lang="text">		<syntaxhighlight lang="text">

Digimer: /* Fedora 18 */

2013-09-28T22:06:46Z

Fedora 18

← Older revision		Revision as of 22:06, 28 September 2013
Line 22:		Line 22:
	This tutorial will use [[Fedora]] 18 as the host operating system.		This tutorial will use [[Fedora]] 18 as the host operating system.

	== Fedora 18 ==		== Fedora 18/19 ==

	Configuring the host is a three step process;		Configuring the host is a three step process;
Line 30:		Line 30:
	# Configure the <span class="code">fence_virtd</span> daemon.		# Configure the <span class="code">fence_virtd</span> daemon.

	=== Fedora 18 Host; Installing Components ===		=== Fedora 18/19 Host; Installing Components ===

	The first step is to install the <span class="code">fence_virtd</span> components;		The first step is to install the <span class="code">fence_virtd</span> components;
Line 102:		Line 102:
	For security reasons, we want to use a file with a secret key in it. This will ensure that only nodes in our cluster can send requests to terminate VMs to our <span class="code">fence_virtd</span> daemon.		For security reasons, we want to use a file with a secret key in it. This will ensure that only nodes in our cluster can send requests to terminate VMs to our <span class="code">fence_virtd</span> daemon.

	This key is stored in a file called <span class="code">/etc~~/cluster~~/fence_xvm.key</span> file. You can use any string you want, including simple words or traditional passwords. In our case though, we will use <span class="code">[[dd]]</span> to read in <span class="code">512</span> [[bytes]] from the kernel's <span class="code">[[/dev/random]]</span> (pseudo-)random number generator to create a nice, strong and random key.		This key is stored in a file called <span class="code">/etc/fence_xvm.key</span> file. You can use any string you want, including simple words or traditional passwords. In our case though, we will use <span class="code">[[dd]]</span> to read in <span class="code">512</span> [[bytes]] from the kernel's <span class="code">[[/dev/random]]</span> (pseudo-)random number generator to create a nice, strong and random key.

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	dd if=/dev/random of=/etc~~/cluster~~/fence_xvm.key bs=512 count=1		dd if=/dev/random of=/etc/fence_xvm.key bs=512 count=1
	</syntaxhighlight>		</syntaxhighlight>
	<syntaxhighlight lang="text">		<syntaxhighlight lang="text">
Line 183:		Line 183:
	</syntaxhighlight>		</syntaxhighlight>
	<syntaxhighlight lang="text">		<syntaxhighlight lang="text">
	Key File [/etc~~/cluster~~/fence_xvm.key]:		Key File [/etc/fence_xvm.key]:
	</syntaxhighlight>		</syntaxhighlight>
	<syntaxhighlight lang="text">		<syntaxhighlight lang="text">
Line 212:		Line 212:
	interface = "virbr0";		interface = "virbr0";
	address = "225.0.0.12";		address = "225.0.0.12";
	key_file = "/etc~~/cluster~~/fence_xvm.key";		key_file = "/etc/fence_xvm.key";
	}		}

Digimer: /* Fedora 18 Host; Configure the Daemon */

2013-07-01T19:57:50Z

Fedora 18 Host; Configure the Daemon

← Older revision		Revision as of 19:57, 1 July 2013
Line 119:		Line 119:
	There is a nice configuration mode for <span class="code">fence_virtd</span> that makes configuring it very simple. In this example, we will use all default values.		There is a nice configuration mode for <span class="code">fence_virtd</span> that makes configuring it very simple. In this example, we will use all default values.

	{{note\|1=We will use [[multicast]] for communication between the guests and the host. The multicast address ~~does~~ '''not''' ~~have to~~ match the multicast address used by your cluster.}}		{{note\|1=We will use [[multicast]] for communication between the guests and the host. The multicast address should '''not''' match the multicast address used by your cluster.}}

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">

Digimer: /* Configuring fence_xvm */

2013-07-01T19:48:40Z

Configuring fence_xvm

@@ Line 477: / Line 477: @@
 = Configuring fence_xvm =
 There are two ways you can configure pacemaker; You can use <span class="code">crm</span> or, as of recently, <span class="code">pcs</span>. Given that [[Red Hat]]'s plan is to use <span class="code">pcs</span> in [[RHEL]] 7, we will use that here.
@@ Line 639: / Line 635: @@
 Once it's back up, rejoin it to the cluster and test crashing 'pcmk1'. Done!
 {{footer}}

Digimer: /* Configuring fence_xvm */

2013-07-01T19:47:15Z

Configuring fence_xvm

← Older revision		Revision as of 19:47, 1 July 2013
Line 493:		Line 493:
	<syntaxhighlight lang="text">		<syntaxhighlight lang="text">
	Cluster name: an-cluster-03		Cluster name: an-cluster-03
	Last updated: Mon Jul 1 14:21:16 2013		WARNING: no stonith devices and stonith-enabled is not false
	Last change: Mon Jul 1 14:13:13 2013 via cibadmin on pcmk1.alteeve.ca		Last updated: Mon Jul 1 17:11:57 2013
			Last change: Mon Jul 1 17:10:45 2013 via cibadmin on pcmk1.alteeve.ca
	Stack: corosync		Stack: corosync
	Current DC: pcmk1.alteeve.ca (1) - partition with quorum		Current DC: pcmk1.alteeve.ca (1) - partition with quorum
	Version: 1.1.9-3.fc19-781a388		Version: 1.1.9-3.fc19-781a388
	2 Nodes configured, unknown expected votes		2 Nodes configured, unknown expected votes
	2 Resources configured.		0 Resources configured.


Line 506:		Line 507:
	Full list of resources:		Full list of resources:
	</syntaxhighlight>		</syntaxhighlight>

			{{note\|1=The <span class="code">port="..."</span> must match the name shown on the host when you call <span class="code">virsh list --all</span>.}}

	Now add the two fence primitives.		Now add the two fence primitives.

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	pcs stonith create fence_pcmk1_xvm fence_xvm pcmk_host_list="pcmk1.alteeve.ca"		pcs stonith create fence_pcmk1_xvm fence_xvm port="pcmk1" pcmk_host_list="pcmk1.alteeve.ca"
	pcs stonith create fence_pcmk2_xvm fence_xvm pcmk_host_list="pcmk2.alteeve.ca"		pcs stonith create fence_pcmk2_xvm fence_xvm port="pcmk2" pcmk_host_list="pcmk2.alteeve.ca"
	</syntaxhighlight>		</syntaxhighlight>

Line 521:		Line 524:
	<syntaxhighlight lang="text">		<syntaxhighlight lang="text">
	Cluster name: an-cluster-03		Cluster name: an-cluster-03
	Last updated: Mon Jul 1 16:21:16 2013		Last updated: Mon Jul 1 17:14:36 2013
	Last change: Mon Jul 1 16:13:13 2013 via cibadmin on pcmk1.alteeve.ca		Last change: Mon Jul 1 17:14:29 2013 via cibadmin on pcmk1.alteeve.ca
	Stack: corosync		Stack: corosync
	Current DC: pcmk1.alteeve.ca (1) - partition with quorum		Current DC: pcmk1.alteeve.ca (1) - partition with quorum
Line 534:		Line 537:
	Full list of resources:		Full list of resources:

	fence_pcmk1_xvm (stonith:fence_xvm): Started pcmk1.alteeve.ca		fence_pcmk1_xvm (stonith:fence_xvm): Started pcmk1.alteeve.ca
	fence_pcmk2_xvm (stonith:fence_xvm): Started pcmk2.alteeve.ca		fence_pcmk2_xvm (stonith:fence_xvm): Started pcmk2.alteeve.ca
	</syntaxhighlight>		</syntaxhighlight>

Line 603:		Line 606:

	<syntaxhighlight lang="text">		<syntaxhighlight lang="text">
	~~Failed!~~ >_< - ~~fix later~~		Jul 1 17:15:22 pcmk1 corosync[387]: [TOTEM ] A processor joined or left the membership and a new membership (192.168.122.11:40) was formed.
			Jul 1 17:15:22 pcmk1 corosync[387]: [MAIN ] Completed service synchronization, ready to provide service.
			Jul 1 17:15:23 pcmk1 crmd[405]: warning: match_down_event: No match for shutdown action on 2
			Jul 1 17:15:23 pcmk1 crmd[405]: notice: peer_update_callback: Stonith/shutdown of pcmk2.alteeve.ca not matched
			Jul 1 17:15:23 pcmk1 attrd[403]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
			Jul 1 17:15:23 pcmk1 attrd[403]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
			Jul 1 17:15:23 pcmk1 pengine[404]: notice: unpack_config: On loss of CCM Quorum: Ignore
			Jul 1 17:15:23 pcmk1 pengine[404]: warning: pe_fence_node: Node pcmk2.alteeve.ca will be fenced because the node is no longer part of the cluster
			Jul 1 17:15:23 pcmk1 pengine[404]: warning: determine_online_status: Node pcmk2.alteeve.ca is unclean
			Jul 1 17:15:23 pcmk1 pengine[404]: warning: custom_action: Action fence_pcmk2_xvm_stop_0 on pcmk2.alteeve.ca is unrunnable (offline)
			Jul 1 17:15:23 pcmk1 pengine[404]: warning: stage6: Scheduling Node pcmk2.alteeve.ca for STONITH
			Jul 1 17:15:23 pcmk1 pengine[404]: notice: LogActions: Move fence_pcmk2_xvm#011(Started pcmk2.alteeve.ca -> pcmk1.alteeve.ca)
			Jul 1 17:15:23 pcmk1 pengine[404]: warning: process_pe_message: Calculated Transition 23: /var/lib/pacemaker/pengine/pe-warn-1.bz2
			Jul 1 17:15:23 pcmk1 crmd[405]: notice: te_fence_node: Executing reboot fencing operation (9) on pcmk2.alteeve.ca (timeout=60000)
			Jul 1 17:15:23 pcmk1 stonith-ng[401]: notice: handle_request: Client crmd.405.be9644f2 wants to fence (reboot) 'pcmk2.alteeve.ca' with device '(any)'
			Jul 1 17:15:23 pcmk1 stonith-ng[401]: notice: initiate_remote_stonith_op: Initiating remote operation reboot for pcmk2.alteeve.ca: 1359b987-68ba-48ee-812f-9d8afafa9e41 (0)
			Jul 1 17:15:24 pcmk1 stonith-ng[401]: notice: log_operation: Operation 'reboot' [1635] (call 3 from crmd.405) for host 'pcmk2.alteeve.ca' with device 'fence_pcmk2_xvm' returned: 0 (OK)
			Jul 1 17:15:24 pcmk1 stonith-ng[401]: notice: remote_op_done: Operation reboot of pcmk2.alteeve.ca by pcmk1.alteeve.ca for crmd.405@pcmk1.alteeve.ca.1359b987: OK
			Jul 1 17:15:24 pcmk1 crmd[405]: notice: tengine_stonith_callback: Stonith operation 3/9:23:0:49d423be-6d17-47bd-83b7-528c0c6faf82: OK (0)
			Jul 1 17:15:24 pcmk1 crmd[405]: notice: tengine_stonith_notify: Peer pcmk2.alteeve.ca was terminated (reboot) by pcmk1.alteeve.ca for pcmk1.alteeve.ca: OK (ref=1359b987-68ba-48ee-812f-9d8afafa9e41) by client crmd.405
			Jul 1 17:15:24 pcmk1 crmd[405]: notice: te_rsc_command: Initiating action 7: start fence_pcmk2_xvm_start_0 on pcmk1.alteeve.ca (local)
			Jul 1 17:15:24 pcmk1 stonith-ng[401]: notice: stonith_device_register: Added 'fence_pcmk2_xvm' to the device list (2 active devices)
			Jul 1 17:15:25 pcmk1 crmd[405]: notice: process_lrm_event: LRM operation fence_pcmk2_xvm_start_0 (call=42, rc=0, cib-update=141, confirmed=true) ok
			Jul 1 17:15:25 pcmk1 crmd[405]: notice: run_graph: Transition 23 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-1.bz2): Complete
			Jul 1 17:15:25 pcmk1 pengine[404]: notice: unpack_config: On loss of CCM Quorum: Ignore
			Jul 1 17:15:25 pcmk1 pengine[404]: notice: process_pe_message: Calculated Transition 24: /var/lib/pacemaker/pengine/pe-input-17.bz2
			Jul 1 17:15:25 pcmk1 crmd[405]: notice: run_graph: Transition 24 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-17.bz2): Complete
			Jul 1 17:15:25 pcmk1 crmd[405]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
	</syntaxhighlight>		</syntaxhighlight>

			You should see that 'pcmk2' has now rebooted!

			Once it's back up, rejoin it to the cluster and test crashing 'pcmk1'. Done!

			== Configuring RHCS's cluster.conf ==

			ToDo.

	<span class="code"></span>		<span class="code"></span>

Digimer: /* Configuring fence_xvm */

2013-07-01T19:30:46Z

Configuring fence_xvm

@@ Line 487: / Line 487: @@
 Here is our start point;
 <syntaxhighlight lang="bash">
@@ Line 511: / Line 563: @@
 </syntaxhighlight>
-Now add the two fence primitives.
+Enabling it is as easy as:
 <syntaxhighlight lang="bash">
 </syntaxhighlight>
 <syntaxhighlight lang="text">
 </syntaxhighlight>

Digimer: /* Guest Configuration */

2013-06-30T23:16:35Z

Guest Configuration

Show changes

Digimer: /* Fedora 18 Host; Bridge Configuration Issue */

2013-06-21T16:31:33Z

Fedora 18 Host; Bridge Configuration Issue

← Older revision		Revision as of 16:31, 21 June 2013
Line 331:		Line 331:
	You should now be able to run <span class="code">fence_xvm -o list</span> on the guests without issue.		You should now be able to run <span class="code">fence_xvm -o list</span> on the guests without issue.

	If you wish to make this persistent, ~~a contributor who wishes to remain anonymous~~ shared a nice little [[udev]] rule and [[bash]] script to do this.		If you wish to make this persistent, <span class="code">trapier</span> from [[Red Hat]] shared a nice little [[udev]] rule and [[bash]] script to do this.

	First, as the <span class="code">root</span> user, or using <span class="code">sudo</span>, create the script that loops through all bridges named <span class="code">virbrX</span> and enables their multicast querier.		First, as the <span class="code">root</span> user, or using <span class="code">sudo</span>, create the script that loops through all bridges named <span class="code">virbrX</span> and enables their multicast querier.