Watchdog Recovery - Revision history

Digimer: /* Configuration cman */

2012-12-11T21:01:51Z

Configuration cman

← Older revision		Revision as of 21:01, 11 December 2012
Line 171:		Line 171:
	The key attributes are:		The key attributes are:

	* <span class="code">host_id="x"</span>; This tells <span class="code">fence_sanlock</span> which node ID to act on. Generally it matches the corresponding <span class="code"><clusternode ... nodeid="x"></span> value, though it must be unique.		* <span class="code">host_id="x"</span>; This tells <span class="code">fence_sanlock</span> which node ID to act on. Generally it matches the corresponding <span class="code"><clusternode ... nodeid="x"></span> value. It doesn't have to, though, but it must be unique.
	* <span class="code">path="/dev/x"</span>; This tells the cluster where each node's lock can be found. Here we used <span class="code">/dev/sdb/</span>, which is the device we initialized in the previous step.		* <span class="code">path="/dev/x"</span>; This tells the cluster where each node's lock can be found. Here we used <span class="code">/dev/sdb/</span>, which is the device we initialized in the previous step.

Digimer: /* Configuration cman */

2012-12-11T20:59:23Z

Configuration cman

← Older revision		Revision as of 20:59, 11 December 2012
Line 171:		Line 171:
	The key attributes are:		The key attributes are:

	* <span class="code">host_id="x"</span>; This tells <span class="code">fence_sanlock</span> which node ID to act on. ~~It must match~~ the corresponding <span class="code"><clusternode ... nodeid="x"></span> value.		* <span class="code">host_id="x"</span>; This tells <span class="code">fence_sanlock</span> which node ID to act on. Generally it matches the corresponding <span class="code"><clusternode ... nodeid="x"></span> value, though it must be unique.
	* <span class="code">path="/dev/x"</span>; This tells the cluster where each node's lock can be found. Here we used <span class="code">/dev/sdb/</span>, which is the device we initialized in the previous step.		* <span class="code">path="/dev/x"</span>; This tells the cluster where each node's lock can be found. Here we used <span class="code">/dev/sdb/</span>, which is the device we initialized in the previous step.

Digimer at 07:33, 2 November 2012

2012-11-02T07:33:34Z

← Older revision		Revision as of 07:33, 2 November 2012
Line 461:		Line 461:
	At this time, using both <span class="code">fence_sanlock</span> and <span class="code">checkquorum.wdmd</span> is not supported.		At this time, using both <span class="code">fence_sanlock</span> and <span class="code">checkquorum.wdmd</span> is not supported.

			= Permissions =

			[[Digimer\|I]] give unrestricted permission to [https://redhat.com Red Hat, Inc.] to copy this document in whole or in part.

	{{footer}}		{{footer}}

Digimer: /* Using checkquorum.wdmd */

2012-10-27T08:36:15Z

Using checkquorum.wdmd

← Older revision		Revision as of 08:36, 27 October 2012
Line 400:		Line 400:
	systemctl start cman.service		systemctl start cman.service
	</source>		</source>

	~~{{note\|1=As noted before, it can take several minutes for the nodes to complete their <span class="code">unfence</span> action during cluster start up, please be patient.}}~~

	== Testing checkquorum.wdmd ==		== Testing checkquorum.wdmd ==

Digimer: /* checkquorum.wdmd */

2012-10-27T08:35:13Z

checkquorum.wdmd

← Older revision		Revision as of 08:35, 27 October 2012
Line 255:		Line 255:

	To get around this 2-node limitation, we can use <span class="code">[[qdisk]]</span> on shared storage. This provides a third vote and allows quorum to be used properly.		To get around this 2-node limitation, we can use <span class="code">[[qdisk]]</span> on shared storage. This provides a third vote and allows quorum to be used properly.

			{{note\|1=The <span class="code">qdisk</span> device '''must''' be in <span class="code">master_wins</span> mode. Please see <span class="code">man 5 qdisk</span> for more information.}}

	== Requirements ==		== Requirements ==

Digimer: /* checkquorum.wdmd */

2012-10-27T07:51:01Z

checkquorum.wdmd

← Older revision		Revision as of 07:51, 27 October 2012
Line 252:		Line 252:
	The <span class="code">checkquorum.wdmd</span> is not really a fence device in the traditional sense. The only way for a <span class="code">fence</span> action to be considered a success is to have the failed node rejoin the cluster in a clean (freshly started) state. If a node fails in such a way that it can node start back up, the cluster will remain blocked until cleared by an administrator issuing a <span class="code">fence_ack_manual</span> on the survivor node.		The <span class="code">checkquorum.wdmd</span> is not really a fence device in the traditional sense. The only way for a <span class="code">fence</span> action to be considered a success is to have the failed node rejoin the cluster in a clean (freshly started) state. If a node fails in such a way that it can node start back up, the cluster will remain blocked until cleared by an administrator issuing a <span class="code">fence_ack_manual</span> on the survivor node.

	The <span class="code">checkquorum.wdmd</span> works by having <span class="code">[[~~corosync~~]]</span> stop resetting the system's watchdog timer if the node loses [[quorum]]. For this reason, <span class="code">checkquorum.wdmd</span> will not work if the 2-node (<span class="code"><cman expected_votes="1" two_node="1"/></span>) option is set in <span class="code">/etc/cluster/cluster.conf</span>. This is because quorum is effectively disabled. When configured for 2-node clusters, a node will never lose quorum, thus, <span class="code">corosync</span> will never stop resetting the watchdog timer.		The <span class="code">checkquorum.wdmd</span> works by having <span class="code">[[wdmd]]</span> stop resetting the system's watchdog timer if the node loses [[quorum]]. For this reason, <span class="code">checkquorum.wdmd</span> will not work if the 2-node (<span class="code"><cman expected_votes="1" two_node="1"/></span>) option is set in <span class="code">/etc/cluster/cluster.conf</span>. This is because quorum is effectively disabled. When configured for 2-node clusters, a node will never lose quorum, thus, <span class="code">corosync</span> will never stop resetting the watchdog timer.

	To get around this 2-node limitation, we can use <span class="code">[[qdisk]]</span> on shared storage. This provides a third vote and allows quorum to be used properly.		To get around this 2-node limitation, we can use <span class="code">[[qdisk]]</span> on shared storage. This provides a third vote and allows quorum to be used properly.

Digimer: /* Setting Up IPMI-Based Watchdog Timer */

2012-10-23T06:18:07Z

Setting Up IPMI-Based Watchdog Timer

← Older revision		Revision as of 06:18, 23 October 2012
Line 461:		Line 461:
	At this time, using both <span class="code">fence_sanlock</span> and <span class="code">checkquorum.wdmd</span> is not supported.		At this time, using both <span class="code">fence_sanlock</span> and <span class="code">checkquorum.wdmd</span> is not supported.

	~~= Setting Up IPMI-Based Watchdog Timer =~~

	~~Different devices will have different methods for installing and setting up.~~

	~~<source lang="bash">~~
	~~# Install and start the ipmi daemon~~
	~~yum install freeipmi-bmc-watchdog watchdog OpenIPMI~~
	~~systemctl enable ipmi.service~~
	~~systemctl start ipmi.service~~

	~~# See the current state of the watchdog timer~~
	~~ipmitool mc watchdog get~~

	~~# Disable the timer.~~
	~~ipmitool mc watchdog off~~

	~~# Enable the timer.~~
	~~ipmitool mc watchdog reset~~
	~~</source>~~

	{{footer}}		{{footer}}

Digimer: /* Setting Up IPMI-Based Watchdog Timer */

2012-10-23T06:17:47Z

Setting Up IPMI-Based Watchdog Timer

← Older revision		Revision as of 06:17, 23 October 2012
Line 480:		Line 480:
	ipmitool mc watchdog reset		ipmitool mc watchdog reset
	</source>		</source>


	~~<span class="code"></span>~~
	~~<source lang="bash">~~
	~~</source>~~
	~~<source lang="text">~~
	~~</source>~~


	{{footer}}		{{footer}}

Digimer: /* Configuring qdisk (2-Node Only) */

2012-10-23T06:14:58Z

Configuring qdisk (2-Node Only)

← Older revision		Revision as of 06:14, 23 October 2012
Line 325:		Line 325:
	</source>		</source>

	With this now created, edit your <span class="code">cluster.conf</span> file to use the quorum disk. This involves removing the <span class="code"><cman expected_votes="1" two_node="1"/></span> entry and replacing it with the quorum configuration. The important value we will need to set is <span class="code">~~master_wins~~="1"</span> which is ~~required for~~ <span class="code">~~checkquorum.wdmd~~</span> ~~to work~~. For more information of the various available <span class="code">qdisk</span> attributes and values by reading <span class="code">man 5 qdisk</span>.		With this now created, edit your <span class="code">cluster.conf</span> file to use the quorum disk. This involves removing the <span class="code"><cman expected_votes="1" two_node="1"/></span> entry and replacing it with the quorum configuration. The important value we will need to set is <span class="code">label="an-cluster-01.wdmd"</span> which tells <span class="code">cman</span> which is it's <span class="code">qdisk</span> device. For more information of the various available <span class="code">qdisk</span> attributes and values by reading <span class="code">man 5 qdisk</span>.

	<source lang="bash">		<source lang="bash">
Line 332:		Line 332:
	<source lang="xml">		<source lang="xml">
	<?xml version="1.0"?>		<?xml version="1.0"?>
	<cluster config_version="2" name="an-cluster-01">		<cluster config_version="3" name="an-cluster-01">
	<quorumd ~~votes~~="~~1" master_wins="1" device="/dev/sdb~~"/>		<quorumd label="an-cluster-01.wdmd"/>
	<clusternodes>		<clusternodes>
	<clusternode name="an-c01n01.alteeve.ca" nodeid="1" />		<clusternode name="an-c01n01.alteeve.ca" nodeid="1"/>
	<clusternode name="an-c01n02.alteeve.ca" nodeid="2" />		<clusternode name="an-c01n02.alteeve.ca" nodeid="2"/>
	</clusternodes>		</clusternodes>
	<fencedevices/>		<fencedevices/>

Digimer: /* Using fence_sanlock And checkquorum.wdmd */

2012-10-23T06:11:11Z

Using fence_sanlock And checkquorum.wdmd

← Older revision		Revision as of 06:11, 23 October 2012
Line 400:		Line 400:

	{{note\|1=As noted before, it can take several minutes for the nodes to complete their <span class="code">unfence</span> action during cluster start up, please be patient.}}		{{note\|1=As noted before, it can take several minutes for the nodes to complete their <span class="code">unfence</span> action during cluster start up, please be patient.}}

			== Testing checkquorum.wdmd ==

			If everything is working, you should now be able to recover from a hard-crash on a node in the cluster. To test this, you can <span class="code">echo c</span> to <span class="code">/proc/sysrq-trigger</span> and the other node should, eventually, fence the lost node. Remember; The cluster will remain blocked until ''either'';
			* The lost node reboots and rejoins the cluster in a clean state.
			''or'';
			* You manually clear the fence by issuing <span class="code">fence_ack_manual</span>.

			{{warning\|1=You '''must''' be certain that the failed node has been manually powered off ''before'' manually clearing the fence. Failing to do so could cause a serious failure in your cluster!}}

			<source lang="bash">
			echo c > /proc/sysrq-trigger
			</source>

			In the system log of the other node, you will see messages similar to those in the example below.

			<source lang="text">
			Oct 22 22:01:25 an-c01n01 qdiskd[2138]: Writing eviction notice for node 2
			Oct 22 22:01:26 an-c01n01 qdiskd[2138]: Node 2 evicted
			Oct 22 22:01:28 an-c01n01 corosync[2084]: [TOTEM ] A processor failed, forming new configuration.
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CLM ] CLM CONFIGURATION CHANGE
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CLM ] New Configuration:
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CLM ] #011r(0) ip(10.20.10.1)
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CLM ] Members Left:
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CLM ] #011r(0) ip(10.20.10.2)
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CLM ] Members Joined:
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [QUORUM] Members[1]: 1
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CLM ] CLM CONFIGURATION CHANGE
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CLM ] New Configuration:
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CLM ] #011r(0) ip(10.20.10.1)
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CLM ] Members Left:
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CLM ] Members Joined:
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [CPG ] chosen downlist: sender r(0) ip(10.20.10.1) ; members(old:2 left:1)
			Oct 22 22:01:30 an-c01n01 kernel: [13038.313753] dlm: closing connection to node 2
			Oct 22 22:01:30 an-c01n01 corosync[2084]: [MAIN ] Completed service synchronization, ready to provide service.
			Oct 22 22:01:30 an-c01n01 fenced[2218]: fencing node an-c01n02.alteeve.ca
			Oct 22 22:01:30 an-c01n01 fenced[2218]: fence an-c01n02.alteeve.ca dev 0.0 agent none result: error no method
			Oct 22 22:01:30 an-c01n01 fenced[2218]: fence an-c01n02.alteeve.ca failed
			Oct 22 22:01:33 an-c01n01 fenced[2218]: fencing node an-c01n02.alteeve.ca
			Oct 22 22:01:33 an-c01n01 fenced[2218]: fence an-c01n02.alteeve.ca dev 0.0 agent none result: error no method
			Oct 22 22:01:33 an-c01n01 fenced[2218]: fence an-c01n02.alteeve.ca failed
			Oct 22 22:01:36 an-c01n01 fenced[2218]: fencing node an-c01n02.alteeve.ca
			Oct 22 22:01:37 an-c01n01 fenced[2218]: fence an-c01n02.alteeve.ca dev 0.0 agent none result: error no method
			Oct 22 22:01:37 an-c01n01 fenced[2218]: fence an-c01n02.alteeve.ca failed
			</source>

			Note the errors;

			<source lang="text">
			Oct 22 22:01:30 an-c01n01 fenced[2218]: fencing node an-c01n02.alteeve.ca
			Oct 22 22:01:30 an-c01n01 fenced[2218]: fence an-c01n02.alteeve.ca dev 0.0 agent none result: error no method
			Oct 22 22:01:30 an-c01n01 fenced[2218]: fence an-c01n02.alteeve.ca failed
			</source>

			This is expected and can be safely ignored. After a minute or so, the hung node should reboot. Once it's back online and rejoins the cluster, the cluster should go back to normal operation.

	= Using fence_sanlock And checkquorum.wdmd =		= Using fence_sanlock And checkquorum.wdmd =