AN!Cluster Recovery - Revision history

Digimer: /* Recovery Steps When Both Nodes Are Running */

2012-05-22T04:23:04Z

Recovery Steps When Both Nodes Are Running

← Older revision		Revision as of 04:23, 22 May 2012
Line 23:		Line 23:
	## On the top and bottom '''right''' terminals, run <span class="code">clear; tail -f -n 0 /var/log/messages</span>.		## On the top and bottom '''right''' terminals, run <span class="code">clear; tail -f -n 0 /var/log/messages</span>.
	# Run <span class="code">clustat</span> on the top and bottom '''left''' windows.		# Run <span class="code">clustat</span> on the top and bottom '''left''' windows.
	## Examine each VM (<span class="code">vm:vmXXXX-desc<span>.		## Examine each VM (<span class="code">vm:vmXXXX-desc</span>.
	## For each VM whose state is <span class="code">failed</span>, you need to disable and then enable it. The '''disable''' step is critical as that is how you tell the cluster that you have examined the problem and determined that the service is safe to recover.		## For each VM whose state is <span class="code">failed</span>, you need to disable and then enable it. The '''disable''' step is critical as that is how you tell the cluster that you have examined the problem and determined that the service is safe to recover.
	## Run: <span class="code">clusvcadm -d vm:vmXXXX-desc<span>, then re-run <span class="code">clustat<span>. Ensure the new state is <span class="code">stopped<span>.		## Run: <span class="code">clusvcadm -d vm:vmXXXX-desc</span>, then re-run <span class="code">clustat</span>. Ensure the new state is <span class="code">stopped</span>.
	## Run: <span class="code">clusvcadm -e vm:vmXXXX-desc<span>, then re-run <span class="code">clustat<span>. Ensure the new state is <span class="code">started<span>.		## Run: <span class="code">clusvcadm -e vm:vmXXXX-desc</span>, then re-run <span class="code">clustat</span>. Ensure the new state is <span class="code">started</span>.



	{{footer}}		{{footer}}

Digimer at 04:22, 22 May 2012

2012-05-22T04:22:14Z

← Older revision		Revision as of 04:22, 22 May 2012
Line 12:		Line 12:

	== Recovery Steps When Both Nodes Are Running ==		== Recovery Steps When Both Nodes Are Running ==

			{{note\|1=In this case, enabling the service should be performed on the known-good node. For this reason, the <span class="code">-m <target node></span> is intentionally left off the '''enable''' step. This will cause the VM to restart on the known-good node, assuming you make the call on the node which had not rebooted.}}

	This recovery procedure assumes that the rebooted node was restored.		This recovery procedure assumes that the rebooted node was restored.
Line 17:		Line 19:
	# Log into your cluster monitor machine (or a workstation with <span class="code">virt-manager</span>) installed.		# Log into your cluster monitor machine (or a workstation with <span class="code">virt-manager</span>) installed.
	## Open up four terminals [[2-Node Red Hat KVM Cluster Tutorial#Starting the Cluster for the First Time\|as described here]].		## Open up four terminals [[2-Node Red Hat KVM Cluster Tutorial#Starting the Cluster for the First Time\|as described here]].
	## On the top two terminals, [[ssh]] into the first node in the cluster. On the top-right ~~terminal~~, run <span class="code">clear; tail -f -n 0 /var/log/messages</span>.		## On the '''top''' two terminals, [[ssh]] into the '''first''' node in the cluster.
			## On the '''bottom''' two terminals, [[ssh]] into the '''second''' node in the cluster.
			## On the top and bottom '''right''' terminals, run <span class="code">clear; tail -f -n 0 /var/log/messages</span>.
			# Run <span class="code">clustat</span> on the top and bottom '''left''' windows.
			## Examine each VM (<span class="code">vm:vmXXXX-desc<span>.
			## For each VM whose state is <span class="code">failed</span>, you need to disable and then enable it. The '''disable''' step is critical as that is how you tell the cluster that you have examined the problem and determined that the service is safe to recover.
			## Run: <span class="code">clusvcadm -d vm:vmXXXX-desc<span>, then re-run <span class="code">clustat<span>. Ensure the new state is <span class="code">stopped<span>.
			## Run: <span class="code">clusvcadm -e vm:vmXXXX-desc<span>, then re-run <span class="code">clustat<span>. Ensure the new state is <span class="code">started<span>.



	{{footer}}		{{footer}}

Digimer: Created page with "{{howto_header}} {{note|1=This tutorial is designed for users and customers who are running AN!Clusters, based on the 2-Node Red Hat KVM Cluster Tutorial tutorial.}} This t..."

2012-05-22T03:06:56Z

Created page with "{{howto_header}} {{note|1=This tutorial is designed for users and customers who are running AN!Clusters, based on the 2-Node Red Hat KVM Cluster Tutorial tutorial.}} This t..."

Páàjì titun

{{howto_header}}

{{note|1=This tutorial is designed for users and customers who are running AN!Clusters, based on the [[2-Node Red Hat KVM Cluster Tutorial]] tutorial.}}

This tutorial is designed to provide very basic, to the point steps to guide you through recovery of an '''AN!Cluster''' after a fault. It is '''''not''''' designed as a primary education on this topic. For a proper understanding of the steps below, please take the time to study the [[2-Node Red Hat KVM Cluster Tutorial|main tutorial]].

= Node Was Rebooted Uncleanly Rebooted, VM(s) Offline =

If a node was rebooted using the reboot command, then it is possible for the VMs to have crashed in such a manner to leave them in the failed state. Once a VM goes into a failed state, it will not recover without human intervention. This section checks for and recovers from this scenario.

{{note|1=This section assumes you have a dedicate "cluster monitor" machine. If you don't, then use a machine running [[linux]] with [[virt-manager]] installed.

== Recovery Steps When Both Nodes Are Running ==

This recovery procedure assumes that the rebooted node was restored.

# Log into your cluster monitor machine (or a workstation with virt-manager) installed.
## Open up four terminals [[2-Node Red Hat KVM Cluster Tutorial#Starting the Cluster for the First Time|as described here]].
## On the top two terminals, [[ssh]] into the first node in the cluster. On the top-right terminal, run clear; tail -f -n 0 /var/log/messages.

{{footer}}