Anvil! Tutorial 3: Difference between revisions

From Alteeve Wiki
Jump to navigation Jump to search
 
(95 intermediate revisions by the same user not shown)
Line 3: Line 3:
{{warning|1=This tutorial is incomplete, flawed and generally sucks at this time. Do not follow this and expect anything to work. In large part, it's a dumping ground for notes and little else. This warning will be removed when the tutorial is completed.}}
{{warning|1=This tutorial is incomplete, flawed and generally sucks at this time. Do not follow this and expect anything to work. In large part, it's a dumping ground for notes and little else. This warning will be removed when the tutorial is completed.}}


{{warning|1=This tutorial is built on [[Red Hat]]'s Enterprise Linux 7 beta. [[Red Hat]] never confirms what a future release will contain until it is actually released, so there is a real chance that what is in the beta will '''not''' be in the final release.}}
This is the third '''Anvil!''' tutorial built on [[Red Hat]]'s Enterprise Linux 7. It marks the third generation of the [[Anvil!]] High-Availability Platform.


This is the third '''AN!Cluster''' tutorial built on [[Red Hat]]'s Enterprise Linux 7. It improves on the [[Red Hat Cluster Service 2 Tutorial|RHEL 5, RHCS stable 2]] and [[2-Node Red Hat KVM Cluster Tutorial|RHEL 6, RHCS stable3]] tutorials.
As with the previous tutorials, the end goal of this tutorial is an ''Anvil!'' platform for high-availability virtual servers. It's design attempts to remove all single points of failure from the system. Power and networking are made fully redundant in this version, along with minimizing the node failures which would lead to service interruption. This tutorial also covers the [[Striker]] dashboard and [[ScanCore]] monitoring and self-healing tools.  
 
As with the previous tutorials, the end goal of this tutorial is a 2-node cluster providing a platform for high-availability virtual servers. It's design attempts to remove all single points of failure from the system. Power and networking are made fully redundant in this version, along with minimizing the node failures which would lead to service interruption. This tutorial also covers the [[AN!Utilities]]; [[AN!Cluster Dashboard]], [[AN!Cluster Monitor]] and [[AN!Safe Cluster Shutdown]].  


As it the previous tutorial, [[KVM]] will be the hypervisor used for facilitating virtual machines. The old <span class="code">[[cman]]</span> and <span class="code">[[rgmanager]]</span> tools are replaced in favour of <span class="code">[[pacemaker]]</span> for resource management.  
As it the previous tutorial, [[KVM]] will be the hypervisor used for facilitating virtual machines. The old <span class="code">[[cman]]</span> and <span class="code">[[rgmanager]]</span> tools are replaced in favour of <span class="code">[[pacemaker]]</span> for resource management.  
Line 13: Line 11:
= Before We Begin =
= Before We Begin =


This tutorial '''does not''' require prior cluster experience, but it does expect familiarity with Linux and a low-intermediate understanding of networking. Where possible, steps are explained in detail and rationale is provided for why certain decisions are made.
This tutorial '''does not''' require prior Anvil! experience (or any clustering experience), but it does expect a certain familiarity with Linux and a low-intermediate understanding of networking. Where possible, steps are explained in detail and rationale is provided for why certain decisions are made.


'''For those with cluster experience''';
'''For those with Anvil! experience''';


Please be careful not to skip too much. There are some major and some subtle changes from previous tutorials.  
Please be careful not to skip too much. There are some major and some subtle changes from previous tutorials.  
Line 21: Line 19:
= OS Setup =
= OS Setup =


{{warning|1=I used Fedora 19 at this point, obviously things will change, possibly a lot, once RHEL 7 is released.}}
This tutorial assumes a <span class="code">minimal</span> install of either [[RHEL]] or [[CentOS]] version 7.


== Install ==
== Post OS Install ==


Not all of these are required, but most are used at one point or another in this tutorial.
{{note|1=With RHEL7, <span class="code">[[biosdevname]]</span> tries to give network devices predictable names. It's very likely that your initial device names will differ from those in this tutorial.}}


<syntaxhighlight lang="bash">
=== If you are running RHEL ===
yum install bridge-utils corosync net-tools ntp pacemaker pcs rsync syslinux wget fence-agents-all fence-agents-virsh
</syntaxhighlight>


Optional stuff:
Before you can download any packages, you will need to register your nodes with Red Hat's subscription manager;


<syntaxhighlight lang="bash">
{|class="wikitable"
yum install gpm man vim screen mlocate syslinux
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
subscription-manager register --username $username --password $password --auto-attach
subscription-manager repos --enable=rhel-ha-for-rhel-7-server-rpms
subscription-manager repos --enable=rhel-7-server-optional-rpms
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
The system has been registered with ID: 9c578d87-bd80-4637-9f41-6076efb9e20e


If you want to use your mouse at the node's terminal, run the following;
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:      Subscribed
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
subscription-manager register --username $username --password $password --auto-attach
subscription-manager repos --enable=rhel-ha-for-rhel-7-server-rpms
subscription-manager repos --enable=rhel-7-server-optional-rpms
</syntaxhighlight>
<syntaxhighlight lang="text">
The system has been registered with ID: a55c83e5-e4ec-4fcf-b7b7-b9455b3e07cf


<syntaxhighlight lang="bash">
Installed Product Current Status:
systemctl enable gpm.service
Product Name: Red Hat Enterprise Linux Server
systemctl start gpm.service
Status:      Subscribed
</syntaxhighlight>
</syntaxhighlight>
|}


== Setting the Hostname ==
== Adding LINBIT Repos ==
 
Fedora 19 is '''very''' different from [[EL6]].


{{note|1=The '<span class="code">--pretty</span>' line currently doesn't work as there is [https://bugzilla.redhat.com/show_bug.cgi?id=895299 a bug (rhbz#895299)] with single-quotes.}}
If you purchased full [https://my.linbit.com LINBIT support], you can add their repos in order to get DRBD 9 and associated tools.
{{note|1=The '<span class="code">--static</span>' option is currently needed to prevent the '<span class="code">.</span>' from being removed. See [https://bugzilla.redhat.com/show_bug.cgi?id=896756 this bug (rhbz#896756)].}}


Use a format that works for you. For the tutorial, node names are based on the following;
First, download their registration tool.
* A two-letter prefix identifying the company/user (<span class="code">an</span>, for "Alteeve's Niche!")
* A sequential cluster ID number in the form of <span class="code">cXX</span> (<span class="code">c01</span> for "Cluster 01", <span class="code">c02</span> for Cluster 02, etc)
* A sequential node ID number in the form of <span class="code">nYY</span>


In my case, this is my third cluster and I use the company prefix <span class="code">an</span>, so my two nodes will be;
{|class="wikitable"
* <span class="code">an-c03n01</span> - node 1
!<span class="code">an-a04n01</span>
* <span class="code">an-c03n02</span> - node 2
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
cd /root
wget https://my.linbit.com/linbit-manage-node.py
</syntaxhighlight>
<syntaxhighlight lang="text">
--2016-11-19 10:22:21--  https://my.linbit.com/linbit-manage-node.py
Resolving my.linbit.com (my.linbit.com)... 212.69.166.235
Connecting to my.linbit.com (my.linbit.com)|212.69.166.235|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26797 (26K) [application/x-python-script]
Saving to: ‘linbit-manage-node.py’


Folks who've read my earlier tutorials will note that this is a departure in naming. I find this method spans and scales much better. Further, it the simply required in order to use the [[AN!CDB|AN! Cluster Dashboard]].
100%[========================================================================================>] 26,797      --.-K/s  in 0.1s   


<syntaxhighlight lang="bash">
2016-11-19 10:22:21 (175 KB/s) - ‘linbit-manage-node.py’ saved [26797/26797]
hostnamectl set-hostname an-c03n01.alteeve.ca --static
</syntaxhighlight>
hostnamectl set-hostname --pretty "Alteeve's Niche! - Cluster 03, Node 01"
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
cd /root
wget https://my.linbit.com/linbit-manage-node.py
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
--2016-11-19 10:26:52--  https://my.linbit.com/linbit-manage-node.py
Resolving my.linbit.com (my.linbit.com)... 212.69.166.235
Connecting to my.linbit.com (my.linbit.com)|212.69.166.235|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26797 (26K) [application/x-python-script]
Saving to: ‘linbit-manage-node.py’


If you want the new host name to take effect immediately, you can use the traditional <span class="code">hostname</span> command:
100%[========================================================================================>] 26,797      --.-K/s  in 0.1s   


<syntaxhighlight lang="bash">
2016-11-19 10:26:53 (182 KB/s) - ‘linbit-manage-node.py’ saved [26797/26797]
hostname an-c03n01.alteeve.ca
</syntaxhighlight>
</syntaxhighlight>
|}


'''Alternatively'''
Make it executable.


If you have trouble with those commands, you can directly edit the files that contain the host names.
{|class="wikitable"
 
!<span class="code">an-a04n01</span>
The host name is stored in <span class="code">/etc/hostname</span>:
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 
chmod 755 linbit-manage-node.py
<syntaxhighlight lang="bash">
ls -lah linbit-manage-node.py
echo an-c03n01.alteeve.ca > /etc/hostname
cat /etc/hostname
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
an-c03n01.alteeve.ca
-rwxr-xr-x. 1 root root 27K Oct 11 05:54 linbit-manage-node.py
</syntaxhighlight>
</syntaxhighlight>
 
|-
The "pretty" host name is stored in <span class="code">/etc/machine-info</span> as the unquoted value for the <span class="code">PRETTY_HOSTNAME</span> value.
!<span class="code">an-a04n02</span>
 
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
chmod 755 linbit-manage-node.py
vim /etc/machine-info
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
PRETTY_HOSTNAME=Alteeves Niche! - Cluster 01, Node 01
-rwxr-xr-x. 1 root root 27K Oct 11 05:54 linbit-manage-node.py
</syntaxhighlight>
</syntaxhighlight>
|}


If you can't get the <span class="code">hostname</span> command to work for some reason, you can reboot to have the system read the new values.
{{note|1=If you get the error: '<span class="code">ERR: Could not detect MAC addresses of your node</span>', then the version of '<span class="code">linbit-manage-node.py</span>' does not yet recognise bridges or slaved interfaces in bonds. For now, you can download a [https://alteeve.ca/files/linbit-manage-node_anvil.py modified version from Alteeve] instead.}}


== Optional - Video Problems ==
Now run the tool interactively.


On my servers, [[Fedora]] 19 doesn't detect or use the video card properly. To resolve this, I need to add <span class="code">nomodeset</span> to the kernel line when installing and again after the install is complete.
{|class="wikitable"
 
!<span class="code">an-a04n01</span>
Once installed
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 
/root/linbit-manage-node.py
Edit the <span class="code">/etc/default/grub</span> and append <span class="code">nomodeset</span> to the end of the <span class="code">GRUB_CMDLINE_LINUX</span> variable.
</syntaxhighlight>
 
<syntaxhighlight lang="text">
<syntaxhighlight lang="bash">
linbit-manage-node.py (Version: 1.11)
vim /etc/default/grub
Checking if version is up to date
[OK] Your version is up to date
Username:
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
/root/linbit-manage-node.py
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
<syntaxhighlight lang="text">
GRUB_TIMEOUT=5
linbit-manage-node.py (Version: 1.11)
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
Checking if version is up to date
GRUB_DEFAULT=saved
[OK] Your version is up to date
GRUB_CMDLINE_LINUX="nomodeset rd.md=0 rd.lvm=0 rd.dm=0 $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rd.luks=0 vconsole.keymap=us nomodeset"
Username:  
GRUB_DISABLE_RECOVERY="true"
GRUB_THEME="/boot/grub2/themes/system/theme.txt"
</syntaxhighlight>
</syntaxhighlight>
|}


Save that. and then rewrite the [[grub2]] configuration file.
Enter the user name and password given to you by LINBIT when you registered with them.


<syntaxhighlight lang="bash">
{|class="wikitable"
grub2-mkconfig -o /boot/grub2/grub.cfg
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
Username: xxxxxx
Credential (will not be echoed):
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
[OK] Login successful
The following contracts are available:
Will this node form a cluster with...


Next time you reboot, you should get a stock 80x25 character display. It's not much, but it will work on esoteric video cards or weird monitors.
1) Contract: silver 2017-01-07 (ID: xxxx)


== What Security? ==
--> Please enter a number in range and press return:
 
</syntaxhighlight>
{{note|1=The final version of this tutorial '''will''' use the firewall and selinux. It's disabled to simplify debugging during the development stage of the tutorial only.}}
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
Username: xxxxxx
Credential (will not be echoed):
</syntaxhighlight>
<syntaxhighlight lang="text">
[OK] Login successful
The following contracts are available:
Will this node form a cluster with...


This section will be re-added at the end. For now;
1) Contract: silver 2017-01-07 (ID: xxxx)


<syntaxhighlight lang="bash">
--> Please enter a number in range and press return:
setenforce 0
sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
systemctl disable firewalld.service
systemctl stop firewalld.service
</syntaxhighlight>
</syntaxhighlight>
|}


== Network ==
If you have multiple contracts, select the number to the left of the contract identification. Otherwise, select '<span class="code">1</span>'.


We want static, named network devices. Follow this;
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
--> Please enter a number in range and press return: 1
</syntaxhighlight>
<syntaxhighlight lang="text">
Writing registration data:
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
--> Please enter a number in range and press return: 1
</syntaxhighlight>
<syntaxhighlight lang="text">
Writing registration data:
</syntaxhighlight>
|}


* [[Changing Ethernet Device Names in EL7 and Fedora 15+]]
Confirm that you want to write out the license file. Once you accept, you will be presented with a menu of which repositories you want to use from LINBIT. We're only going to enable the '<span class="field">drbd-9.0</span>' repo and leave the pacemaker repos disabled as we'll pull them from Red Hat.


Then, use these configuration files;
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
--> Write to file (/var/lib/drbd-support/registration.json)? [y/N]
</syntaxhighlight>
<syntaxhighlight lang="text">
  Here are the repositories you can enable:


Build the bridge;
    1) pacemaker-1.1.15(Disabled)
    2) pacemaker-1.1.12(Disabled)
    3) pacemaker-1.1(Disabled)
    4) drbd-9.0(Disabled)
    5) drbd-8.4(Disabled)


<syntaxhighlight lang="bash">
  Enter the number of the repository you wish to enable/disable. Hit 0 when you are done.
vim /etc/sysconfig/network-scripts/ifcfg-ifn-vbr1
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
<syntaxhighlight lang="text">
# Internet-Facing Network - Bridge
  Enable/Disable: 4
DEVICE="ifn-vbr1"
TYPE="Bridge"
BOOTPROTO="none"
IPADDR="10.255.10.1"
NETMASK="255.255.0.0"
GATEWAY="10.255.255.254"
DNS1="8.8.8.8"
DNS2="8.8.4.4"
DEFROUTE="yes"
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
  Here are the repositories you can enable:


Now build the bonds;
    1) pacemaker-1.1.15(Disabled)
    2) pacemaker-1.1.12(Disabled)
    3) pacemaker-1.1(Disabled)
    4) drbd-9.0(Enabled)
    5) drbd-8.4(Disabled)


<syntaxhighlight lang="bash">
  Enter the number of the repository you wish to enable/disable. Hit 0 when you are done.
vim /etc/sysconfig/network-scripts/ifcfg-ifn-bond1
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
<syntaxhighlight lang="text">
# Internet-Facing Network - Bond
  Enable/Disable: 0
DEVICE="ifn-bond1"
</syntaxhighlight>
BRIDGE="ifn-vbr1"
|-
BOOTPROTO="none"
!<span class="code">an-a04n02</span>
NM_CONTROLLED="no"
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
ONBOOT="yes"
--> Write to file (/var/lib/drbd-support/registration.json)? [y/N]
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=ifn1"
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
  Here are the repositories you can enable:


<syntaxhighlight lang="bash">
    1) pacemaker-1.1.15(Disabled)
vim /etc/sysconfig/network-scripts/ifcfg-sn-bond1
    2) pacemaker-1.1.12(Disabled)
    3) pacemaker-1.1(Disabled)
    4) drbd-9.0(Disabled)
    5) drbd-8.4(Disabled)
 
  Enter the number of the repository you wish to enable/disable. Hit 0 when you are done.
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
<syntaxhighlight lang="text">
# Storage Network - Bond
  Enable/Disable: 4
DEVICE="sn-bond1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=sn1"
IPADDR="10.10.10.1"
NETMASK="255.255.0.0"
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
  Here are the repositories you can enable:


<syntaxhighlight lang="bash">
    1) pacemaker-1.1.15(Disabled)
vim /etc/sysconfig/network-scripts/ifcfg-bcn-bond1
    2) pacemaker-1.1.12(Disabled)
    3) pacemaker-1.1(Disabled)
    4) drbd-9.0(Enabled)
    5) drbd-8.4(Disabled)
 
  Enter the number of the repository you wish to enable/disable. Hit 0 when you are done.
</syntaxhighlight>
<syntaxhighlight lang="text">
  Enable/Disable: 0
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
|}
# Back-Channel Network - Bond
 
DEVICE="bcn-bond1"
{{warning|1=The repository will include a node-specific hash string in the '<span class="code">baseurl</span>'. Keep this private!}}
BOOTPROTO="none"
 
NM_CONTROLLED="no"
Once you select '<span class="code">0</span>' to exit that menu, a summary of the repo will be displayed and you will be asked if you want to save it or not.
ONBOOT="yes"
 
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=bcn1"
{|class="wikitable"
IPADDR="10.20.10.1"
!<span class="code">an-a04n01</span>
NETMASK="255.255.0.0"
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
Writing repository config:
Content:
[drbd-8.4]
name=LINBIT Packages for drbd-8.4 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/drbd-8.4/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1
 
[pacemaker-1.1]
name=LINBIT Packages for pacemaker-1.1 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1
 
[pacemaker-1.1.15]
name=LINBIT Packages for pacemaker-1.1.15 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1.15/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1
 
[pacemaker-1.1.12]
name=LINBIT Packages for pacemaker-1.1.12 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1.12/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1
 
[drbd-9.0]
name=LINBIT Packages for drbd-9.0 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/drbd-9.0/$basearch
enabled=1
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1
</syntaxhighlight>
<syntaxhighlight lang="text">
--> Write to file (/etc/yum.repos.d/linbit.repo)? [y/N] y
</syntaxhighlight>
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
Writing repository config:
Content:
[drbd-8.4]
name=LINBIT Packages for drbd-8.4 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/drbd-8.4/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1


Now tell the interfaces to be slaves to their bonds;
[pacemaker-1.1]
name=LINBIT Packages for pacemaker-1.1 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1
 
[pacemaker-1.1.15]
name=LINBIT Packages for pacemaker-1.1.15 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1.15/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1


Internet-Facing Network;
[pacemaker-1.1.12]
name=LINBIT Packages for pacemaker-1.1.12 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1.12/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1


<syntaxhighlight lang="bash">
[drbd-9.0]
vim /etc/sysconfig/network-scripts/ifcfg-ifn1
name=LINBIT Packages for drbd-9.0 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/drbd-9.0/$basearch
enabled=1
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
<syntaxhighlight lang="text">
# Internet-Facing Network - Link 1
--> Write to file (/etc/yum.repos.d/linbit.repo)? [y/N] y
DEVICE="ifn1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="ifn-bond1"
</syntaxhighlight>
</syntaxhighlight>
|}
When you accept, it will download the yum plugins and then ask you if you want to save their PGP key.


<syntaxhighlight lang="bash">
{|class="wikitable"
vim /etc/sysconfig/network-scripts/ifcfg-ifn2
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="text">
[OK] Repository configuration written
Downloading LINBIT yum plugin
Downloading LINBIT yum plugin config
Final Notes:
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
<syntaxhighlight lang="text">
# Back-Channel Network - Link 2
--> Add linbit signing key to keyring now? [y/N] y
DEVICE="ifn2"
</syntaxhighlight>
NM_CONTROLLED="no"
<syntaxhighlight lang="text">
BOOTPROTO="none"
Now update your package information and install
ONBOOT="yes"
LINBIT's kernel module and/or user space utilities
SLAVE="yes"
[OK] Congratulations! Your node was successfully configured.
MASTER="ifn-bond1"
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="text">
[OK] Repository configuration written
Downloading LINBIT yum plugin
Downloading LINBIT yum plugin config
Final Notes:
</syntaxhighlight>
<syntaxhighlight lang="text">
--> Add linbit signing key to keyring now? [y/N] y
</syntaxhighlight>
<syntaxhighlight lang="text">
Now update your package information and install
LINBIT's kernel module and/or user space utilities
[OK] Congratulations! Your node was successfully configured.
</syntaxhighlight>
</syntaxhighlight>
|}
Done!
== Install ==


Storage Network;
Not all of these are required, but most are used at one point or another in this tutorial.
 
{{note|1=The <span class="code">fence-agents-virsh</span> package is not available in RHEL 7 beta. Further, it's only needed if you're building your Anvil! using VMs.}}


<syntaxhighlight lang="bash">
{|class="wikitable"
vim /etc/sysconfig/network-scripts/ifcfg-sn1
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
yum install rsync pacemaker bridge-utils ntp corosync pcs wget gpm man vim screen mlocate syslinux bzip2 \
            openssh-clients fence-agents-all fence-agents-virsh policycoreutils-python drbd drbd-bash-completion \
            drbd-pacemaker drbd-udev drbd-utils drbdmanage
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
|-
# Storage Network - Link 1
!<span class="code">an-a04n02</span>
DEVICE="sn1"
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
NM_CONTROLLED="no"
<same>
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="sn-bond1"
</syntaxhighlight>
</syntaxhighlight>
|}


<syntaxhighlight lang="bash">
 
vim /etc/sysconfig/network-scripts/ifcfg-sn2
== Making ssh faster when the net is down ==
 
By default, the nodes will try to resolve the host name of an incoming ssh connection. When the internet connection is down, DNS lookups have to time out, which can make login times quite slow. When something goes wrong, seconds count and waiting for up to a minute for an SSH password prompt can be maddening.
 
For this reason, we will make two changes to <span class="code">/etc/ssh/sshd_config</span> that disable this login delay.
 
Please be aware that this can reduce security. If this is a concern, skip this step.
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
sed -i.anvil 's/#GSSAPIAuthentication no/GSSAPIAuthentication no/' /etc/ssh/sshd_config
sed -i 's/GSSAPIAuthentication yes/#GSSAPIAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
systemctl restart sshd.service
diff -u /etc/ssh/sshd_config.anvil /etc/ssh/sshd_config
</syntaxhighlight>
<syntaxhighlight lang="diff">
--- /etc/ssh/sshd_config.anvil 2014-06-09 21:15:52.000000000 -0400
+++ /etc/ssh/sshd_config 2014-07-27 08:41:03.296760761 -0400
@@ -89,8 +89,8 @@
#KerberosUseKuserok yes
# GSSAPI options
-#GSSAPIAuthentication no
-GSSAPIAuthentication yes
+GSSAPIAuthentication no
+#GSSAPIAuthentication yes
#GSSAPICleanupCredentials yes
GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
@@ -127,7 +127,7 @@
#ClientAliveInterval 0
#ClientAliveCountMax 3
#ShowPatchLevel no
-#UseDNS yes
+UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
same
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
<syntaxhighlight lang="text">
# Storage Network - Link 1
same
DEVICE="sn2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="sn-bond1"
</syntaxhighlight>
</syntaxhighlight>
|}


Back-Channel Network
Subsequent logins when the net is down should be quick.
 
=== Configuring the network ===
 
If you want to make any other changes, like configuring the interface to have a static IP, do so now. Once you're done editing;


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
vim /etc/sysconfig/network-scripts/ifcfg-bcn1
nmcli connection reload
</syntaxhighlight>
systemctl restart NetworkManager.service
<syntaxhighlight lang="bash">
ip addr show
# Back-Channel Network - Link 1
DEVICE="bcn1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="bcn-bond1"
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:a7:9d:17 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.201/24 scope global eth0
      valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fea7:9d17/64 scope link
      valid_lft forever preferred_lft forever
</syntaxhighlight>
The interface should now start on boot properly.
== Setting the Hostname ==


<syntaxhighlight lang="bash">
Fedora 19 is '''very''' different from [[EL6]].
vim /etc/sysconfig/network-scripts/ifcfg-bcn2
</syntaxhighlight>
<syntaxhighlight lang="bash">
# Storage Network - Link 1
DEVICE="bcn2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="bcn-bond1"
</syntaxhighlight>


Now restart the network, confirm that the bonds and bridge are up and you are ready to proceed.
{{note|1=The '<span class="code">--pretty</span>' line currently doesn't work as there is [https://bugzilla.redhat.com/show_bug.cgi?id=895299 a bug (rhbz#895299)] with single-quotes.}}
{{note|1=The '<span class="code">--static</span>' option is currently needed to prevent the '<span class="code">.</span>' from being removed. See [https://bugzilla.redhat.com/show_bug.cgi?id=896756 this bug (rhbz#896756)].}}


== Setup The hosts File ==
Use a format that works for you. For the tutorial, node names are based on the following;
* A two-letter prefix identifying the company/user (<span class="code">an</span>, for "Alteeve's Niche!")
* A sequential Anvil! ID number in the form of <span class="code">aXX</span> (<span class="code">a01</span> for "Anvil! 01", <span class="code">a02</span> for Anvil! 02, etc)
* A sequential node ID number in the form of <span class="code">nYY</span>


You can use [[DNS]] if you prefer. For now, lets use <span class="code">/etc/hosts</span> for node name resolution.
In our case, this is my third Anvil! and we use the company prefix <span class="code">an</span>, so these two nodes will be;
* <span class="code">an-a04n01</span> - node 1
* <span class="code">an-a04n02</span> - node 2


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
vim /etc/hosts
hostnamectl set-hostname an-a04n01.alteeve.ca --static
hostnamectl set-hostname --pretty "Alteeve's Niche! - Anvil! 03, Node 01"
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
127.0.0.1  localhost localhost.localdomain localhost4 localhost4.localdomain4
::1        localhost localhost.localdomain localhost6 localhost6.localdomain6


# AN!Cluster 01, Node 01
If you want the new host name to take effect immediately, you can use the traditional <span class="code">hostname</span> command:
10.255.10.1    an-c01n01.ifn
10.10.10.1      an-c01n01.sn
10.20.10.1      an-c01n01.bcn an-c01n01 an-c01n01.alteeve.ca
10.20.11.1      an-c01n01.ipmi


# AN!Cluster 01, Node 02
<syntaxhighlight lang="bash">
10.255.10.2    an-c01n02.ifn
hostname an-a04n01.alteeve.ca
10.10.10.2      an-c01n02.sn
10.20.10.2      an-c01n02.bcn an-c01n02 an-c01n02.alteeve.ca
10.20.11.2      an-c01n02.ipmi
 
# Foundation Pack
10.20.2.7      an-p03 an-p03.alteeve.ca
</syntaxhighlight>
</syntaxhighlight>


== Setup SSH ==
The "pretty" host name is stored in <span class="code">/etc/machine-info</span> as the unquoted value for the <span class="code">PRETTY_HOSTNAME</span> value.
 
Same as [[AN!Cluster_Tutorial_2#Setting_up_SSH|before]].
 
== Populating And Pushing ~/.ssh/known_hosts ==
 
Same as [[2-Node_Red_Hat_KVM_Cluster_Tutorial#Populating_And_Pushing_~/.ssh/known_hosts|before]].


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
ssh root@an-c03n01.alteeve.ca
vim /etc/machine-info
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
The authenticity of host 'an-c03n01.alteeve.ca (10.20.30.1)' can't be established.
PRETTY_HOSTNAME=Alteeves Niche! - Anvil! 03, Node 01
RSA key fingerprint is 7b:dd:0d:aa:c5:f5:9e:a6:b6:4d:40:69:d6:80:4d:09.
Are you sure you want to continue connecting (yes/no)?
</syntaxhighlight>
</syntaxhighlight>


Type <span class="code">yes</span>
If you can't get the <span class="code">hostname</span> command to work for some reason, you can reboot to have the system read the new values.
 
== Network ==
 
{{note|1=(Note for myself) - Consider using '<span class="code">[https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sec-Specific_Kernel_Module_Capabilities.html#sec-Using_Channel_Bonding primary_reselect=1]</span>.}}


<syntaxhighlight lang="text">
We want static, named network devices. Follow this;
Are you sure you want to continue connecting (yes/no)? yes
</syntaxhighlight>
<syntaxhighlight lang="text">
Warning: Permanently added 'an-c03n01.alteeve.ca,10.20.30.1' (RSA) to the list of known hosts.
Last login: Thu Feb 14 15:18:33 2013 from 10.20.5.100
</syntaxhighlight>


You will now be logged into the <span class="code">an-c03n01</span> node, which in this case is the same machine on a new session in the same terminal.
* [[Changing Ethernet Device Names in EL7 and Fedora 15+]]


<syntaxhighlight lang="text">
Then, use these configuration files;
[root@an-c03n01 ~]#
</syntaxhighlight>


You can logout by typing <span class="code">exit</span>.
Build the bridge;


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
exit
vim /etc/sysconfig/network-scripts/ifcfg-ifn_bridge1
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="bash">
logout
# Internet-Facing Network - Bridge
Connection to an-c03n01.alteeve.ca closed.
DEVICE="ifn_bridge1"
TYPE="Bridge"
BOOTPROTO="none"
IPADDR="10.255.40.1"
NETMASK="255.255.0.0"
GATEWAY="10.255.255.254"
DNS1="8.8.8.8"
DNS2="8.8.4.4"
DEFROUTE="yes"
</syntaxhighlight>
</syntaxhighlight>


Now we have to repeat the steps for all the other variations on the names of the hosts. This is annoying and tedious, sorry.
Now build the bonds;


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
ssh root@an-c03n01
vim /etc/sysconfig/network-scripts/ifcfg-ifn_bond1
ssh root@an-c03n01.bcn
ssh root@an-c03n01.sn
ssh root@an-c03n01.ifn
ssh root@an-c03n02.alteeve.ca
ssh root@an-c03n02
ssh root@an-c03n02.bcn
ssh root@an-c03n02.sn
ssh root@an-c03n02.ifn
</syntaxhighlight>
</syntaxhighlight>
Your <span class="code">~/.ssh/known_hosts</span> file will now be populated with both nodes' ssh fingerprints. Copy it over to the second node to save all that typing a second time.
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
rsync -av ~/.ssh/known_hosts root@an-c03n02:/root/.ssh/
# Internet-Facing Network - Bond
DEVICE="ifn_bond1"
BRIDGE="ifn_bridge1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 primary=ifn_link1 updelay=120000 downdelay=0 fail_over_mac=none miimon=100 primary_reselect=better resend_igmp=5"
</syntaxhighlight>
</syntaxhighlight>
== Keeping Time in Sync ==
It's not as critical as it used to be to keep the clocks on the nodes in sync, but it's still a good idea.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl start ntpd.service
vim /etc/sysconfig/network-scripts/ifcfg-sn_bond1
systemctl enable ntpd.service
</syntaxhighlight>
<syntaxhighlight lang="bash">
# Storage Network - Bond
DEVICE="sn_bond1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 primary=sn_link1 updelay=120000 downdelay=0 fail_over_mac=none miimon=100 primary_reselect=better resend_igmp=5"
IPADDR="10.10.40.1"
NETMASK="255.255.0.0"
</syntaxhighlight>
</syntaxhighlight>
== Configuring IPMI ==
F19 specifics based on the [[IPMI]] tutorial.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
yum -y install ipmitools OpenIPMI
vim /etc/sysconfig/network-scripts/ifcfg-bcn_bond1
systemctl start ipmi.service
systemctl enable ipmi.service
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="bash">
ln -s '/usr/lib/systemd/system/ipmi.service' '/etc/systemd/system/multi-user.target.wants/ipmi.service'
# Back-Channel Network - Bond
DEVICE="bcn_bond1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 primary=bcn_link1 updelay=120000 downdelay=0 fail_over_mac=none miimon=100 primary_reselect=better resend_igmp=5"
IPADDR="10.20.40.1"
NETMASK="255.255.0.0"
</syntaxhighlight>
</syntaxhighlight>


Our servers use lan channel 2, yours might be 1 or something else. Experiment.
Now tell the interfaces to be slaves to their bonds;
 
Internet-Facing Network;


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
ipmitool lan print 2
vim /etc/sysconfig/network-scripts/ifcfg-ifn_link1
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="bash">
Set in Progress        : Set Complete
# Internet-Facing Network - Link 1
Auth Type Support      : NONE MD5 PASSWORD
DEVICE="ifn_link1"
Auth Type Enable        : Callback : NONE MD5 PASSWORD
NM_CONTROLLED="no"
                        : User    : NONE MD5 PASSWORD
BOOTPROTO="none"
                        : Operator : NONE MD5 PASSWORD
ONBOOT="yes"
                        : Admin    : NONE MD5 PASSWORD
SLAVE="yes"
                        : OEM      : NONE MD5 PASSWORD
MASTER="ifn_bond1"
IP Address Source      : BIOS Assigned Address
</syntaxhighlight>
IP Address              : 10.20.51.1
 
Subnet Mask            : 255.255.0.0
<syntaxhighlight lang="bash">
MAC Address            : 00:19:99:9a:d8:e8
vim /etc/sysconfig/network-scripts/ifcfg-ifn_link2
SNMP Community String  : public
</syntaxhighlight>
IP Header              : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
<syntaxhighlight lang="bash">
Default Gateway IP      : 10.20.255.254
# Internet-Facing Network - Link 2
802.1q VLAN ID          : Disabled
DEVICE="ifn_link2"
802.1q VLAN Priority    : 0
NM_CONTROLLED="no"
RMCP+ Cipher Suites    : 0,1,2,3,6,7,8,17
BOOTPROTO="none"
Cipher Suite Priv Max  : OOOOOOOOXXXXXXX
ONBOOT="yes"
                        :    X=Cipher Suite Unused
SLAVE="yes"
                        :    c=CALLBACK
MASTER="ifn_bond1"
                        :    u=USER
                        :    o=OPERATOR
                        :    a=ADMIN
                        :    O=OEM
</syntaxhighlight>
</syntaxhighlight>


I need to set the IPs to <span class="code">10.20.31.1/16</span> and <span class="code">10.20.31.2/16</span> for nodes 1 and 2, respectively. I also want to set the password to <span class="code">secret</span> for the <span class="code">admin</span> user.
Storage Network;


'''Node 01''' IP;
<syntaxhighlight lang="bash">
vim /etc/sysconfig/network-scripts/ifcfg-sn_link1
</syntaxhighlight>
<syntaxhighlight lang="bash">
# Storage Network - Link 1
DEVICE="sn_link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="sn_bond1"
</syntaxhighlight>


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
ipmitool lan set 2 ipsrc static
vim /etc/sysconfig/network-scripts/ifcfg-sn_link2
ipmitool lan set 2 ipaddr 10.20.31.
ipmitool lan set 2 netmask 255.255.0.0
ipmitool lan set 2 defgw ipaddr 10.20.255.254
ipmitool lan print 2
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="bash">
Set in Progress        : Set Complete
# Storage Network - Link 2
Auth Type Support      : NONE MD5 PASSWORD
DEVICE="sn_link2"
Auth Type Enable        : Callback : NONE MD5 PASSWORD
NM_CONTROLLED="no"
                        : User    : NONE MD5 PASSWORD
BOOTPROTO="none"
                        : Operator : NONE MD5 PASSWORD
ONBOOT="yes"
                        : Admin    : NONE MD5 PASSWORD
SLAVE="yes"
                        : OEM      : NONE MD5 PASSWORD
MASTER="sn_bond1"
IP Address Source      : Static Address
</syntaxhighlight>
IP Address              : 10.20.31.1
 
Subnet Mask            : 255.255.0.0
Back-Channel Network
MAC Address            : 00:19:99:9a:d8:e8
 
SNMP Community String  : public
<syntaxhighlight lang="bash">
IP Header              : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
vim /etc/sysconfig/network-scripts/ifcfg-bcn_link1
Default Gateway IP      : 10.20.255.254
</syntaxhighlight>
802.1q VLAN ID          : Disabled
<syntaxhighlight lang="bash">
802.1q VLAN Priority    : 0
# Back-Channel Network - Link 1
RMCP+ Cipher Suites    : 0,1,2,3,6,7,8,17
DEVICE="bcn_link1"
Cipher Suite Priv Max  : OOOOOOOOXXXXXXX
NM_CONTROLLED="no"
                        :    X=Cipher Suite Unused
BOOTPROTO="none"
                        :    c=CALLBACK
ONBOOT="yes"
                        :    u=USER
SLAVE="yes"
                        :    o=OPERATOR
MASTER="bcn_bond1"
                        :    a=ADMIN
                        :    O=OEM
</syntaxhighlight>
</syntaxhighlight>
'''Node 01''' IP;


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
ipmitool lan set 2 ipsrc static
vim /etc/sysconfig/network-scripts/ifcfg-bcn_link2
ipmitool lan set 2 ipaddr 10.20.31.2
ipmitool lan set 2 netmask 255.255.0.0
ipmitool lan set 2 defgw ipaddr 10.20.255.254
ipmitool lan print 2
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="bash">
Set in Progress        : Set Complete
# Back-Channel Network - Link 2
Auth Type Support      : NONE MD5 PASSWORD
DEVICE="bcn_link2"
Auth Type Enable        : Callback : NONE MD5 PASSWORD
NM_CONTROLLED="no"
                        : User    : NONE MD5 PASSWORD
BOOTPROTO="none"
                        : Operator : NONE MD5 PASSWORD
ONBOOT="yes"
                        : Admin    : NONE MD5 PASSWORD
SLAVE="yes"
                        : OEM      : NONE MD5 PASSWORD
MASTER="bcn_bond1"
IP Address Source      : Static Address
IP Address              : 10.20.31.2
Subnet Mask            : 255.255.0.0
MAC Address            : 00:19:99:9a:b1:78
SNMP Community String  : public
IP Header              : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP      : 10.20.255.254
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites    : 0,1,2,3,6,7,8,17
Cipher Suite Priv Max  : OOOOOOOOXXXXXXX
                        :    X=Cipher Suite Unused
                        :    c=CALLBACK
                        :    u=USER
                        :    o=OPERATOR
                        :    a=ADMIN
                        :    O=OEM
</syntaxhighlight>
</syntaxhighlight>


Set the password.
Now restart the network, confirm that the bonds and bridge are up and you are ready to proceed.
 
== Setup The hosts File ==
 
You can use [[DNS]] if you prefer. For now, lets use <span class="code">/etc/hosts</span> for node name resolution.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
ipmitool user list 2
vim /etc/hosts
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
ID  Name     Callin  Link Auth IPMI Msg  Channel Priv Limit
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
1                   true    true      true      Unknown (0x00)
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
2  admin            true    true      true      OEM
Get User Access command failed (channel 2, user 3): Unknown (0x32)
</syntaxhighlight>
 
(ignore the error, it's harmless... *BOOM*)


We want to set <span class="code">admin</span>'s password, so we do:
# Anvil! 03, Node 01
10.255.40.1 an-a04n01.ifn
10.10.40.1 an-a04n01.sn
10.20.40.1 an-a04n01.bcn an-a04n01 an-a04n01.alteeve.ca
10.20.41.1 an-a04n01.ipmi


{{note|1=The <span class="code">2</span> below is the ID number, not the LAN channel.}}
# Anvil! 03, Node 02
10.255.40.2 an-a04n02.ifn
10.10.40.2 an-a04n02.sn
10.20.40.2 an-a04n02.bcn an-a04n02 an-a04n02.alteeve.ca
10.20.41.2 an-a04n02.ipmi


<syntaxhighlight lang="bash">
# Foundation Pack
ipmitool user set password 2 secret
### Foundation Pack
# Network Switches
10.20.1.1 an-switch01 an-switch01.alteeve.ca
10.20.1.2 an-switch02 an-switch02.alteeve.ca # Only accessible when out of the stack
# Switched PDUs
10.20.2.1 an-pdu01 an-pdu01.alteeve.ca
10.20.2.2 an-pdu02 an-pdu02.alteeve.ca
# Network-monitored UPSes
10.20.3.1 an-ups01 an-ups01.alteeve.ca
10.20.3.2 an-ups02 an-ups02.alteeve.ca
### Monitor Packs
10.20.4.1 an-striker01 an-striker01.alteeve.ca
10.255.4.1 an-striker01.ifn
10.20.4.2 an-striker02 an-striker02.alteeve.ca
10.255.4.2 an-striker02.ifn
</syntaxhighlight>
</syntaxhighlight>


Done!
== Setup SSH ==


= Configuring the Cluster =
Same as [[AN!Cluster_Tutorial_2#Setting_up_SSH|before]].


Now we're getting down to business!
== Populating And Pushing ~/.ssh/known_hosts ==


For this section, we will be working on <span class="code">an-c03n01</span> and using [[ssh]] to perform tasks on <span class="code">an-c03n02</span>.


{{note|1=TODO: explain what this is and how it works.}}
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
ssh-keygen -t rsa -N "" -b 8191 -f ~/.ssh/id_rsa
</syntaxhighlight>
<syntaxhighlight lang="text">
Generating public/private rsa key pair.


== Enable the pcs Daemon ==
Your identification has been saved in /root/.ssh/id_rsa.
 
Your public key has been saved in /root/.ssh/id_rsa.pub.
{{note|1=Most of this section comes more or less verbatim from the main [http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Clusters_from_Scratch/index.html Clusters from Scratch] tutorial.}}
The key fingerprint is:
 
be:17:cc:23:8e:b1:b4:76:a1:e4:2a:91:cb:cd:d8:3a root@an-a04n01.alteeve.ca
We will use [[pcs]], the Pacemaker Configuration System, to configure our cluster.
The key's randomart image is:
 
+--[ RSA 8191]----+
<syntaxhighlight lang="bash">
|                |
systemctl start pcsd.service
|                |
systemctl enable pcsd.service
|                |
|                |
|  .    So      |
|  o  +.o =      |
| . B + B.o o    |
|  E + B o..     |
|  .+.o ...      |
+-----------------+
</syntaxhighlight>
|-
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
ssh-keygen -t rsa -N "" -b 8191 -f ~/.ssh/id_rsa
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
ln -s '/usr/lib/systemd/system/pcsd.service' '/etc/systemd/system/multi-user.target.wants/pcsd.service'
Generating public/private rsa key pair.
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
71:b1:9d:31:9f:7a:c9:10:74:e0:4c:69:53:8f:e4:70 root@an-a04n02.alteeve.ca
The key's randomart image is:
+--[ RSA 8191]----+
|          ..O+E  |
|          B+% + |
|        . o.*.= .|
|        o  + . |
|        S  . +  |
|            .   |
|                |
|                |
|                |
+-----------------+
</syntaxhighlight>
</syntaxhighlight>
|}


Now we need to set a password for the <span class="code">hacluster</span> user. This is the account used by <span class="code">pcs</span> on one node to talk to the <span class="code">pcs</span> [[daemon]] on the other node. For this tutorial, we will use the password <span class="code">secret</span>. You will want to use [https://xkcd.com/936/ a stronger password], of course.
Setup autorized_keys:


<syntaxhighlight lang="bash">
{|class="wikitable"
echo secret | passwd --stdin hacluster
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh root@an-a04n02 "cat /root/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys
rsync -av ~/.ssh/authorized_keys root@an-a04n02:/root/.ssh/
ssh-keyscan an-a04n01.alteeve.ca >> ~/.ssh/known_hosts
ssh-keyscan an-a04n01 >> ~/.ssh/known_hosts
ssh-keyscan an-a04n01.bcn >> ~/.ssh/known_hosts
ssh-keyscan an-a04n01.sn >> ~/.ssh/known_hosts
ssh-keyscan an-a04n01.ifn >> ~/.ssh/known_hosts
ssh-keyscan an-a04n02.alteeve.ca >> ~/.ssh/known_hosts
ssh-keyscan an-a04n02 >> ~/.ssh/known_hosts
ssh-keyscan an-a04n02.bcn >> ~/.ssh/known_hosts
ssh-keyscan an-a04n02.sn >> ~/.ssh/known_hosts
ssh-keyscan an-a04n02.ifn >> ~/.ssh/known_hosts
rsync -av ~/.ssh/known_hosts root@an-a04n02:/root/.ssh/
rsync -av /etc/hosts root@an-a04n02:/etc/
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
|-
Changing password for user hacluster.
!<span class="code">an-a04n01</span>
passwd: all authentication tokens updated successfully.
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
</syntaxhighlight>
</syntaxhighlight>
|}


== Initializing the Cluster ==
== Keeping Time in Sync ==


One of the biggest reasons we're using the [[pcs]] tool, over something like [[crm]], is that it has been written to simplify the setup of clusters on [[Red Hat]] style operating systems. It will configure [[corosync]] automatically.
It's not as critical as it used to be to keep the clocks on the nodes in sync, but it's still a good idea.


First, we need to know what <span class="code">hostname</span> we will need to use for <span class="code">[[pcs]]</span>.
<syntaxhighlight lang="bash">
ln -sf /usr/share/zoneinfo/America/Toronto /etc/localtime
systemctl start ntpd.service
systemctl enable ntpd.service
</syntaxhighlight>


'''Node 01''':
== Configuring IPMI ==


<syntaxhighlight lang="bash">
F19 specifics based on the [[IPMI]] tutorial.
hostname
</syntaxhighlight>
<syntaxhighlight lang="bash">
an-c03n01.alteeve.ca
</syntaxhighlight>
 
'''Node 02''':


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
hostname
yum -y install ipmitools OpenIPMI
systemctl start ipmi.service
systemctl enable ipmi.service
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
<syntaxhighlight lang="text">
an-c03n02.alteeve.ca
ln -s '/usr/lib/systemd/system/ipmi.service' '/etc/systemd/system/multi-user.target.wants/ipmi.service'
</syntaxhighlight>
</syntaxhighlight>


Next, authenticate against the cluster nodes.
Our servers use lan channel 2, yours might be 1 or something else. Experiment.
 
'''Both nodes''':


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
pcs cluster auth an-c03n01.alteeve.ca an-c03n02.alteeve.ca -u hacluster
ipmitool lan print 2
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
Set in Progress        : Set Complete
Auth Type Support      : NONE MD5 PASSWORD
Auth Type Enable        : Callback : NONE MD5 PASSWORD
                        : User    : NONE MD5 PASSWORD
                        : Operator : NONE MD5 PASSWORD
                        : Admin    : NONE MD5 PASSWORD
                        : OEM      : NONE MD5 PASSWORD
IP Address Source      : BIOS Assigned Address
IP Address              : 10.20.41.1
Subnet Mask            : 255.255.0.0
MAC Address            : 00:19:99:9a:d8:e8
SNMP Community String  : public
IP Header              : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP      : 10.20.255.254
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites    : 0,1,2,3,6,7,8,17
Cipher Suite Priv Max  : OOOOOOOOXXXXXXX
                        :    X=Cipher Suite Unused
                        :    c=CALLBACK
                        :    u=USER
                        :    o=OPERATOR
                        :    a=ADMIN
                        :    O=OEM
</syntaxhighlight>
I need to set the IPs to <span class="code">10.20.41.1/16</span> and <span class="code">10.20.41.2/16</span> for nodes 1 and 2, respectively. I also want to set the password to <span class="code">secret</span> for the <span class="code">admin</span> user.


This will ask you for the user name and password. The default user name is <span class="code">hacluster</span> and we set the password to <span class="code">secret</span>.
'''Node 01''' IP;
 
<syntaxhighlight lang="text">
Password:
an-c03n02.alteeve.ca: Authorized
</syntaxhighlight>
 
'''Do this on one node only''':
 
Now to initialize the cluster's communication and membership layer.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
pcs cluster setup --name an-cluster-03 an-c03n01.alteeve.ca an-c03n02.alteeve.ca
ipmitool lan set 2 ipsrc static
ipmitool lan set 2 ipaddr 10.20.41.1
ipmitool lan set 2 netmask 255.255.0.0
ipmitool lan set 2 defgw ipaddr 10.20.255.254
ipmitool lan print 2
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
an-c03n01.alteeve.ca: Succeeded
Set in Progress        : Set Complete
an-c03n02.alteeve.ca: Succeeded
Auth Type Support      : NONE MD5 PASSWORD
</syntaxhighlight>
Auth Type Enable        : Callback : NONE MD5 PASSWORD
 
                        : User    : NONE MD5 PASSWORD
This will create the corosync configuration file <span class="code">/etc/corosync/corosync.conf</span>;
                        : Operator : NONE MD5 PASSWORD
 
                        : Admin    : NONE MD5 PASSWORD
<syntaxhighlight lang="bash">
                        : OEM      : NONE MD5 PASSWORD
cat /etc/corosync/corosync.conf
IP Address Source      : Static Address
</syntaxhighlight>
IP Address              : 10.20.41.1
<syntaxhighlight lang="text">
Subnet Mask            : 255.255.0.0
totem {
MAC Address            : 00:19:99:9a:d8:e8
version: 2
SNMP Community String  : public
secauth: off
IP Header              : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
cluster_name: an-cluster-03
Default Gateway IP      : 10.20.255.254
transport: udpu
802.1q VLAN ID          : Disabled
}
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites    : 0,1,2,3,6,7,8,17
Cipher Suite Priv Max  : OOOOOOOOXXXXXXX
                        :    X=Cipher Suite Unused
                        :    c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
</syntaxhighlight>
 
'''Node 01''' IP;


nodelist {
<syntaxhighlight lang="bash">
  node {
ipmitool lan set 2 ipsrc static
         ring0_addr: an-c03n01.alteeve.ca
ipmitool lan set 2 ipaddr 10.20.41.2
        nodeid: 1
ipmitool lan set 2 netmask 255.255.0.0
      }
ipmitool lan set 2 defgw ipaddr 10.20.255.254
   node {
ipmitool lan print 2
        ring0_addr: an-c03n02.alteeve.ca
</syntaxhighlight>
        nodeid: 2
<syntaxhighlight lang="text">
      }
Set in Progress         : Set Complete
}
Auth Type Support      : NONE MD5 PASSWORD
 
Auth Type Enable        : Callback : NONE MD5 PASSWORD
quorum {
                        : User    : NONE MD5 PASSWORD
provider: corosync_votequorum
                        : Operator : NONE MD5 PASSWORD
}
                        : Admin    : NONE MD5 PASSWORD
 
                        : OEM      : NONE MD5 PASSWORD
logging {
IP Address Source      : Static Address
to_syslog: yes
IP Address              : 10.20.41.2
}
Subnet Mask            : 255.255.0.0
MAC Address            : 00:19:99:9a:b1:78
SNMP Community String   : public
IP Header              : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP      : 10.20.255.254
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites    : 0,1,2,3,6,7,8,17
Cipher Suite Priv Max  : OOOOOOOOXXXXXXX
                        :    X=Cipher Suite Unused
                        :     c=CALLBACK
                        :    u=USER
                        :    o=OPERATOR
                        :    a=ADMIN
                        :     O=OEM
</syntaxhighlight>
</syntaxhighlight>


== Start the Cluster For the First Time ==
Set the password.
 
 
This starts the cluster communication and membership layer for the first time.
 
'''On one node only''';
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
pcs cluster start --all
ipmitool user list 2
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
an-c03n01.alteeve.ca: Starting Cluster...
ID  Name     Callin  Link Auth IPMI Msg  Channel Priv Limit
an-c03n02.alteeve.ca: Starting Cluster...
1                    true    true      true      Unknown (0x00)
2  admin            true    true      true      OEM
Get User Access command failed (channel 2, user 3): Unknown (0x32)
</syntaxhighlight>
</syntaxhighlight>


After a few moments, you should be able to check the status;
(ignore the error, it's harmless... *BOOM*)
 
We want to set <span class="code">admin</span>'s password, so we do:
 
{{note|1=The <span class="code">2</span> below is the ID number, not the LAN channel.}}


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
pcs status
ipmitool user set password 2 secret
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
Cluster name: an-cluster-03
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Mon Jun 24 23:28:29 2013
Last change: Mon Jun 24 23:28:10 2013 via crmd on an-c03n01.alteeve.ca
Current DC: NONE
2 Nodes configured, unknown expected votes
0 Resources configured.


Done!


Node an-c03n01.alteeve.ca (1): UNCLEAN (offline)
= Configuring the Anvil! =
Node an-c03n02.alteeve.ca (2): UNCLEAN (offline)


Full list of resources:
Now we're getting down to business!
</syntaxhighlight>


The other node should show almost the identical output.
For this section, we will be working on <span class="code">an-a04n01</span> and using [[ssh]] to perform tasks on <span class="code">an-a04n02</span>.


{{warning|1=We '''only''' disable <span class="code">stonith</span> long enough to configure it. You should '''''NEVER''''' run a cluster without fencing. No matter how simple it is and certainly not because "it's just a test cluster". Fencing is always, always required. Without it, your cluster with hang, crash and fail in unexpected and hard to debug ways.}}
{{note|1=TODO: explain what this is and how it works.}}


The two main things here are errors about <span class="code">[[stonith]]</span> being unconfigured. We will fix this very shortly, but for just this moment, we will disable it and [[quorum]].
== Enable the pcs Daemon ==


<syntaxhighlight lang="text">
{{note|1=Most of this section comes more or less verbatim from the main [http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Clusters_from_Scratch/index.html Clusters from Scratch] tutorial.}}
pcs property set stonith-enabled=false
pcs status
</syntaxhighlight>
<syntaxhighlight lang="text">
Cluster name: an-cluster-03
Last updated: Tue Jun 25 00:12:21 2013
Last change: Tue Jun 25 01:42:04 2013 via cibadmin on an-c03n01.alteeve.ca
Stack: corosync
Current DC: an-c03n02.alteeve.ca (2) - partition with quorum
Version: 1.1.10-3.1670.377aefd.git.el7-377aefd
2 Nodes configured, unknown expected votes
0 Resources configured.


We will use [[pcs]], the Pacemaker Configuration System, to configure our Anvil!.


Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]
Note that pcsd uses TCP port 2224.


Full list of resources:
<syntaxhighlight lang="bash">
systemctl start pcsd.service
systemctl enable pcsd.service
</syntaxhighlight>
<syntaxhighlight lang="text">
ln -s '/usr/lib/systemd/system/pcsd.service' '/etc/systemd/system/multi-user.target.wants/pcsd.service'
</syntaxhighlight>
</syntaxhighlight>


== Disabling Quorum ==
Now we need to set a password for the <span class="code">hacluster</span> user. This is the account used by <span class="code">pcs</span> on one node to talk to the <span class="code">pcs</span> [[daemon]] on the other node. For this tutorial, we will use the password <span class="code">secret</span>. You will want to use [https://xkcd.com/936/ a stronger password], of course.
 
{{note|1=Show the math.}}
 
With quorum enabled, a two node cluster will lose quorum once either node fails. So we have to disable quorum.
 
By default, pacemaker uses quorum. You don't see this initially though;


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
pcs property
echo "super secret password" | passwd --stdin hacluster
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
Cluster Properties:
Changing password for user hacluster.
dc-version: 1.1.9-0.1318.a7966fb.git.fc18-a7966fb
passwd: all authentication tokens updated successfully.
cluster-infrastructure: corosync
</syntaxhighlight>
</syntaxhighlight>


To disable it, we set <span class="code">no-quorum-policy=ignore</span>.
Open up the firewall:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
pcs property set no-quorum-policy=ignore
firewall-cmd --permanent --add-service=high-availability
pcs property
firewall-cmd --reload
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
Cluster Properties:
dc-version: 1.1.9-0.1318.a7966fb.git.fc18-a7966fb
cluster-infrastructure: corosync
no-quorum-policy: ignore
</syntaxhighlight>
</syntaxhighlight>


== Enabling and Configuring Fencing ==
== Initializing the Cluster ==
 
One of the biggest reasons we're using the [[pcs]] tool, over something like [[crm]], is that it has been written to simplify the setup of clusters on [[Red Hat]] style operating systems. It will configure [[corosync]] automatically.


We will use IPMI and PDU based fence devices for redundancy.
First, we need to know what <span class="code">hostname</span> we will need to use for <span class="code">[[pcs]]</span>.


You can see the list of available fence agents here. You will need to find the one for your hardware fence devices.
'''Node 01''':


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
pcs stonith list
hostname
</syntaxhighlight>
<syntaxhighlight lang="bash">
an-a04n01.alteeve.ca
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
 
fence_alom - Fence agent for Sun ALOM
'''Node 02''':
fence_apc - Fence agent for APC over telnet/ssh
 
fence_apc_snmp - Fence agent for APC over SNMP
<syntaxhighlight lang="bash">
fence_baytech - I/O Fencing agent for Baytech RPC switches in combination with a Cyclades Terminal
hostname
                Server
</syntaxhighlight>
fence_bladecenter - Fence agent for IBM BladeCenter
<syntaxhighlight lang="bash">
fence_brocade - Fence agent for Brocade over telnet
an-a04n02.alteeve.ca
fence_bullpap - I/O Fencing agent for Bull FAME architecture controlled by a PAP management console.
</syntaxhighlight>
fence_cisco_mds - Fence agent for Cisco MDS
 
fence_cisco_ucs - Fence agent for Cisco UCS
Next, authenticate against the cluster nodes.
fence_cpint - I/O Fencing agent for GFS on s390 and zSeries VM clusters
 
fence_drac - fencing agent for Dell Remote Access Card
'''Both nodes''':
fence_drac5 - Fence agent for Dell DRAC CMC/5
 
fence_eaton_snmp - Fence agent for Eaton over SNMP
<syntaxhighlight lang="bash">
fence_egenera - I/O Fencing agent for the Egenera BladeFrame
pcs cluster auth an-a04n01.alteeve.ca an-a04n02.alteeve.ca -u hacluster
fence_eps - Fence agent for ePowerSwitch
</syntaxhighlight>
fence_hpblade - Fence agent for HP BladeSystem
 
fence_ibmblade - Fence agent for IBM BladeCenter over SNMP
This will ask you for the user name and password. The default user name is <span class="code">hacluster</span> and we set the password to <span class="code">secret</span>.
fence_idrac - Fence agent for IPMI over LAN
 
fence_ifmib - Fence agent for IF MIB
<syntaxhighlight lang="text">
fence_ilo - Fence agent for HP iLO
Password:
fence_ilo2 - Fence agent for HP iLO
an-a04n01.alteeve.ca: 6e9f7e98-dfb7-4305-b8e0-d84bf4f93ce3
fence_ilo3 - Fence agent for IPMI over LAN
an-a04n01.alteeve.ca: Authorized
fence_ilo_mp - Fence agent for HP iLO MP
an-a04n02.alteeve.ca: ffee6a85-ddac-4d03-9b97-f136d532b478
fence_imm - Fence agent for IPMI over LAN
an-a04n02.alteeve.ca: Authorized
fence_intelmodular - Fence agent for Intel Modular
</syntaxhighlight>
fence_ipdu - Fence agent for iPDU over SNMP
 
fence_ipmilan - Fence agent for IPMI over LAN
'''Do this on one node only''':
fence_kdump - Fence agent for use with kdump
 
fence_ldom - Fence agent for Sun LDOM
Now to initialize the cluster's communication and membership layer.
fence_lpar - Fence agent for IBM LPAR
 
fence_mcdata - I/O Fencing agent for McData FC switches
<syntaxhighlight lang="bash">
fence_rackswitch - fence_rackswitch - I/O Fencing agent for RackSaver RackSwitch
pcs cluster setup --name an-anvil-03 an-a04n01.alteeve.ca an-a04n02.alteeve.ca
fence_rhevm - Fence agent for RHEV-M REST API
</syntaxhighlight>
fence_rsa - Fence agent for IBM RSA
<syntaxhighlight lang="text">
fence_rsb - I/O Fencing agent for Fujitsu-Siemens RSB
an-a04n01.alteeve.ca: Succeeded
fence_sanbox2 - Fence agent for QLogic SANBox2 FC switches
an-a04n02.alteeve.ca: Succeeded
fence_scsi - fence agent for SCSI-3 persistent reservations
fence_virsh - Fence agent for virsh
fence_vixel - I/O Fencing agent for Vixel FC switches
fence_vmware - Fence agent for VMWare
fence_vmware_soap - Fence agent for VMWare over SOAP API
fence_wti - Fence agent for WTI
fence_xcat - I/O Fencing agent for xcat environments
fence_xenapi - XenAPI based fencing for the Citrix XenServer virtual machines.
fence_zvm - I/O Fencing agent for GFS on s390 and zSeries VM clusters
</syntaxhighlight>
</syntaxhighlight>


We will use <span class="code">fence_ipmilan</span> and <span class="code">fence_apc_snmp</span>.
This will create the corosync configuration file <span class="code">/etc/corosync/corosync.conf</span>;


=== Configuring IPMI Fencing ===
<syntaxhighlight lang="bash">
cat /etc/corosync/corosync.conf
</syntaxhighlight>
<syntaxhighlight lang="text">
totem {
version: 2
secauth: off
cluster_name: an-anvil-03
transport: udpu
}
 
nodelist {
  node {
        ring0_addr: an-a04n01.alteeve.ca
        nodeid: 1
      }
  node {
        ring0_addr: an-a04n02.alteeve.ca
        nodeid: 2
      }
}


Every fence agent has a possibly unique subset of options that can be used. You can see a brief description of these options with the <span class="code">pcs stonith describe fence_X</span> command. Let's look at the options available for <span class="code">fence_ipmilan</span>.
quorum {
provider: corosync_votequorum
two_node: 1
}


<syntaxhighlight lang="bash">
logging {
pcs stonith describe fence_ipmilan
to_syslog: yes
</syntaxhighlight>
}
<syntaxhighlight lang="text">
Stonith options for: fence_ipmilan
  auth: IPMI Lan Auth type (md5, password, or none)
  ipaddr: IPMI Lan IP to talk to
  passwd: Password (if required) to control power on IPMI device
  passwd_script: Script to retrieve password (if required)
  lanplus: Use Lanplus
  login: Username/Login (if required) to control power on IPMI device
  action: Operation to perform. Valid operations: on, off, reboot, status, list, diag, monitor or metadata
  timeout: Timeout (sec) for IPMI operation
  cipher: Ciphersuite to use (same as ipmitool -C parameter)
  method: Method to fence (onoff or cycle)
  power_wait: Wait X seconds after on/off operation
  delay: Wait X seconds before fencing is started
  privlvl: Privilege level on IPMI device
  verbose: Verbose mode
</syntaxhighlight>
</syntaxhighlight>


One of the nice things about pcs is that it allows us to create a test file to prepare all our changes in. Then, when we're happy with the changes, merge them into the running cluster. So let's make a copy called <span class="code">stonith_cfg</span>
== Start the Cluster For the First Time ==


<syntaxhighlight lang="bash">
This starts the cluster communication and membership layer for the first time.
pcs cluster cib stonith_cfg
</syntaxhighlight>


Now add [[IPMI]] fencing.
'''On one node only''';


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
#                  unique name    fence agent  target node                          device addr            options
pcs cluster start --all
pcs stonith create fence_n01_ipmi fence_ipmilan pcmk_host_list="an-c03n01.alteeve.ca" ipaddr="an-c03n01.ipmi" action="reboot" login="admin" passwd="secret" delay=15 op monitor interval=60s
pcs stonith create fence_n02_ipmi fence_ipmilan pcmk_host_list="an-c03n02.alteeve.ca" ipaddr="an-c03n02.ipmi" action="reboot" login="admin" passwd="secret" op monitor interval=60s
</syntaxhighlight>
</syntaxhighlight>
 
<syntaxhighlight lang="text">
Note that <span class="code">fence_n01_ipmi</span> has a <span class="code">delay=15</span> set but <span class="code">fence_n02_ipmi</span> does not. If the network connection breaks between the two nodes, they will both try to fence each other at the same time. If <span class="code">acpid</span> is running, the slower node will not die right away. It will continue to run for up to four more seconds, ample time for it to also initiate a fence against the faster node. The end result is that both nodes get fenced. The ten-second delay protects against this by causing <span class="code">an-c03n02</span> to pause for <span class="code">10</span> seconds before initiating a fence against <span class="code">an-c03n01</span>. If both nodes are alive, <span class="code">an-c03n02</span> will power off before the 10 seconds pass, so it will never fence <span class="code">an-c03n01</span>. However, if <span class="code">an-c03n01</span> really is dead, after the ten seconds have elapsed, fencing will proceed as normal.
an-a04n01.alteeve.ca: Starting Cluster...
 
an-a04n02.alteeve.ca: Starting Cluster...
{{note|1=At the time of writing, <span class="code">pcmk_reboot_action</span> is needed to override pacemaker's global fence action and <span class="code">pcmk_reboot_action</span> is not recognized by pcs. Both of these issues will be resolved shortly; Pacemaker will honour <span class="code">action="..."</span> in v1.1.10 and pcs will recognize <span class="code">pcmk_*</span> special attributes "real soon now". Until then, the <span class="code">--force</span> switch is needed.}}
 
Next, add the [[PDU]] fencing. This requires distinct "off" and "on" actions for each outlet on each PDU. With two nodes, each with two [[PSU]]s, this translates to eight commands. The "off" commands will be monitored to alert us if the PDU fails for some reason. There is no reason to monitor the "on" actions (it would be redundant). Note also that we don't bother using a "delay". The IPMI fence method will go first, before the PDU actions, so the PDU is already delayed.
 
<syntaxhighlight lang="bash">
# Node 1 - off
pcs stonith create fence_n01_pdu1_off fence_apc_snmp pcmk_host_list="an-c03n01.alteeve.ca" ipaddr="an-p01" action="off" port="1" op monitor interval="60s"
pcs stonith create fence_n01_pdu2_off fence_apc_snmp pcmk_host_list="an-c03n01.alteeve.ca" ipaddr="an-p02" action="off" port="1" power_wait="5" op monitor interval="60s"
 
# Node 1 - on
pcs stonith create fence_n01_pdu1_on fence_apc_snmp pcmk_host_list="an-c03n01.alteeve.ca" ipaddr="an-p01" action="on" port="1"
pcs stonith create fence_n01_pdu2_on fence_apc_snmp pcmk_host_list="an-c03n01.alteeve.ca" ipaddr="an-p02" action="on" port="1"
 
# Node 2 - off
pcs stonith create fence_n02_pdu1_off fence_apc_snmp pcmk_host_list="an-c03n02.alteeve.ca" ipaddr="an-p01" action="off" port="2" op monitor interval="60s"
pcs stonith create fence_n02_pdu2_off fence_apc_snmp pcmk_host_list="an-c03n02.alteeve.ca" ipaddr="an-p02" action="off" port="2" power_wait="5" op monitor interval="60s"
 
# Node 2 - on
pcs stonith create fence_n02_pdu1_on fence_apc_snmp pcmk_host_list="an-c03n02.alteeve.ca" ipaddr="an-p01" action="on" port="2"
pcs stonith create fence_n02_pdu2_on fence_apc_snmp pcmk_host_list="an-c03n02.alteeve.ca" ipaddr="an-p02" action="on" port="2"
</syntaxhighlight>
</syntaxhighlight>


We can check the new configuration now;
After a few moments, you should be able to check the status;


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
Line 874: Line 1,137:
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
Cluster name: an-cluster-03
Cluster name: an-anvil-04
Last updated: Tue Jul  2 16:41:55 2013
WARNING: no stonith devices and stonith-enabled is not false
Last change: Tue Jul  2 16:41:44 2013 via cibadmin on an-c03n01.alteeve.ca
Last updated: Mon Jun 24 23:28:29 2013
Stack: corosync
Last change: Mon Jun 24 23:28:10 2013 via crmd on an-a04n01.alteeve.ca
Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
Current DC: NONE
Version: 1.1.9-3.fc19-781a388
2 Nodes configured, unknown expected votes
2 Nodes configured, unknown expected votes
10 Resources configured.
0 Resources configured.




Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]
Node an-a04n01.alteeve.ca (1): UNCLEAN (offline)
Node an-a04n02.alteeve.ca (2): UNCLEAN (offline)


Full list of resources:
Full list of resources:
</syntaxhighlight>
The other node should show almost the identical output.


fence_n01_ipmi (stonith:fence_ipmilan): Started an-c03n01.alteeve.ca
== Disabling Quorum ==
fence_n02_ipmi (stonith:fence_ipmilan): Started an-c03n02.alteeve.ca
 
fence_n01_pdu1_off (stonith:fence_apc_snmp): Started an-c03n01.alteeve.ca
{{note|1=Show the math.}}
fence_n01_pdu2_off (stonith:fence_apc_snmp): Started an-c03n02.alteeve.ca
 
  fence_n02_pdu1_off (stonith:fence_apc_snmp): Started an-c03n01.alteeve.ca
With quorum enabled, a two node cluster will lose quorum once either node fails. So we have to disable quorum.
fence_n02_pdu2_off (stonith:fence_apc_snmp): Started an-c03n02.alteeve.ca
 
fence_n01_pdu1_on (stonith:fence_apc_snmp): Started an-c03n01.alteeve.ca
By default, pacemaker uses quorum. You don't see this initially though;
fence_n01_pdu2_on (stonith:fence_apc_snmp): Started an-c03n02.alteeve.ca
 
  fence_n02_pdu1_on (stonith:fence_apc_snmp): Started an-c03n01.alteeve.ca
<syntaxhighlight lang="bash">
fence_n02_pdu2_on (stonith:fence_apc_snmp): Started an-c03n02.alteeve.ca
pcs property
</syntaxhighlight>
<syntaxhighlight lang="text">
Cluster Properties:
  dc-version: 1.1.9-0.1318.a7966fb.git.fc18-a7966fb
  cluster-infrastructure: corosync
</syntaxhighlight>
</syntaxhighlight>


Before we proceed, we need to tell pacemaker to use fencing;
To disable it, we set <span class="code">no-quorum-policy=ignore</span>.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
pcs property set stonith-enabled=true
pcs property set no-quorum-policy=ignore
pcs property
pcs property
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
Cluster Properties:
Cluster Properties:
Cluster Properties:
dc-version: 1.1.9-0.1318.a7966fb.git.fc18-a7966fb
  cluster-infrastructure: corosync
  cluster-infrastructure: corosync
dc-version: 1.1.9-3.fc19-781a388
  no-quorum-policy: ignore
  no-quorum-policy: ignore
stonith-enabled: true
</syntaxhighlight>
</syntaxhighlight>


Excellent!
== Enabling and Configuring Fencing ==


== Configuring Fence Levels ==
We will use IPMI and PDU based fence devices for redundancy.


The goal of fence levels is to tell pacemaker that there are "fence methods" to try and to impose an order on those methods. Each method composes one or more fence primitives and, when 2 or more primitives are tied together, that all primitives must succeed for the overall method to succeed.
You can see the list of available fence agents here. You will need to find the one for your hardware fence devices.
 
So in our case; the order we want is;
 
* IPMI -> PDUs
 
The reason is that when IPMI fencing succeeds, we can be very certain the node is truly fenced. When PDU fencing succeeds, it only confirms that the power outlets were cycled. If someone moved a node's power cables to another outlet, we'll get a false positive. On that topic, tie-down the node's PSU cables to the PDU's cable tray when possible, clearly label the power cables and wrap the fingers of anyone who might move them around.
 
The PDU fencing needs to be implemented using four steps;
 
* PDU 1, outlet X -> off
* PDU 2, outlet X -> off
** The <span class="code">power_wait="5"</span> setting for the <span class="code">fence_n0X_pdu2_off</span> primitives will cause a 5 second delay here, giving ample time to ensure the nodes lose power
* PDU 1, outlet X -> on
* PDU 2, outlet X -> on
 
This is to ensure that both outlets are off at the same time, ensuring that the node loses power. This works because <span class="code">fencing_topology</span> acts serially.
 
Putting all this together, we issue this command;


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
pcs stonith level add 1 an-c03n01.alteeve.ca fence_n01_ipmi
pcs stonith list
pcs stonith level add 1 an-c03n02.alteeve.ca fence_n02_ipmi
</syntaxhighlight>
</syntaxhighlight>
 
<syntaxhighlight lang="text">
The <span class="code">1</span> tells pacemaker that this is our highest priority fence method. We can see that this was set using pcs;
fence_alom - Fence agent for Sun ALOM
 
fence_apc - Fence agent for APC over telnet/ssh
<syntaxhighlight lang="bash">
fence_apc_snmp - Fence agent for APC over SNMP
pcs stonith level
fence_baytech - I/O Fencing agent for Baytech RPC switches in combination with a Cyclades Terminal
</syntaxhighlight>
                Server
<syntaxhighlight lang="bash">
fence_bladecenter - Fence agent for IBM BladeCenter
Node: an-c03n01.alteeve.ca
fence_brocade - Fence agent for Brocade over telnet
  Level 1 - fence_n01_ipmi
fence_bullpap - I/O Fencing agent for Bull FAME architecture controlled by a PAP management console.
Node: an-c03n02.alteeve.ca
fence_cisco_mds - Fence agent for Cisco MDS
  Level 1 - fence_n02_ipmi
fence_cisco_ucs - Fence agent for Cisco UCS
</syntaxhighlight>
fence_cpint - I/O Fencing agent for GFS on s390 and zSeries VM clusters
 
fence_drac - fencing agent for Dell Remote Access Card
Now we'll tell pacemaker to use the PDUs as the second fence method. Here we tie together the two <span class="code">off</span> calls and the two <span class="code">on</span> calls into a single method.
fence_drac5 - Fence agent for Dell DRAC CMC/5
 
fence_eaton_snmp - Fence agent for Eaton over SNMP
<syntaxhighlight lang="bash">
fence_egenera - I/O Fencing agent for the Egenera BladeFrame
pcs stonith level add 2 an-c03n01.alteeve.ca fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on
fence_eps - Fence agent for ePowerSwitch
pcs stonith level add 2 an-c03n02.alteeve.ca fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on
fence_hpblade - Fence agent for HP BladeSystem
</syntaxhighlight>
fence_ibmblade - Fence agent for IBM BladeCenter over SNMP
 
fence_idrac - Fence agent for IPMI over LAN
Check again and we'll see that the new methods were added.
fence_ifmib - Fence agent for IF MIB
 
fence_ilo - Fence agent for HP iLO
<syntaxhighlight lang="bash">
fence_ilo2 - Fence agent for HP iLO
pcs stonith level
fence_ilo3 - Fence agent for IPMI over LAN
</syntaxhighlight>
fence_ilo_mp - Fence agent for HP iLO MP
<syntaxhighlight lang="bash">
fence_imm - Fence agent for IPMI over LAN
Node: an-c03n01.alteeve.ca
fence_intelmodular - Fence agent for Intel Modular
  Level 1 - fence_n01_ipmi
fence_ipdu - Fence agent for iPDU over SNMP
  Level 2 - fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on
fence_ipmilan - Fence agent for IPMI over LAN
Node: an-c03n02.alteeve.ca
fence_kdump - Fence agent for use with kdump
  Level 1 - fence_n02_ipmi
fence_ldom - Fence agent for Sun LDOM
  Level 2 - fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on
fence_lpar - Fence agent for IBM LPAR
fence_mcdata - I/O Fencing agent for McData FC switches
fence_rackswitch - fence_rackswitch - I/O Fencing agent for RackSaver RackSwitch
fence_rhevm - Fence agent for RHEV-M REST API
fence_rsa - Fence agent for IBM RSA
fence_rsb - I/O Fencing agent for Fujitsu-Siemens RSB
fence_sanbox2 - Fence agent for QLogic SANBox2 FC switches
fence_scsi - fence agent for SCSI-3 persistent reservations
fence_virsh - Fence agent for virsh
fence_vixel - I/O Fencing agent for Vixel FC switches
fence_vmware - Fence agent for VMWare
fence_vmware_soap - Fence agent for VMWare over SOAP API
fence_wti - Fence agent for WTI
fence_xcat - I/O Fencing agent for xcat environments
fence_xenapi - XenAPI based fencing for the Citrix XenServer virtual machines.
fence_zvm - I/O Fencing agent for GFS on s390 and zSeries VM clusters
</syntaxhighlight>
</syntaxhighlight>


For those of us who are [[XML]] fans, this is what the [[cib]] looks like now:
We will use <span class="code">fence_ipmilan</span> and <span class="code">fence_apc_snmp</span>.
 
=== Configuring IPMI Fencing ===
 
Every fence agent has a possibly unique subset of options that can be used. You can see a brief description of these options with the <span class="code">pcs stonith describe fence_X</span> command. Let's look at the options available for <span class="code">fence_ipmilan</span>.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
cat /var/lib/pacemaker/cib/cib.xml
pcs stonith describe fence_ipmilan
</syntaxhighlight>
<syntaxhighlight lang="xml">
<cib epoch="18" num_updates="0" admin_epoch="0" validate-with="pacemaker-1.2" cib-last-written="Thu Jul 18 13:15:53 2013" update-origin="an-c03n01.alteeve.ca" update-client="cibadmin" crm_feature_set="3.0.7" have-quorum="1" dc-uuid="1">
  <configuration>
    <crm_config>
      <cluster_property_set id="cib-bootstrap-options">
        <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.9-dde1c52"/>
        <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
        <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/>
      </cluster_property_set>
    </crm_config>
    <nodes>
      <node id="1" uname="an-c03n01.alteeve.ca"/>
      <node id="2" uname="an-c03n02.alteeve.ca"/>
    </nodes>
    <resources>
      <primitive class="stonith" id="fence_n01_ipmi" type="fence_ipmilan">
        <instance_attributes id="fence_n01_ipmi-instance_attributes">
          <nvpair id="fence_n01_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-c03n01.alteeve.ca"/>
          <nvpair id="fence_n01_ipmi-instance_attributes-ipaddr" name="ipaddr" value="an-c03n01.ipmi"/>
          <nvpair id="fence_n01_ipmi-instance_attributes-action" name="action" value="reboot"/>
          <nvpair id="fence_n01_ipmi-instance_attributes-login" name="login" value="admin"/>
          <nvpair id="fence_n01_ipmi-instance_attributes-passwd" name="passwd" value="secret"/>
          <nvpair id="fence_n01_ipmi-instance_attributes-delay" name="delay" value="15"/>
        </instance_attributes>
        <operations>
          <op id="fence_n01_ipmi-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n02_ipmi" type="fence_ipmilan">
        <instance_attributes id="fence_n02_ipmi-instance_attributes">
          <nvpair id="fence_n02_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-c03n02.alteeve.ca"/>
          <nvpair id="fence_n02_ipmi-instance_attributes-ipaddr" name="ipaddr" value="an-c03n02.ipmi"/>
          <nvpair id="fence_n02_ipmi-instance_attributes-action" name="action" value="reboot"/>
          <nvpair id="fence_n02_ipmi-instance_attributes-login" name="login" value="admin"/>
          <nvpair id="fence_n02_ipmi-instance_attributes-passwd" name="passwd" value="secret"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_ipmi-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n01_pdu1_off" type="fence_apc_snmp">
        <instance_attributes id="fence_n01_pdu1_off-instance_attributes">
          <nvpair id="fence_n01_pdu1_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-c03n01.alteeve.ca"/>
          <nvpair id="fence_n01_pdu1_off-instance_attributes-ipaddr" name="ipaddr" value="an-p01"/>
          <nvpair id="fence_n01_pdu1_off-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_n01_pdu1_off-instance_attributes-port" name="port" value="1"/>
        </instance_attributes>
        <operations>
          <op id="fence_n01_pdu1_off-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n01_pdu2_off" type="fence_apc_snmp">
        <instance_attributes id="fence_n01_pdu2_off-instance_attributes">
          <nvpair id="fence_n01_pdu2_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-c03n01.alteeve.ca"/>
          <nvpair id="fence_n01_pdu2_off-instance_attributes-ipaddr" name="ipaddr" value="an-p02"/>
          <nvpair id="fence_n01_pdu2_off-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_n01_pdu2_off-instance_attributes-port" name="port" value="1"/>
          <nvpair id="fence_n01_pdu2_off-instance_attributes-power_wait" name="power_wait" value="5"/>
        </instance_attributes>
        <operations>
          <op id="fence_n01_pdu2_off-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n01_pdu1_on" type="fence_apc_snmp">
        <instance_attributes id="fence_n01_pdu1_on-instance_attributes">
          <nvpair id="fence_n01_pdu1_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-c03n01.alteeve.ca"/>
          <nvpair id="fence_n01_pdu1_on-instance_attributes-ipaddr" name="ipaddr" value="an-p01"/>
          <nvpair id="fence_n01_pdu1_on-instance_attributes-action" name="action" value="on"/>
          <nvpair id="fence_n01_pdu1_on-instance_attributes-port" name="port" value="1"/>
        </instance_attributes>
        <operations>
          <op id="fence_n01_pdu1_on-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n01_pdu2_on" type="fence_apc_snmp">
        <instance_attributes id="fence_n01_pdu2_on-instance_attributes">
          <nvpair id="fence_n01_pdu2_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-c03n01.alteeve.ca"/>
          <nvpair id="fence_n01_pdu2_on-instance_attributes-ipaddr" name="ipaddr" value="an-p02"/>
          <nvpair id="fence_n01_pdu2_on-instance_attributes-action" name="action" value="on"/>
          <nvpair id="fence_n01_pdu2_on-instance_attributes-port" name="port" value="1"/>
        </instance_attributes>
        <operations>
          <op id="fence_n01_pdu2_on-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n02_pdu1_off" type="fence_apc_snmp">
        <instance_attributes id="fence_n02_pdu1_off-instance_attributes">
          <nvpair id="fence_n02_pdu1_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-c03n02.alteeve.ca"/>
          <nvpair id="fence_n02_pdu1_off-instance_attributes-ipaddr" name="ipaddr" value="an-p01"/>
          <nvpair id="fence_n02_pdu1_off-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_n02_pdu1_off-instance_attributes-port" name="port" value="2"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_pdu1_off-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n02_pdu2_off" type="fence_apc_snmp">
        <instance_attributes id="fence_n02_pdu2_off-instance_attributes">
          <nvpair id="fence_n02_pdu2_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-c03n02.alteeve.ca"/>
          <nvpair id="fence_n02_pdu2_off-instance_attributes-ipaddr" name="ipaddr" value="an-p02"/>
          <nvpair id="fence_n02_pdu2_off-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_n02_pdu2_off-instance_attributes-port" name="port" value="2"/>
          <nvpair id="fence_n02_pdu2_off-instance_attributes-power_wait" name="power_wait" value="5"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_pdu2_off-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n02_pdu1_on" type="fence_apc_snmp">
        <instance_attributes id="fence_n02_pdu1_on-instance_attributes">
          <nvpair id="fence_n02_pdu1_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-c03n02.alteeve.ca"/>
          <nvpair id="fence_n02_pdu1_on-instance_attributes-ipaddr" name="ipaddr" value="an-p01"/>
          <nvpair id="fence_n02_pdu1_on-instance_attributes-action" name="action" value="on"/>
          <nvpair id="fence_n02_pdu1_on-instance_attributes-port" name="port" value="2"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_pdu1_on-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n02_pdu2_on" type="fence_apc_snmp">
        <instance_attributes id="fence_n02_pdu2_on-instance_attributes">
          <nvpair id="fence_n02_pdu2_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-c03n02.alteeve.ca"/>
          <nvpair id="fence_n02_pdu2_on-instance_attributes-ipaddr" name="ipaddr" value="an-p02"/>
          <nvpair id="fence_n02_pdu2_on-instance_attributes-action" name="action" value="on"/>
          <nvpair id="fence_n02_pdu2_on-instance_attributes-port" name="port" value="2"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_pdu2_on-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
    </resources>
    <constraints/>
    <fencing-topology>
      <fencing-level devices="fence_n01_ipmi" id="fl-an-c03n01.alteeve.ca-1" index="1" target="an-c03n01.alteeve.ca"/>
      <fencing-level devices="fence_n02_ipmi" id="fl-an-c03n02.alteeve.ca-1" index="1" target="an-c03n02.alteeve.ca"/>
      <fencing-level devices="fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on" id="fl-an-c03n01.alteeve.ca-2" index="2" target="an-c03n01.alteeve.ca"/>
      <fencing-level devices="fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on" id="fl-an-c03n02.alteeve.ca-2" index="2" target="an-c03n02.alteeve.ca"/>
    </fencing-topology>
  </configuration>
</cib>
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
Stonith options for: fence_ipmilan
  auth: IPMI Lan Auth type (md5, password, or none)
  ipaddr: IPMI Lan IP to talk to
  passwd: Password (if required) to control power on IPMI device
  passwd_script: Script to retrieve password (if required)
  lanplus: Use Lanplus
  login: Username/Login (if required) to control power on IPMI device
  action: Operation to perform. Valid operations: on, off, reboot, status, list, diag, monitor or metadata
  timeout: Timeout (sec) for IPMI operation
  cipher: Ciphersuite to use (same as ipmitool -C parameter)
  method: Method to fence (onoff or cycle)
  power_wait: Wait X seconds after on/off operation
  delay: Wait X seconds before fencing is started
  privlvl: Privilege level on IPMI device
  verbose: Verbose mode
</syntaxhighlight>
One of the nice things about pcs is that it allows us to create a test file to prepare all our changes in. Then, when we're happy with the changes, merge them into the running cluster. So let's make a copy called <span class="code">stonith_cfg</span>
<syntaxhighlight lang="bash">
pcs cluster cib stonith_cfg
</syntaxhighlight>
Now add [[IPMI]] fencing.
<syntaxhighlight lang="bash">
#                  unique name    fence agent  target node                          device addr            options
pcs stonith create fence_n01_ipmi fence_ipmilan pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-a04n01.ipmi" action="reboot" login="admin" passwd="secret" delay=15 op monitor interval=60s
pcs stonith create fence_n02_ipmi fence_ipmilan pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-a04n02.ipmi" action="reboot" login="admin" passwd="secret" op monitor interval=60s
</syntaxhighlight>
Note that <span class="code">fence_n01_ipmi</span> has a <span class="code">delay=15</span> set but <span class="code">fence_n02_ipmi</span> does not. If the network connection breaks between the two nodes, they will both try to fence each other at the same time. If <span class="code">acpid</span> is running, the slower node will not die right away. It will continue to run for up to four more seconds, ample time for it to also initiate a fence against the faster node. The end result is that both nodes get fenced. The ten-second delay protects against this by causing <span class="code">an-a04n02</span> to pause for <span class="code">10</span> seconds before initiating a fence against <span class="code">an-a04n01</span>. If both nodes are alive, <span class="code">an-a04n02</span> will power off before the 10 seconds pass, so it will never fence <span class="code">an-a04n01</span>. However, if <span class="code">an-a04n01</span> really is dead, after the ten seconds have elapsed, fencing will proceed as normal.
{{note|1=At the time of writing, <span class="code">pcmk_reboot_action</span> is needed to override pacemaker's global fence action and <span class="code">pcmk_reboot_action</span> is not recognized by pcs. Both of these issues will be resolved shortly; Pacemaker will honour <span class="code">action="..."</span> in v1.1.10 and pcs will recognize <span class="code">pcmk_*</span> special attributes "real soon now". Until then, the <span class="code">--force</span> switch is needed.}}
Next, add the [[PDU]] fencing. This requires distinct "off" and "on" actions for each outlet on each PDU. With two nodes, each with two [[PSU]]s, this translates to eight commands. The "off" commands will be monitored to alert us if the PDU fails for some reason. There is no reason to monitor the "on" actions (it would be redundant). Note also that we don't bother using a "delay". The IPMI fence method will go first, before the PDU actions, so the PDU is already delayed.
<syntaxhighlight lang="bash">
# Node 1 - off
pcs stonith create fence_n01_pdu1_off fence_apc_snmp pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-pdu01" action="off" port="1" op monitor interval="60s"
pcs stonith create fence_n01_pdu2_off fence_apc_snmp pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-pdu02" action="off" port="1" power_wait="5" op monitor interval="60s"
# Node 1 - on
pcs stonith create fence_n01_pdu1_on fence_apc_snmp pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-pdu01" action="on" port="1"
pcs stonith create fence_n01_pdu2_on fence_apc_snmp pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-pdu02" action="on" port="1"
# Node 2 - off
pcs stonith create fence_n02_pdu1_off fence_apc_snmp pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-pdu01" action="off" port="2" op monitor interval="60s"
pcs stonith create fence_n02_pdu2_off fence_apc_snmp pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-pdu02" action="off" port="2" power_wait="5" op monitor interval="60s"
# Node 2 - on
pcs stonith create fence_n02_pdu1_on fence_apc_snmp pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-pdu01" action="on" port="2"
pcs stonith create fence_n02_pdu2_on fence_apc_snmp pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-pdu02" action="on" port="2"
</syntaxhighlight>
We can check the new configuration now;
<syntaxhighlight lang="bash">
pcs status
</syntaxhighlight>
<syntaxhighlight lang="text">
Cluster name: an-anvil-04
Last updated: Tue Jul  2 16:41:55 2013
Last change: Tue Jul  2 16:41:44 2013 via cibadmin on an-a04n01.alteeve.ca
Stack: corosync
Current DC: an-a04n01.alteeve.ca (1) - partition with quorum
Version: 1.1.9-3.fc19-781a388
2 Nodes configured, unknown expected votes
10 Resources configured.
Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
Full list of resources:
fence_n01_ipmi (stonith:fence_ipmilan): Started an-a04n01.alteeve.ca
fence_n02_ipmi (stonith:fence_ipmilan): Started an-a04n02.alteeve.ca
fence_n01_pdu1_off (stonith:fence_apc_snmp): Started an-a04n01.alteeve.ca
fence_n01_pdu2_off (stonith:fence_apc_snmp): Started an-a04n02.alteeve.ca
fence_n02_pdu1_off (stonith:fence_apc_snmp): Started an-a04n01.alteeve.ca
fence_n02_pdu2_off (stonith:fence_apc_snmp): Started an-a04n02.alteeve.ca
fence_n01_pdu1_on (stonith:fence_apc_snmp): Started an-a04n01.alteeve.ca
fence_n01_pdu2_on (stonith:fence_apc_snmp): Started an-a04n02.alteeve.ca
fence_n02_pdu1_on (stonith:fence_apc_snmp): Started an-a04n01.alteeve.ca
fence_n02_pdu2_on (stonith:fence_apc_snmp): Started an-a04n02.alteeve.ca
</syntaxhighlight>
Before we proceed, we need to tell pacemaker to use fencing;
<syntaxhighlight lang="bash">
pcs property set stonith-enabled=true
pcs property
</syntaxhighlight>
<syntaxhighlight lang="text">
Cluster Properties:
Cluster Properties:
cluster-infrastructure: corosync
dc-version: 1.1.9-3.fc19-781a388
no-quorum-policy: ignore
stonith-enabled: true
</syntaxhighlight>
Excellent!
== Configuring Fence Levels ==
The goal of fence levels is to tell pacemaker that there are "fence methods" to try and to impose an order on those methods. Each method composes one or more fence primitives and, when 2 or more primitives are tied together, that all primitives must succeed for the overall method to succeed.
So in our case; the order we want is;
* IPMI -> PDUs
The reason is that when IPMI fencing succeeds, we can be very certain the node is truly fenced. When PDU fencing succeeds, it only confirms that the power outlets were cycled. If someone moved a node's power cables to another outlet, we'll get a false positive. On that topic, tie-down the node's PSU cables to the PDU's cable tray when possible, clearly label the power cables and wrap the fingers of anyone who might move them around.
The PDU fencing needs to be implemented using four steps;


== Fencing using fence_virsh ==
* PDU 1, outlet X -> off
* PDU 2, outlet X -> off
** The <span class="code">power_wait="5"</span> setting for the <span class="code">fence_n0X_pdu2_off</span> primitives will cause a 5 second delay here, giving ample time to ensure the nodes lose power
* PDU 1, outlet X -> on
* PDU 2, outlet X -> on


{{note|1=To write this section, I used two virtual machines called <span class="code">pcmk1</span> and <span class="code">pcmk2</span>.}}
This is to ensure that both outlets are off at the same time, ensuring that the node loses power. This works because <span class="code">fencing_topology</span> acts serially.


If you are trying to learn fencing using KVM or Xen virtual machines, you can use the <span class="code">fence_virsh</span>. You can also use <span class="code">[[Fencing KVM Virtual Servers|fence_virtd]]</span>, which is actually recommended by many, but I have found it to be rather unreliable.
Putting all this together, we issue this command;
 
To use <span class="code">fence_virsh</span>, first install it.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
yum -y install fence-agents-virsh
pcs stonith level add 1 an-a04n01.alteeve.ca fence_n01_ipmi
pcs stonith level add 1 an-a04n02.alteeve.ca fence_n02_ipmi
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
Resolving Dependencies
--> Running transaction check
---> Package fence-agents-virsh.x86_64 0:4.0.3-1.fc19 will be installed
--> Processing Dependency: /usr/bin/virsh for package: fence-agents-virsh-4.0.3-1.fc19.x86_64
--> Running transaction check
---> Package libvirt-client.x86_64 0:1.0.5.5-1.fc19 will be installed
--> Processing Dependency: pm-utils for package: libvirt-client-1.0.5.5-1.fc19.x86_64
--> Processing Dependency: nc for package: libvirt-client-1.0.5.5-1.fc19.x86_64
--> Processing Dependency: libnuma.so.1(libnuma_1.2)(64bit) for package: libvirt-client-1.0.5.5-1.fc19.x86_64
--> Processing Dependency: libnuma.so.1(libnuma_1.1)(64bit) for package: libvirt-client-1.0.5.5-1.fc19.x86_64
--> Processing Dependency: gnutls-utils for package: libvirt-client-1.0.5.5-1.fc19.x86_64
--> Processing Dependency: cyrus-sasl-md5 for package: libvirt-client-1.0.5.5-1.fc19.x86_64
--> Processing Dependency: libyajl.so.2()(64bit) for package: libvirt-client-1.0.5.5-1.fc19.x86_64
--> Processing Dependency: libwsman_curl_client_transport.so.1()(64bit) for package: libvirt-client-1.0.5.5-1.fc19.x86_64
--> Processing Dependency: libwsman_client.so.1()(64bit) for package: libvirt-client-1.0.5.5-1.fc19.x86_64
--> Processing Dependency: libwsman.so.1()(64bit) for package: libvirt-client-1.0.5.5-1.fc19.x86_64
--> Processing Dependency: libnuma.so.1()(64bit) for package: libvirt-client-1.0.5.5-1.fc19.x86_64
--> Running transaction check
---> Package cyrus-sasl-md5.x86_64 0:2.1.26-9.fc19 will be installed
---> Package gnutls-utils.x86_64 0:3.1.11-1.fc19 will be installed
---> Package libwsman1.x86_64 0:2.3.6-6.fc19 will be installed
---> Package nmap-ncat.x86_64 2:6.40-2.fc19 will be installed
---> Package numactl-libs.x86_64 0:2.0.8-4.fc19 will be installed
---> Package pm-utils.x86_64 0:1.4.1-24.fc19 will be installed
---> Package yajl.x86_64 0:2.0.4-2.fc19 will be installed
--> Finished Dependency Resolution


Dependencies Resolved
The <span class="code">1</span> tells pacemaker that this is our highest priority fence method. We can see that this was set using pcs;


=========================================================================================================
<syntaxhighlight lang="bash">
Package                        Arch              Version                    Repository          Size
pcs stonith level
=========================================================================================================
</syntaxhighlight>
Installing:
<syntaxhighlight lang="bash">
  fence-agents-virsh            x86_64            4.0.3-1.fc19                updates            7.7 k
  Node: an-a04n01.alteeve.ca
Installing for dependencies:
  Level 1 - fence_n01_ipmi
cyrus-sasl-md5                x86_64            2.1.26-9.fc19              updates              54 k
  Node: an-a04n02.alteeve.ca
gnutls-utils                  x86_64            3.1.11-1.fc19              fedora              261 k
  Level 1 - fence_n02_ipmi
  libvirt-client                x86_64            1.0.5.5-1.fc19              updates            4.9 M
</syntaxhighlight>
libwsman1                      x86_64            2.3.6-6.fc19                fedora              120 k
nmap-ncat                      x86_64            2:6.40-2.fc19              updates            198 k
numactl-libs                  x86_64            2.0.8-4.fc19                fedora              28 k
pm-utils                      x86_64            1.4.1-24.fc19              updates            139 k
yajl                          x86_64            2.0.4-2.fc19                fedora              38 k


Transaction Summary
Now we'll tell pacemaker to use the PDUs as the second fence method. Here we tie together the two <span class="code">off</span> calls and the two <span class="code">on</span> calls into a single method.
=========================================================================================================
Install  1 Package (+8 Dependent packages)


Total download size: 5.7 M
<syntaxhighlight lang="bash">
Installed size: 23 M
pcs stonith level add 2 an-a04n01.alteeve.ca fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on
Downloading packages:
pcs stonith level add 2 an-a04n02.alteeve.ca fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on
(1/9): fence-agents-virsh-4.0.3-1.fc19.x86_64.rpm                                | 7.7 kB  00:00:01   
</syntaxhighlight>
(2/9): cyrus-sasl-md5-2.1.26-9.fc19.x86_64.rpm                                    |  54 kB  00:00:01   
(3/9): libwsman1-2.3.6-6.fc19.x86_64.rpm                                          | 120 kB  00:00:01   
(4/9): numactl-libs-2.0.8-4.fc19.x86_64.rpm                                      |  28 kB  00:00:00   
(5/9): pm-utils-1.4.1-24.fc19.x86_64.rpm                                          | 139 kB  00:00:00   
(6/9): nmap-ncat-6.40-2.fc19.x86_64.rpm                                          | 198 kB  00:00:01   
(7/9): yajl-2.0.4-2.fc19.x86_64.rpm                                              |  38 kB  00:00:00   
(8/9): libvirt-client-1.0.5.5-1.fc19.x86_64.rpm                                  | 4.9 MB  00:00:12   
(9/9): gnutls-utils-3.1.11-1.fc19.x86_64.rpm                                      | 261 kB  00:01:28   
---------------------------------------------------------------------------------------------------------
Total                                                                    66 kB/s | 5.7 MB    01:28   
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : cyrus-sasl-md5-2.1.26-9.fc19.x86_64                                                  1/9
  Installing : yajl-2.0.4-2.fc19.x86_64                                                              2/9
  Installing : 2:nmap-ncat-6.40-2.fc19.x86_64                                                        3/9
  Installing : libwsman1-2.3.6-6.fc19.x86_64                                                        4/9
  Installing : numactl-libs-2.0.8-4.fc19.x86_64                                                      5/9
  Installing : gnutls-utils-3.1.11-1.fc19.x86_64                                                    6/9
  Installing : pm-utils-1.4.1-24.fc19.x86_64                                                        7/9
  Installing : libvirt-client-1.0.5.5-1.fc19.x86_64                                                  8/9
  Installing : fence-agents-virsh-4.0.3-1.fc19.x86_64                                                9/9
  Verifying  : pm-utils-1.4.1-24.fc19.x86_64                                                        1/9
  Verifying  : gnutls-utils-3.1.11-1.fc19.x86_64                                                    2/9
  Verifying  : numactl-libs-2.0.8-4.fc19.x86_64                                                      3/9
  Verifying  : libwsman1-2.3.6-6.fc19.x86_64                                                        4/9
  Verifying  : 2:nmap-ncat-6.40-2.fc19.x86_64                                                        5/9
  Verifying  : yajl-2.0.4-2.fc19.x86_64                                                              6/9
  Verifying  : fence-agents-virsh-4.0.3-1.fc19.x86_64                                                7/9
  Verifying  : libvirt-client-1.0.5.5-1.fc19.x86_64                                                  8/9
  Verifying  : cyrus-sasl-md5-2.1.26-9.fc19.x86_64                                                  9/9


Installed:
Check again and we'll see that the new methods were added.
  fence-agents-virsh.x86_64 0:4.0.3-1.fc19                                                             


Dependency Installed:
<syntaxhighlight lang="bash">
  cyrus-sasl-md5.x86_64 0:2.1.26-9.fc19                gnutls-utils.x86_64 0:3.1.11-1.fc19             
pcs stonith level
  libvirt-client.x86_64 0:1.0.5.5-1.fc19              libwsman1.x86_64 0:2.3.6-6.fc19                 
  nmap-ncat.x86_64 2:6.40-2.fc19                      numactl-libs.x86_64 0:2.0.8-4.fc19             
  pm-utils.x86_64 0:1.4.1-24.fc19                      yajl.x86_64 0:2.0.4-2.fc19                     
 
Complete!
</syntaxhighlight>
</syntaxhighlight>
Now test it from the command line. To do this, we need to know a few things;
* The VM host is at IP <span class="code">192.168.122.1</span>
* The username and password (<span class="code">-l</span> and <span class="code">-p</span> respectively) are the credentials used to log into VM host over [[SSH]].
* The name of the target VM, as shown by <span class="code">virsh</span>, is the node (<span class="code">-n</span>) value
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
fence_virsh -a 192.168.122.1 -l root -p "secret" -n pcmk2 -o status
Node: an-a04n01.alteeve.ca
</syntaxhighlight>
  Level 1 - fence_n01_ipmi
<syntaxhighlight lang="text">
  Level 2 - fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on
Status: ON
Node: an-a04n02.alteeve.ca
  Level 1 - fence_n02_ipmi
  Level 2 - fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on
</syntaxhighlight>
</syntaxhighlight>


Excellent! Now to configure it in pacemaker;
For those of us who are [[XML]] fans, this is what the [[cib]] looks like now:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
pcs stonith create fence_pcmk1_virsh fence_virsh ipaddr="192.168.122.1" login="root" passwd="secret" port="pcmk1" delay="15" pcmk_host_list="pcmk1.alteeve.ca"
cat /var/lib/pacemaker/cib/cib.xml
pcs stonith create fence_pcmk2_virsh fence_virsh ipaddr="192.168.122.1" login="root" passwd="secret" port="pcmk2" pcmk_host_list="pcmk2.alteeve.ca"
pcs status
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="xml">
Cluster name: an-pcmk-01
<cib epoch="18" num_updates="0" admin_epoch="0" validate-with="pacemaker-1.2" cib-last-written="Thu Jul 18 13:15:53 2013" update-origin="an-a04n01.alteeve.ca" update-client="cibadmin" crm_feature_set="3.0.7" have-quorum="1" dc-uuid="1">
Last updated: Mon Sep 30 12:08:30 2013
  <configuration>
Last change: Sun Sep 29 22:18:40 2013 via cibadmin on pcmk2.alteeve.ca
    <crm_config>
Stack: corosync
      <cluster_property_set id="cib-bootstrap-options">
Current DC: pcmk1.alteeve.ca (1) - partition with quorum
        <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.9-dde1c52"/>
Version: 1.1.10-1.fc19-368c726
        <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
2 Nodes configured
        <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/>
4 Resources configured
      </cluster_property_set>
 
    </crm_config>
 
    <nodes>
Online: [ pcmk1.alteeve.ca pcmk2.alteeve.ca ]
      <node id="1" uname="an-a04n01.alteeve.ca"/>
 
      <node id="2" uname="an-a04n02.alteeve.ca"/>
Full list of resources:
    </nodes>
 
    <resources>
fence_pcmk1_virsh (stonith:fence_virsh): Started pcmk1.alteeve.ca  
      <primitive class="stonith" id="fence_n01_ipmi" type="fence_ipmilan">
fence_pcmk2_virsh (stonith:fence_virsh): Started pcmk2.alteeve.ca  
        <instance_attributes id="fence_n01_ipmi-instance_attributes">
</syntaxhighlight>
          <nvpair id="fence_n01_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n01.alteeve.ca"/>
 
          <nvpair id="fence_n01_ipmi-instance_attributes-ipaddr" name="ipaddr" value="an-a04n01.ipmi"/>
= Shared Storage =
          <nvpair id="fence_n01_ipmi-instance_attributes-action" name="action" value="reboot"/>
 
          <nvpair id="fence_n01_ipmi-instance_attributes-login" name="login" value="admin"/>
== DRBD ==
          <nvpair id="fence_n01_ipmi-instance_attributes-passwd" name="passwd" value="secret"/>
 
          <nvpair id="fence_n01_ipmi-instance_attributes-delay" name="delay" value="15"/>
We will use DRBD 8.4.
        </instance_attributes>
 
        <operations>
<syntaxhighlight lang="bash">
          <op id="fence_n01_ipmi-monitor-interval-60s" interval="60s" name="monitor"/>
yum -y install drbd drbd-pacemaker drbd-bash-completion
        </operations>
</syntaxhighlight>
      </primitive>
 
      <primitive class="stonith" id="fence_n02_ipmi" type="fence_ipmilan">
Configure <span class="code">global-common.conf</span>;
        <instance_attributes id="fence_n02_ipmi-instance_attributes">
 
          <nvpair id="fence_n02_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n02.alteeve.ca"/>
<syntaxhighlight lang="bash">
          <nvpair id="fence_n02_ipmi-instance_attributes-ipaddr" name="ipaddr" value="an-a04n02.ipmi"/>
vim /etc/drbd.d/global_common.conf
          <nvpair id="fence_n02_ipmi-instance_attributes-action" name="action" value="reboot"/>
</syntaxhighlight>
          <nvpair id="fence_n02_ipmi-instance_attributes-login" name="login" value="admin"/>
<syntaxhighlight lang="bash">
          <nvpair id="fence_n02_ipmi-instance_attributes-passwd" name="passwd" value="secret"/>
# These are options to set for the DRBD daemon sets the default values for
        </instance_attributes>
# resources.
        <operations>
global {
          <op id="fence_n02_ipmi-monitor-interval-60s" interval="60s" name="monitor"/>
# This tells DRBD that you allow it to report this installation to
        </operations>
# LINBIT for statistical purposes. If you have privacy concerns, set
      </primitive>
# this to 'no'. The default is 'ask' which will prompt you each time
      <primitive class="stonith" id="fence_n01_pdu1_off" type="fence_apc_snmp">
# DRBD is updated. Set to 'yes' to allow it without being prompted.
        <instance_attributes id="fence_n01_pdu1_off-instance_attributes">
usage-count no;
          <nvpair id="fence_n01_pdu1_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n01.alteeve.ca"/>
 
          <nvpair id="fence_n01_pdu1_off-instance_attributes-ipaddr" name="ipaddr" value="an-pdu01"/>
# minor-count dialog-refresh disable-ip-verification
          <nvpair id="fence_n01_pdu1_off-instance_attributes-action" name="action" value="off"/>
}
          <nvpair id="fence_n01_pdu1_off-instance_attributes-port" name="port" value="1"/>
 
        </instance_attributes>
common {
        <operations>
handlers {
          <op id="fence_n01_pdu1_off-monitor-interval-60s" interval="60s" name="monitor"/>
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
        </operations>
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
      </primitive>
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
      <primitive class="stonith" id="fence_n01_pdu2_off" type="fence_apc_snmp">
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
        <instance_attributes id="fence_n01_pdu2_off-instance_attributes">
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
          <nvpair id="fence_n01_pdu2_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n01.alteeve.ca"/>
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
          <nvpair id="fence_n01_pdu2_off-instance_attributes-ipaddr" name="ipaddr" value="an-pdu02"/>
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
          <nvpair id="fence_n01_pdu2_off-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_n01_pdu2_off-instance_attributes-port" name="port" value="1"/>
# Hook into Pacemaker's fencing.
          <nvpair id="fence_n01_pdu2_off-instance_attributes-power_wait" name="power_wait" value="5"/>
fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
        </instance_attributes>
}
        <operations>
 
          <op id="fence_n01_pdu2_off-monitor-interval-60s" interval="60s" name="monitor"/>
startup {
        </operations>
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
      </primitive>
}
      <primitive class="stonith" id="fence_n01_pdu1_on" type="fence_apc_snmp">
 
        <instance_attributes id="fence_n01_pdu1_on-instance_attributes">
options {
          <nvpair id="fence_n01_pdu1_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n01.alteeve.ca"/>
# cpu-mask on-no-data-accessible
          <nvpair id="fence_n01_pdu1_on-instance_attributes-ipaddr" name="ipaddr" value="an-pdu01"/>
}
          <nvpair id="fence_n01_pdu1_on-instance_attributes-action" name="action" value="on"/>
 
          <nvpair id="fence_n01_pdu1_on-instance_attributes-port" name="port" value="1"/>
disk {
        </instance_attributes>
# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
        <operations>
# disk-drain md-flushes resync-rate resync-after al-extents
          <op id="fence_n01_pdu1_on-monitor-interval-60s" interval="60s" name="monitor"/>
                # c-plan-ahead c-delay-target c-fill-target c-max-rate
        </operations>
                # c-min-rate disk-timeout
      </primitive>
                fencing resource-and-stonith;
      <primitive class="stonith" id="fence_n01_pdu2_on" type="fence_apc_snmp">
}
        <instance_attributes id="fence_n01_pdu2_on-instance_attributes">
 
          <nvpair id="fence_n01_pdu2_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n01.alteeve.ca"/>
net {
          <nvpair id="fence_n01_pdu2_on-instance_attributes-ipaddr" name="ipaddr" value="an-pdu02"/>
# protocol timeout max-epoch-size max-buffers unplug-watermark
          <nvpair id="fence_n01_pdu2_on-instance_attributes-action" name="action" value="on"/>
# connect-int ping-int sndbuf-size rcvbuf-size ko-count
          <nvpair id="fence_n01_pdu2_on-instance_attributes-port" name="port" value="1"/>
# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
        </instance_attributes>
# after-sb-1pri after-sb-2pri always-asbp rr-conflict
        <operations>
# ping-timeout data-integrity-alg tcp-cork on-congestion
          <op id="fence_n01_pdu2_on-monitor-interval-60s" interval="60s" name="monitor"/>
# congestion-fill congestion-extents csums-alg verify-alg
        </operations>
# use-rle
      </primitive>
 
      <primitive class="stonith" id="fence_n02_pdu1_off" type="fence_apc_snmp">
# Protocol "C" tells DRBD not to tell the operating system that
        <instance_attributes id="fence_n02_pdu1_off-instance_attributes">
# the write is complete until the data has reach persistent
          <nvpair id="fence_n02_pdu1_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n02.alteeve.ca"/>
# storage on both nodes. This is the slowest option, but it is
          <nvpair id="fence_n02_pdu1_off-instance_attributes-ipaddr" name="ipaddr" value="an-pdu01"/>
# also the only one that guarantees consistency between the
          <nvpair id="fence_n02_pdu1_off-instance_attributes-action" name="action" value="off"/>
# nodes. It is also required for dual-primary, which we will
          <nvpair id="fence_n02_pdu1_off-instance_attributes-port" name="port" value="2"/>
# be using.
        </instance_attributes>
protocol C;
        <operations>
 
          <op id="fence_n02_pdu1_off-monitor-interval-60s" interval="60s" name="monitor"/>
# Tell DRBD to allow dual-primary. This is needed to enable
        </operations>
# live-migration of our servers.
      </primitive>
allow-two-primaries yes;
      <primitive class="stonith" id="fence_n02_pdu2_off" type="fence_apc_snmp">
 
        <instance_attributes id="fence_n02_pdu2_off-instance_attributes">
# This tells DRBD what to do in the case of a split-brain when
          <nvpair id="fence_n02_pdu2_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n02.alteeve.ca"/>
# neither node was primary, when one node was primary and when
          <nvpair id="fence_n02_pdu2_off-instance_attributes-ipaddr" name="ipaddr" value="an-pdu02"/>
# both nodes are primary. In our case, we'll be running
          <nvpair id="fence_n02_pdu2_off-instance_attributes-action" name="action" value="off"/>
# dual-primary, so we can not safely recover automatically. The
          <nvpair id="fence_n02_pdu2_off-instance_attributes-port" name="port" value="2"/>
# only safe option is for the nodes to disconnect from one
          <nvpair id="fence_n02_pdu2_off-instance_attributes-power_wait" name="power_wait" value="5"/>
# another and let a human decide which node to invalidate. Of
        </instance_attributes>
after-sb-0pri discard-zero-changes;
        <operations>
after-sb-1pri discard-secondary;
          <op id="fence_n02_pdu2_off-monitor-interval-60s" interval="60s" name="monitor"/>
after-sb-2pri disconnect;
        </operations>
}
      </primitive>
}
      <primitive class="stonith" id="fence_n02_pdu1_on" type="fence_apc_snmp">
        <instance_attributes id="fence_n02_pdu1_on-instance_attributes">
          <nvpair id="fence_n02_pdu1_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n02.alteeve.ca"/>
          <nvpair id="fence_n02_pdu1_on-instance_attributes-ipaddr" name="ipaddr" value="an-pdu01"/>
          <nvpair id="fence_n02_pdu1_on-instance_attributes-action" name="action" value="on"/>
          <nvpair id="fence_n02_pdu1_on-instance_attributes-port" name="port" value="2"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_pdu1_on-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n02_pdu2_on" type="fence_apc_snmp">
        <instance_attributes id="fence_n02_pdu2_on-instance_attributes">
          <nvpair id="fence_n02_pdu2_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n02.alteeve.ca"/>
          <nvpair id="fence_n02_pdu2_on-instance_attributes-ipaddr" name="ipaddr" value="an-pdu02"/>
          <nvpair id="fence_n02_pdu2_on-instance_attributes-action" name="action" value="on"/>
          <nvpair id="fence_n02_pdu2_on-instance_attributes-port" name="port" value="2"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_pdu2_on-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
    </resources>
    <constraints/>
    <fencing-topology>
      <fencing-level devices="fence_n01_ipmi" id="fl-an-a04n01.alteeve.ca-1" index="1" target="an-a04n01.alteeve.ca"/>
      <fencing-level devices="fence_n02_ipmi" id="fl-an-a04n02.alteeve.ca-1" index="1" target="an-a04n02.alteeve.ca"/>
      <fencing-level devices="fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on" id="fl-an-a04n01.alteeve.ca-2" index="2" target="an-a04n01.alteeve.ca"/>
      <fencing-level devices="fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on" id="fl-an-a04n02.alteeve.ca-2" index="2" target="an-a04n02.alteeve.ca"/>
    </fencing-topology>
  </configuration>
</cib>
</syntaxhighlight>
</syntaxhighlight>


And now configure the first resource;
== Fencing using fence_virsh ==
 
{{note|1=To write this section, I used two virtual machines called <span class="code">pcmk1</span> and <span class="code">pcmk2</span>.}}
 
If you are trying to learn fencing using KVM or Xen virtual machines, you can use the <span class="code">fence_virsh</span>. You can also use <span class="code">[[Fencing KVM Virtual Servers|fence_virtd]]</span>, which is actually recommended by many, but I have found it to be rather unreliable.
 
To use <span class="code">fence_virsh</span>, first install it.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
vim /etc/drbd.d/r0.res
yum -y install fence-agents-virsh
</syntaxhighlight>
<syntaxhighlight lang="text">
<lots of yum output>
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
# This is the first DRBD resource. If will store the shared file systems and
# the servers designed to run on node 01.
resource r0 {
# These options here are common to both nodes. If for some reason you
# need to set unique values per node, you can move these to the
# 'on <name> { ... }' section.
# This sets the device name of this DRBD resouce.
device /dev/drbd0;


# This tells DRBD what the backing device is for this resource.
Now test it from the command line. To do this, we need to know a few things;
disk /dev/sda5;
* The VM host is at IP <span class="code">192.168.122.1</span>
* The username and password (<span class="code">-l</span> and <span class="code">-p</span> respectively) are the credentials used to log into VM host over [[SSH]].
** If you don't want your password to be shown, create a little shell script that simply prints your password and then use <span class="code">-S /path/to/script</span> instead of <span class="code">-p "secret"</span>.
* The name of the target VM, as shown by <span class="code">virsh list --all</span> on the host, is the node (<span class="code">-n</span>) value. For me, the nodes are called <span class="code">an-a04n01</span> and <span class="code">an-a04n02</span>.
 
=== Create the Password Script ===


# This controls the location of the metadata. When "internal" is used,
In my case, the host is called '<span class="code">lemass</span>', so I want to create a password script called '<span class="code">/root/lemass.pw</span>'. The name of the script is entirely up to you.
# as we use here, a little space at the end of the backing devices is
# set aside (roughly 32 MB per 1 TB of raw storage). External metadata
# can be used to put the metadata on another partition when converting
# existing file systems to be DRBD backed, when there is no extra space
# available for the metadata.
meta-disk internal;


# NOTE: this is not required or even recommended with pacemaker. remove
{|class="wikitable"
# this options as soon as pacemaker is setup.
!<span class="code">an-a04n01</span>
startup {
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
# This tells DRBD to promote both nodes to 'primary' when this
vim /root/lemass.pw
# resource starts. However, we will let pacemaker control this
</syntaxhighlight>
# so we comment it out, which tells DRBD to leave both nodes
<syntaxhighlight lang="text">
# as secondary when drbd starts.
echo "my secret password"
#become-primary-on both;
</syntaxhighlight>
}
<syntaxhighlight lang="bash">
chmod 755 /root/lemass.pw
/root/lemass.pw
</syntaxhighlight>
<syntaxhighlight lang="text">
my secret password
</syntaxhighlight>
<syntaxhighlight lang="bash">
rsync -av /root/lemass.pw root@an-a04n02:/root/
</syntaxhighlight>
<syntaxhighlight lang="bash">
sending incremental file list
lemass.pw


# NOTE: Later, make it an option in the dashboard to trigger a manual
sent 102 bytes  received 31 bytes  266.00 bytes/sec
# verify and/or schedule periodic automatic runs
total size is 25  speedup is 0.19
net {
</syntaxhighlight>
# TODO: Test performance differences between sha1 and md5
|-
# This tells DRBD how to do a block-by-block verification of
!<span class="code">an-a04n02</span>
# the data stored on the backing devices. Any verification
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
# failures will result in the effected block being marked
/root/lemass.pw
# out-of-sync.
</syntaxhighlight>
verify-alg md5;
<syntaxhighlight lang="text">
my secret password
</syntaxhighlight>
|}
 
Done.


# TODO: Test the performance hit of this being enabled.
=== Test fence_virsh Status from the Command Line ===
# This tells DRBD to generate a checksum for each transmitted
# packet. If the data received data doesn't generate the same
# sum, a retransmit request is generated. This protects against
# otherwise-undetected errors in transmission, like
# bit-flipping. See:
# http://www.drbd.org/users-guide/s-integrity-check.html
data-integrity-alg md5;
}


# WARNING: Confirm that these are safe when the controller's BBU is
{|class="wikitable"
#          depleted/failed and the controller enters write-through
!<span class="code">an-a04n01</span>
#          mode.
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
disk {
fence_virsh -a 192.168.122.1 -l root -S /root/lemass.pw -n an-a04n02 -o status
# TODO: Test the real-world performance differences gained with
</syntaxhighlight>
#      these options.
<syntaxhighlight lang="text">
# This tells DRBD not to bypass the write-back caching on the
Status: ON
# RAID controller. Normally, DRBD forces the data to be flushed
</syntaxhighlight>
# to disk, rather than allowing the write-back cachine to
|-
# handle it. Normally this is dangerous, but with BBU-backed
!<span class="code">an-a04n02</span>
# caching, it is safe. The first option disables disk flushing
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
# and the second disabled metadata flushes.
fence_virsh -a 192.168.122.1 -l root -S /root/lemass.pw -n an-a04n01 -o status
disk-flushes no;
</syntaxhighlight>
md-flushes no;
<syntaxhighlight lang="text">
}
Status: ON
</syntaxhighlight>
|}
 
Excellent! Now to configure it in pacemaker;


# This sets up the resource on node 01. The name used below must be the
{|class="wikitable"
# named returned by "uname -n".
!<span class="code">an-a04n01</span>
on an-c03n01.alteeve.ca {
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
# This is the address and port to use for DRBD traffic on this
pcs stonith create fence_n01_virsh fence_virsh pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="192.168.122.1" action="reboot" login="root" passwd_script="/root/lemass.pw" port="an-a04n01" delay=15 op monitor interval=60s
# node. Multiple resources can use the same IP but the ports
pcs stonith create fence_n02_virsh fence_virsh pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="192.168.122.1" action="reboot" login="root" passwd_script="/root/lemass.pw" port="an-a04n02" op monitor interval=60s
# must differ. By convention, the first resource uses 7788, the
pcs cluster status
# second uses 7789 and so on, incrementing by one for each
</syntaxhighlight>
# additional resource.  
<syntaxhighlight lang="text">
address 10.10.30.1:7788;
Cluster Status:
}
Last updated: Sun Jan 26 15:45:31 2014
on an-c03n02.alteeve.ca {
Last change: Sun Jan 26 15:06:14 2014 via crmd on an-a04n01.alteeve.ca
address 10.10.30.2:7788;
Stack: corosync
}
Current DC: an-a04n02.alteeve.ca (2) - partition with quorum
}
Version: 1.1.10-19.el7-368c726
2 Nodes configured
2 Resources configured
 
PCSD Status:
an-a04n01.alteeve.ca:
  an-a04n01.alteeve.ca: Online
an-a04n02.alteeve.ca:
  an-a04n02.alteeve.ca: Online
</syntaxhighlight>
</syntaxhighlight>
|}
=== Test Fencing ===
ToDo: Kill each node with <span class="code">echo c > /proc/sysrq-trigger</span> and make sure the other node fences it.


Disable <span class="code">drbd</span> from starting on boot.
= Shared Storage =


<syntaxhighlight lang="bash">
== DRBD ==
systemctl disable drbd.service
</syntaxhighlight>
<syntaxhighlight lang="text">
drbd.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig drbd off
</syntaxhighlight>


Load the config;
We will use DRBD 8.4.


<syntaxhighlight lang="bash">
=== Install DRBD 8.4.4 from AN! ===
modprobe drbd
</syntaxhighlight>


Now check the config;
{{warning|1=this doesn't work.}}


<syntaxhighlight lang="bash">
ToDo: Make a proper repo
drbdadm dump
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
rpm -Uvh https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-8.4.4-4.el7.x86_64.rpm \
        https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-bash-completion-8.4.4-4.el7.x86_64.rpm \
        https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-pacemaker-8.4.4-4.el7.x86_64.rpm \
        https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-udev-8.4.4-4.el7.x86_64.rpm \
        https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-utils-8.4.4-4.el7.x86_64.rpm \
        https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-heartbeat-8.4.4-4.el7.x86_64.rpm \
        https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-xen-8.4.4-4.el7.x86_64.rpm
</syntaxhighlight>
<syntaxhighlight lang="text">
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
  --==  Thank you for participating in the global usage survey  ==--
</syntaxhighlight>
The server's response is:
|}
 


you are the 69th user to install this version
=== Install DRBD 8.4.4 From Source ===
/etc/drbd.d/r0.res:3: in resource r0:
become-primary-on is set to both, but allow-two-primaries is not set.
</syntaxhighlight>


Ignore that error. It has been reported and does not effect operation.
At this time, no EPEL repo exists for RHEL7, and the Fedora RPMs don't work, so we will install DRBD 8.4.4 from source.


Create the metadisk;
Install dependencies:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
drbdadm create-md r0
yum -y install gcc flex rpm-build wget kernel-devel
</syntaxhighlight>
wget -c http://oss.linbit.com/drbd/8.4/drbd-8.4.4.tar.gz
<syntaxhighlight lang="text">
tar -xvzf drbd-8.4.4.tar.gz
Writing meta data...
cd drbd-8.4.4
initializing activity log
./configure \
NOT initializing bitmap
  --prefix=/usr \
New drbd meta data block successfully created.
  --localstatedir=/var \
success
  --sysconfdir=/etc \
  --with-km \
  --with-udev \
  --with-pacemaker \
  --with-bashcompletion \
  --with-utils \
  --without-xen \
  --without-rgmanager \
  --without-heartbeat
make
make install
</syntaxhighlight>
</syntaxhighlight>


Start the DRBD resource on both nodes;
Don't let DRBD start on boot (pacemaker will handle it for us).


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
drbdadm up r0
systemctl disable drbd.service
</syntaxhighlight>
<syntaxhighlight lang="text">
drbd.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig drbd off
</syntaxhighlight>
</syntaxhighlight>


Once <span class="code">/proc/drbd</span> shows both nodes connected, force one to primary and it will sync over the second.
Done.
 
=== Optional; Make RPMs ===
 
{{warning|1=I've not been able to get the RPMs genreated here to install yet. I'd recommend skipping this, unless you want to help sort out the problems. :) }}


<syntaxhighlight lang="bash">
After <span class="code">./configure</span> above, you can make RPMs instead of installing directly.
drbdadm primary --force r0
</syntaxhighlight>


You should see the resource syncing now. Push both nodes to primary;
Dependencies:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
drbdadm primary r0
yum install rpmdevtools redhat-rpm-config kernel-devel
</syntaxhighlight>
<syntaxhighlight lang="text">
<install text>
</syntaxhighlight>
</syntaxhighlight>


== DLM, Clustered LVM and GFS2 ==
Setup RPM dev tree:
 
Install [[DLM]] and [[GFS2]];


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
yum -y install dlm dlm-lib lvm2-cluster gfs2-utils
cd ~
rpmdev-setuptree
ls -lah ~/rpmbuild/
wget -c http://oss.linbit.com/drbd/8.4/drbd-8.4.4.tar.gz
tar -xvzf drbd-8.4.4.tar.gz
cd drbd-8.4.4
./configure \
  --prefix=/usr \
  --localstatedir=/var \
  --sysconfdir=/etc \
  --with-km \
  --with-udev \
  --with-pacemaker \
  --with-bashcompletion \
  --with-utils \
  --without-xen \
  --without-heartbeat
</syntaxhighlight>
<syntaxhighlight lang="text">
total 4.0K
drwxr-xr-x. 7 root root  67 Dec 23 20:06 .
dr-xr-x---. 6 root root 4.0K Dec 23 20:06 ..
drwxr-xr-x. 2 root root    6 Dec 23 20:06 BUILD
drwxr-xr-x. 2 root root    6 Dec 23 20:06 RPMS
drwxr-xr-x. 2 root root    6 Dec 23 20:06 SOURCES
drwxr-xr-x. 2 root root    6 Dec 23 20:06 SPECS
drwxr-xr-x. 2 root root    6 Dec 23 20:06 SRPMS
</syntaxhighlight>
</syntaxhighlight>


Disable dlm and clvmd from starting on boot.
Userland tools:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl disable clvmd.service
make rpm
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
clvmd.service is not a native service, redirecting to /sbin/chkconfig.
checking for presence of 8\.4\.4 in various changelog files
Executing /sbin/chkconfig clvmd off
<snip>
+ exit 0
You have now:
/root/rpmbuild/RPMS/x86_64/drbd-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-utils-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-xen-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-udev-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-pacemaker-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-heartbeat-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-bash-completion-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-debuginfo-8.4.4-4.el7.x86_64.rpm
</syntaxhighlight>
</syntaxhighlight>
Kernel module:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl disable dlm.service
make kmp-rpm
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
rm '/etc/systemd/system/multi-user.target.wants/dlm.service'
checking for presence of 8\.4\.4 in various changelog files
<snip>
+ exit 0
You have now:
/root/rpmbuild/RPMS/x86_64/drbd-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-utils-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-xen-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-udev-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-pacemaker-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-heartbeat-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-bash-completion-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-debuginfo-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/kmod-drbd-8.4.4_3.10.0_54.0.1-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-kernel-debuginfo-8.4.4-4.el7.x86_64.rpm
</syntaxhighlight>
</syntaxhighlight>


Edit lvm.conf;
=== Configure DRBD ===
 
Configure <span class="code">global-common.conf</span>;


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
diff -u /etc/lvm/lvm.conf.orig /etc/lvm/lvm.conf
vim /etc/drbd.d/global_common.conf
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="diff">
<syntaxhighlight lang="bash">
--- /etc/lvm/lvm.conf.orig 2013-07-08 16:38:45.603780083 -0500
# These are options to set for the DRBD daemon sets the default values for
+++ /etc/lvm/lvm.conf 2013-07-08 16:47:34.434591848 -0500
# resources.
@@ -65,7 +65,7 @@
global {
# This tells DRBD that you allow it to report this installation to
# LINBIT for statistical purposes. If you have privacy concerns, set
    # By default we accept every block device:
# this to 'no'. The default is 'ask' which will prompt you each time
-   filter = [ "a/.*/" ]
# DRBD is updated. Set to 'yes' to allow it without being prompted.
+    filter = [ "a|/dev/drbd*|", "r/.*/" ]
usage-count no;
 
    # Exclude the cdrom drive
# minor-count dialog-refresh disable-ip-verification
    # filter = [ "r|/dev/cdrom|" ]
}
@@ -405,7 +405,7 @@
 
    # Type 3 uses built-in clustered locking.
common {
    # Type 4 uses read-only locking which forbids any operations that might
handlers {
    # change metadata.
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
-   locking_type = 1
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
+    locking_type = 3
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
    # Set to 0 to fail when a lock request cannot be satisfied immediately.
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
    wait_for_locks = 1
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
@@ -421,7 +421,7 @@
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
    # to 1 an attempt will be made to use local file-based locking (type 1).
    # If this succeeds, only commands against local volume groups will proceed.
# Hook into Pacemaker's fencing.
    # Volume Groups marked as clustered will be ignored.
fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
-    fallback_to_local_locking = 1
}
+    fallback_to_local_locking = 0
 
startup {
    # Local non-LV directory that holds file-based locks while commands are
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
    # in progress.  A directory like /tmp that may get wiped on reboot is OK.
}
@@ -508,7 +508,7 @@
 
    #
options {
    # If lvmetad has been running while use_lvmetad was 0, it MUST be stopped
# cpu-mask on-no-data-accessible
    # before changing use_lvmetad to 1 and started again afterwards.
}
-   use_lvmetad = 1
 
+    use_lvmetad = 0
disk {
# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
    # Full path of the utility called to check that a thin metadata device
# disk-drain md-flushes resync-rate resync-after al-extents
    # is in a state that allows it to be used.
                # c-plan-ahead c-delay-target c-fill-target c-max-rate
                # c-min-rate disk-timeout
                fencing resource-and-stonith;
}
 
net {
# protocol timeout max-epoch-size max-buffers unplug-watermark
# connect-int ping-int sndbuf-size rcvbuf-size ko-count
# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
# after-sb-1pri after-sb-2pri always-asbp rr-conflict
# ping-timeout data-integrity-alg tcp-cork on-congestion
# congestion-fill congestion-extents csums-alg verify-alg
# use-rle
 
# Protocol "C" tells DRBD not to tell the operating system that
# the write is complete until the data has reach persistent
# storage on both nodes. This is the slowest option, but it is
# also the only one that guarantees consistency between the
# nodes. It is also required for dual-primary, which we will  
# be using.
protocol C;
 
# Tell DRBD to allow dual-primary. This is needed to enable
# live-migration of our servers.
allow-two-primaries yes;
 
# This tells DRBD what to do in the case of a split-brain when
# neither node was primary, when one node was primary and when
# both nodes are primary. In our case, we'll be running
# dual-primary, so we can not safely recover automatically. The
# only safe option is for the nodes to disconnect from one
# another and let a human decide which node to invalidate. Of
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
}
}
</syntaxhighlight>
</syntaxhighlight>
 
 
 
And now configure the first resource;
Disable <span class="code">lvmetad</span> as it's not cluster-aware.
 
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
vim /etc/drbd.d/r0.res
systemctl disable lvm2-lvmetad.service
</syntaxhighlight>
systemctl disable lvm2-lvmetad.socket
<syntaxhighlight lang="bash">
systemctl stop lvm2-lvmetad.service
# This is the first DRBD resource. If will store the shared file systems and
</syntaxhighlight>
# the servers designed to run on node 01.
 
resource r0 {
{{note|1=This will be moved to pacemaker}}
# These options here are common to both nodes. If for some reason you
 
# need to set unique values per node, you can move these to the
Start DLM and clvmd;
# 'on <name> { ... }' section.
 
<syntaxhighlight lang="bash">
# This sets the device name of this DRBD resouce.
systemctl start dlm.service
device /dev/drbd0;
systemctl start clvmd.service
 
# This tells DRBD what the backing device is for this resource.
disk /dev/sda5;
 
# This controls the location of the metadata. When "internal" is used,
# as we use here, a little space at the end of the backing devices is
# set aside (roughly 32 MB per 1 TB of raw storage). External metadata
# can be used to put the metadata on another partition when converting
# existing file systems to be DRBD backed, when there is no extra space
# available for the metadata.
meta-disk internal;
 
# NOTE: this is not required or even recommended with pacemaker. remove
# this options as soon as pacemaker is setup.
startup {
# This tells DRBD to promote both nodes to 'primary' when this
# resource starts. However, we will let pacemaker control this
# so we comment it out, which tells DRBD to leave both nodes
# as secondary when drbd starts.
#become-primary-on both;
}
 
# NOTE: Later, make it an option in the dashboard to trigger a manual
# verify and/or schedule periodic automatic runs
net {
# TODO: Test performance differences between sha1 and md5
# This tells DRBD how to do a block-by-block verification of
# the data stored on the backing devices. Any verification
# failures will result in the effected block being marked
# out-of-sync.
verify-alg md5;
 
# TODO: Test the performance hit of this being enabled.
# This tells DRBD to generate a checksum for each transmitted
# packet. If the data received data doesn't generate the same
# sum, a retransmit request is generated. This protects against
# otherwise-undetected errors in transmission, like
# bit-flipping. See:
# http://www.drbd.org/users-guide/s-integrity-check.html
data-integrity-alg md5;
}
 
# WARNING: Confirm that these are safe when the controller's BBU is
#          depleted/failed and the controller enters write-through
#          mode.
disk {
# TODO: Test the real-world performance differences gained with
#      these options.
# This tells DRBD not to bypass the write-back caching on the
# RAID controller. Normally, DRBD forces the data to be flushed
# to disk, rather than allowing the write-back cachine to
# handle it. Normally this is dangerous, but with BBU-backed
# caching, it is safe. The first option disables disk flushing
# and the second disabled metadata flushes.
disk-flushes no;
md-flushes no;
}
 
# This sets up the resource on node 01. The name used below must be the
# named returned by "uname -n".
on an-a04n01.alteeve.ca {
# This is the address and port to use for DRBD traffic on this
# node. Multiple resources can use the same IP but the ports
# must differ. By convention, the first resource uses 7788, the
# second uses 7789 and so on, incrementing by one for each
# additional resource.
address 10.10.40.1:7788;
}
on an-a04n02.alteeve.ca {
address 10.10.40.2:7788;
}
}
</syntaxhighlight>
 
Disable <span class="code">drbd</span> from starting on boot.
 
<syntaxhighlight lang="bash">
systemctl disable drbd.service
</syntaxhighlight>
<syntaxhighlight lang="text">
drbd.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig drbd off
</syntaxhighlight>
 
Load the config;
 
<syntaxhighlight lang="bash">
modprobe drbd
</syntaxhighlight>
 
Now check the config;
 
<syntaxhighlight lang="bash">
drbdadm dump
</syntaxhighlight>
<syntaxhighlight lang="text">
  --==  Thank you for participating in the global usage survey  ==--
The server's response is:
 
you are the 69th user to install this version
/etc/drbd.d/r0.res:3: in resource r0:
become-primary-on is set to both, but allow-two-primaries is not set.
</syntaxhighlight>
 
Ignore that error. It has been reported and does not effect operation.
 
Create the metadisk;
 
<syntaxhighlight lang="bash">
drbdadm create-md r0
</syntaxhighlight>
<syntaxhighlight lang="text">
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success
</syntaxhighlight>
 
Start the DRBD resource on both nodes;
 
<syntaxhighlight lang="bash">
drbdadm up r0
</syntaxhighlight>
 
Once <span class="code">/proc/drbd</span> shows both nodes connected, force one to primary and it will sync over the second.
 
<syntaxhighlight lang="bash">
drbdadm primary --force r0
</syntaxhighlight>
 
You should see the resource syncing now. Push both nodes to primary;
 
<syntaxhighlight lang="bash">
drbdadm primary r0
</syntaxhighlight>
 
== DLM, Clustered LVM and GFS2 ==
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
sed -i.anvil 's^filter = \[ "a/\.\*/" \]^filter = \[ "a|/dev/drbd*|", "r/.*/" \]^' /etc/lvm/lvm.conf
sed -i 's/locking_type = 1$/locking_type = 3/' /etc/lvm/lvm.conf
sed -i 's/fallback_to_local_locking = 1$/fallback_to_local_locking = 0/' /etc/lvm/lvm.conf
sed -i 's/use_lvmetad = 1$/use_lvmetad = 0/' /etc/lvm/lvm.conf
</syntaxhighlight>
<syntaxhighlight lang="diff">
--- /etc/lvm/lvm.conf.anvil 2013-11-27 03:28:08.000000000 -0500
+++ /etc/lvm/lvm.conf 2014-01-26 18:57:41.026928464 -0500
@@ -84,7 +84,7 @@
    # lvmetad is used" comment that is attached to global/use_lvmetad setting.
    # By default we accept every block device:
-    filter = [ "a/.*/" ]
+    filter = [ "a|/dev/drbd*|", "r/.*/" ]
    # Exclude the cdrom drive
    # filter = [ "r|/dev/cdrom|" ]
@@ -451,7 +451,7 @@
    # supported in clustered environment. If use_lvmetad=1 and locking_type=3
    # is set at the same time, LVM always issues a warning message about this
    # and then it automatically disables lvmetad use.
-    locking_type = 1
+    locking_type = 3
    # Set to 0 to fail when a lock request cannot be satisfied immediately.
    wait_for_locks = 1
@@ -467,7 +467,7 @@
    # to 1 an attempt will be made to use local file-based locking (type 1).
    # If this succeeds, only commands against local volume groups will proceed.
    # Volume Groups marked as clustered will be ignored.
-    fallback_to_local_locking = 1
+    fallback_to_local_locking = 0
    # Local non-LV directory that holds file-based locks while commands are
    # in progress.  A directory like /tmp that may get wiped on reboot is OK.
@@ -594,7 +594,7 @@
    # supported in clustered environment. If use_lvmetad=1 and locking_type=3
    # is set at the same time, LVM always issues a warning message about this
    # and then it automatically disables lvmetad use.
-    use_lvmetad = 1
+    use_lvmetad = 0
    # Full path of the utility called to check that a thin metadata device
    # is in a state that allows it to be used.
</syntaxhighlight>
<syntaxhighlight lang="bash">
rsync -av /etc/lvm/lvm.conf* root@an-a04n02:/etc/lvm/
</syntaxhighlight>
<syntaxhighlight lang="text">
sending incremental file list
lvm.conf
lvm.conf.anvil
 
sent 48536 bytes  received 440 bytes  97952.00 bytes/sec
total size is 90673  speedup is 1.85
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
diff -u /etc/lvm/lvm.conf.anvil /etc/lvm/lvm.conf
</syntaxhighlight>
<syntaxhighlight lang="diff">
--- /etc/lvm/lvm.conf.anvil 2013-11-27 03:28:08.000000000 -0500
+++ /etc/lvm/lvm.conf 2014-01-26 18:57:41.000000000 -0500
@@ -84,7 +84,7 @@
    # lvmetad is used" comment that is attached to global/use_lvmetad setting.
    # By default we accept every block device:
-    filter = [ "a/.*/" ]
+    filter = [ "a|/dev/drbd*|", "r/.*/" ]
    # Exclude the cdrom drive
    # filter = [ "r|/dev/cdrom|" ]
@@ -451,7 +451,7 @@
    # supported in clustered environment. If use_lvmetad=1 and locking_type=3
    # is set at the same time, LVM always issues a warning message about this
    # and then it automatically disables lvmetad use.
-    locking_type = 1
+    locking_type = 3
    # Set to 0 to fail when a lock request cannot be satisfied immediately.
    wait_for_locks = 1
@@ -467,7 +467,7 @@
    # to 1 an attempt will be made to use local file-based locking (type 1).
    # If this succeeds, only commands against local volume groups will proceed.
    # Volume Groups marked as clustered will be ignored.
-    fallback_to_local_locking = 1
+    fallback_to_local_locking = 0
    # Local non-LV directory that holds file-based locks while commands are
    # in progress.  A directory like /tmp that may get wiped on reboot is OK.
@@ -594,7 +594,7 @@
    # supported in clustered environment. If use_lvmetad=1 and locking_type=3
    # is set at the same time, LVM always issues a warning message about this
    # and then it automatically disables lvmetad use.
-    use_lvmetad = 1
+    use_lvmetad = 0
    # Full path of the utility called to check that a thin metadata device
    # is in a state that allows it to be used.
</syntaxhighlight>
|}
 
Disable <span class="code">lvmetad</span> as it's not cluster-aware.
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
systemctl disable lvm2-lvmetad.service
systemctl disable lvm2-lvmetad.socket
systemctl stop lvm2-lvmetad.service
</syntaxhighlight>
<syntaxhighlight lang="text">
rm '/etc/systemd/system/sockets.target.wants/lvm2-lvmetad.socket'
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
systemctl disable lvm2-lvmetad.service
systemctl disable lvm2-lvmetad.socket
systemctl stop lvm2-lvmetad.service
</syntaxhighlight>
<syntaxhighlight lang="text">
rm '/etc/systemd/system/sockets.target.wants/lvm2-lvmetad.socket'
</syntaxhighlight>
|}
 
{{note|1=This will be moved to pacemaker shortly. We're enabling it here just long enough to configure pacemaker.}}
 
Start DLM and clvmd;
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
systemctl start dlm.service
systemctl start clvmd.service
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
systemctl start dlm.service
systemctl start clvmd.service
</syntaxhighlight>
|}
 
Create the [[PV]], [[VG]] and the <span class="code">/shared</span> [[LV]];
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pvcreate /dev/drbd0
</syntaxhighlight>
<syntaxhighlight lang="text">
  Physical volume "/dev/drbd0" successfully created
</syntaxhighlight>
<syntaxhighlight lang="bash">
vgcreate an-a04n01_vg0 /dev/drbd0
</syntaxhighlight>
<syntaxhighlight lang="text">
  /proc/devices: No entry for device-mapper found
  Clustered volume group "an-a04n01_vg0" successfully created
</syntaxhighlight>
<syntaxhighlight lang="bash">
lvcreate -L 10G -n shared an-a04n01_vg0
</syntaxhighlight>
<syntaxhighlight lang="text">
  Logical volume "shared" created
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pvscan
</syntaxhighlight>
<syntaxhighlight lang="text">
  PV /dev/drbd0  VG an-a04n01_vg0  lvm2 [20.00 GiB / 20.00 GiB free]
  Total: 1 [20.00 GiB] / in use: 1 [20.00 GiB] / in no VG: 0 [0  ]
</syntaxhighlight>
<syntaxhighlight lang="bash">
vgscan
</syntaxhighlight>
<syntaxhighlight lang="text">
  Reading all physical volumes.  This may take a while...
  Found volume group "an-a04n01_vg0" using metadata type lvm2
</syntaxhighlight>
<syntaxhighlight lang="bash">
lvscan
</syntaxhighlight>
<syntaxhighlight lang="text">
  ACTIVE            '/dev/an-a04n01_vg0/shared' [10.00 GiB] inherit
</syntaxhighlight>
|}
 
Format the <span class="code">/dev/an-a04n01_vg0/shared</span>;
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
mkfs.gfs2 -j 2 -p lock_dlm -t an-anvil-04:shared /dev/an-a04n01_vg0/shared
</syntaxhighlight>
<syntaxhighlight lang="text">
/dev/an-a04n01_vg0/shared is a symbolic link to /dev/dm-0
This will destroy any data on /dev/dm-0
</syntaxhighlight>
<syntaxhighlight lang="text">
Are you sure you want to proceed? [y/n]y
</syntaxhighlight>
<syntaxhighlight lang="text">
Device:                    /dev/an-a04n01_vg0/shared
Block size:                4096
Device size:              10.00 GB (2621440 blocks)
Filesystem size:          10.00 GB (2621438 blocks)
Journals:                  2
Resource groups:          40
Locking protocol:          "lock_dlm"
Lock table:                "an-anvil-04:shared"
UUID:                      20bafdb0-1f86-f424-405b-9bf608c0c486
</syntaxhighlight>
<syntaxhighlight lang="bash">
mkdir /shared
mount /dev/an-a04n01_vg0/shared /shared
df -h
</syntaxhighlight>
<syntaxhighlight lang="text">
Filesystem                        Size  Used Avail Use% Mounted on
/dev/vda3                          18G  5.6G  12G  32% /
devtmpfs                          932M    0  932M  0% /dev
tmpfs                              937M  61M  877M  7% /dev/shm
tmpfs                              937M  1.9M  935M  1% /run
tmpfs                              937M    0  937M  0% /sys/fs/cgroup
/dev/loop0                        4.4G  4.4G    0 100% /mnt/dvd
/dev/vda1                          484M  83M  401M  18% /boot
/dev/mapper/an--a03n01_vg0-shared  10G  259M  9.8G  3% /shared
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
</syntaxhighlight>
<syntaxhighlight lang="text">
Filesystem                        Size  Used Avail Use% Mounted on
/dev/vda3                          18G  5.6G  12G  32% /
devtmpfs                          932M    0  932M  0% /dev
tmpfs                              937M  76M  862M  9% /dev/shm
tmpfs                              937M  2.0M  935M  1% /run
tmpfs                              937M    0  937M  0% /sys/fs/cgroup
/dev/loop0                        4.4G  4.4G    0 100% /mnt/dvd
/dev/vda1                          484M  83M  401M  18% /boot
/dev/mapper/an--a03n01_vg0-shared  10G  259M  9.8G  3% /shared
</syntaxhighlight>
|}
 
Shut down <span class="code">gfs2</span>, <span class="code">clvmd</span> and <span class="code">drbd</span> now.
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
umount /shared/
systemctl stop clvmd.service
drbdadm down r0
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
umount /shared/
systemctl stop clvmd.service
drbdadm down r0
</syntaxhighlight>
|}
 
Done.
 
= Add Storage to Pacemaker =
 
== Configure Dual-Primary DRBD ==
 
Setup DRBD as a dual-primary resource.
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pcs cluster cib drbd_cfg
pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 op monitor interval=60s
pcs -f drbd_cfg resource master drbd_r0_Clone drbd_r0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs cluster cib-push drbd_cfg
</syntaxhighlight>
<syntaxhighlight lang="text">
CIB updated
</syntaxhighlight>
|}
 
Give it a couple minutes to promote both nodes to <span class="code">Master</span> on both nodes. Initially, it will appear as <span class="code">Master</span> on one node only.
 
Once updated, you should see this:
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pcs status
</syntaxhighlight>
<syntaxhighlight lang="text">
Cluster name: an-anvil-04
Last updated: Sun Jan 26 20:26:33 2014
Last change: Sun Jan 26 20:23:23 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: corosync
Current DC: an-a04n02.alteeve.ca (2) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
4 Resources configured
 
 
Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 
Full list of resources:
 
fence_n01_virsh (stonith:fence_virsh): Started an-a04n01.alteeve.ca
fence_n02_virsh (stonith:fence_virsh): Started an-a04n02.alteeve.ca
Master/Slave Set: drbd_r0_Clone [drbd_r0]
    Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 
PCSD Status:
an-a04n01.alteeve.ca:
  an-a04n01.alteeve.ca: Online
an-a04n02.alteeve.ca:
  an-a04n02.alteeve.ca: Online
 
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pcs status
</syntaxhighlight>
<syntaxhighlight lang="text">
Cluster name: an-anvil-04
Last updated: Sun Jan 26 20:26:58 2014
Last change: Sun Jan 26 20:23:23 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: corosync
Current DC: an-a04n02.alteeve.ca (2) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
4 Resources configured
 
 
Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 
Full list of resources:
 
fence_n01_virsh (stonith:fence_virsh): Started an-a04n01.alteeve.ca
fence_n02_virsh (stonith:fence_virsh): Started an-a04n02.alteeve.ca
Master/Slave Set: drbd_r0_Clone [drbd_r0]
    Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 
PCSD Status:
an-a04n01.alteeve.ca:
  an-a04n01.alteeve.ca: Online
an-a04n02.alteeve.ca:
  an-a04n02.alteeve.ca: Online
 
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled
</syntaxhighlight>
|}
 
== Configure DLM ==
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pcs cluster cib dlm_cfg
pcs -f dlm_cfg resource create dlm ocf:pacemaker:controld op monitor interval=60s
pcs -f dlm_cfg resource clone dlm clone-max=2 clone-node-max=1
pcs cluster cib-push dlm_cfg
</syntaxhighlight>
<syntaxhighlight lang="text">
CIB updated
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pcs status
</syntaxhighlight>
<syntaxhighlight lang="text">
Cluster name: an-anvil-04
Last updated: Sun Jan 26 20:34:36 2014
Last change: Sun Jan 26 20:33:31 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: corosync
Current DC: an-a04n02.alteeve.ca (2) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
6 Resources configured
 
 
Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 
Full list of resources:
 
fence_n01_virsh (stonith:fence_virsh): Started an-a04n01.alteeve.ca
fence_n02_virsh (stonith:fence_virsh): Started an-a04n02.alteeve.ca
Master/Slave Set: drbd_r0_Clone [drbd_r0]
    Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
Clone Set: dlm-clone [dlm]
    Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 
PCSD Status:
an-a04n01.alteeve.ca:
  an-a04n01.alteeve.ca: Online
an-a04n02.alteeve.ca:
  an-a04n02.alteeve.ca: Online
 
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled
</syntaxhighlight>
|}
 
== Configure Cluster LVM ==
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pcs cluster cib clvmd_cfg
pcs -f clvmd_cfg resource create clvmd lsb:clvmd params daemon_timeout=30s op monitor interval=60s
pcs -f clvmd_cfg resource clone clvmd clone-max=2 clone-node-max=1
pcs -f clvmd_cfg constraint colocation add dlm-clone clvmd-clone INFINITY
pcs -f clvmd_cfg constraint order start dlm then start clvmd-clone
pcs cluster cib-push clvmd_cfg</syntaxhighlight>
<syntaxhighlight lang="text">
CIB updated
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pcs status
</syntaxhighlight>
<syntaxhighlight lang="text">
Cluster name: an-anvil-04
Last updated: Mon Jan 27 19:00:33 2014
Last change: Mon Jan 27 19:00:19 2014 via crm_resource on an-a04n01.alteeve.ca
Stack: corosync
Current DC: an-a04n01.alteeve.ca (1) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
8 Resources configured
 
 
Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 
Full list of resources:
 
fence_n01_virsh        (stonith:fence_virsh):  Started an-a04n01.alteeve.ca
fence_n02_virsh        (stonith:fence_virsh):  Started an-a04n02.alteeve.ca
Master/Slave Set: drbd_r0_Clone [drbd_r0]
    Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
Clone Set: dlm-clone [dlm]
    Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
Clone Set: clvmd-clone [clvmd]
    Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 
PCSD Status:
an-a04n01.alteeve.ca:
  an-a04n01.alteeve.ca: Online
an-a04n02.alteeve.ca:
  an-a04n02.alteeve.ca: Online
 
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled
</syntaxhighlight>
|}
 
== Configure the /shared GFS2 Partition ==
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pcs cluster cib fs_cfg
pcs -f fs_cfg resource create sharedFS Filesystem device="/dev/an-a04n01_vg0/shared" directory="/shared" fstype="gfs2"
pcs -f fs_cfg resource clone sharedFS
pcs cluster cib-push fs_cfg
</syntaxhighlight>
<syntaxhighlight lang="text">
CIB updated
</syntaxhighlight>
<syntaxhighlight lang="bash">
df -h
</syntaxhighlight>
<syntaxhighlight lang="text">
Filesystem                        Size  Used Avail Use% Mounted on
/dev/vda3                          18G  5.6G  12G  32% /
devtmpfs                          932M    0  932M  0% /dev
tmpfs                              937M  61M  877M  7% /dev/shm
tmpfs                              937M  2.2M  935M  1% /run
tmpfs                              937M    0  937M  0% /sys/fs/cgroup
/dev/loop0                        4.4G  4.4G    0 100% /mnt/dvd
/dev/vda1                          484M  83M  401M  18% /boot
/dev/mapper/an--a03n01_vg0-shared  10G  259M  9.8G  3% /shared
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
df -h
</syntaxhighlight>
<syntaxhighlight lang="text">
Filesystem                        Size  Used Avail Use% Mounted on
/dev/vda3                          18G  5.6G  12G  32% /
devtmpfs                          932M    0  932M  0% /dev
tmpfs                              937M  76M  862M  9% /dev/shm
tmpfs                              937M  2.6M  935M  1% /run
tmpfs                              937M    0  937M  0% /sys/fs/cgroup
/dev/loop0                        4.4G  4.4G    0 100% /mnt/dvd
/dev/vda1                          484M  83M  401M  18% /boot
/dev/mapper/an--a03n01_vg0-shared  10G  259M  9.8G  3% /shared
</syntaxhighlight>
|}
 
== Configuring Constraints ==
 
{|class="wikitable"
!<span class="code">an-a04n01</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pcs cluster cib cst_cfg
pcs -f cst_cfg constraint order start dlm then promote drbd_r0_Clone
pcs -f cst_cfg constraint order promote drbd_r0_Clone then start clvmd-clone
pcs -f cst_cfg constraint order promote clvmd-clone then start sharedFS-clone
pcs cluster cib-push cst_cfg
</syntaxhighlight>
<syntaxhighlight lang="text">
CIB updated
</syntaxhighlight>
<syntaxhighlight lang="bash">
pcs constraint show
</syntaxhighlight>
<syntaxhighlight lang="text">
Location Constraints:
Ordering Constraints:
  start dlm then promote drbd_r0_Clone
  promote drbd_r0_Clone then start clvmd-clone
  start clvmd-clone then start sharedFS-clone
Colocation Constraints:
</syntaxhighlight>
|-
!<span class="code">an-a04n02</span>
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pcs constraint show
</syntaxhighlight>
<syntaxhighlight lang="text">
Location Constraints:
Ordering Constraints:
  start dlm then promote drbd_r0_Clone
  promote drbd_r0_Clone then start clvmd-clone
  start clvmd-clone then start sharedFS-clone
Colocation Constraints:
</syntaxhighlight>
|}
 
= Odds and Sods =
 
This is a section for random notes. The stuff here will be integrated into the finished tutorial or removed.
 
== Determine multicast Address ==
 
Useful if you need to ensure that your switch has persistent multicast addresses set.
 
<syntaxhighlight lang="bash">
corosync-cmapctl | grep mcastaddr
</syntaxhighlight>
<syntaxhighlight lang="text">
totem.interface.0.mcastaddr (str) = 239.192.122.199
</syntaxhighlight>
 
 
 
 
 
<syntaxhighlight lang="bash">
</syntaxhighlight>
<syntaxhighlight lang="text">
</syntaxhighlight>
<syntaxhighlight lang="diff">
</syntaxhighlight>
</syntaxhighlight>


Create the [[PV]], [[VG]] and the <span class="code">/shared</span> [[LV]];
{|class="wikitable"
 
!<span class="code">an-a04n01</span>
<syntaxhighlight lang="bash">
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
pvcreate /dev/drbd0
vgcreate an-c03n01_vg0 /dev/drbd0
lvcreate -L 40G -n shared an-c03n01_vg0
</syntaxhighlight>
 
Format the <span class="code">/dev/an-c03n01_vg0/shared</span>;
 
<syntaxhighlight lang="text">
mkfs.gfs2 -j 2 -p lock_dlm -t an-cluster-03:shared /dev/an-c03n01_vg0/shared
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
/dev/an-c03n01_vg0/shared is a symlink to /dev/dm-0
This will destroy any data on /dev/dm-0.
It appears to contain: data
Are you sure you want to proceed? [y/n]y
Device:                    /dev/an-c03n01_vg0/shared
Blocksize:                4096
Device Size                40.00 GB (10485760 blocks)
Filesystem Size:          40.00 GB (10485758 blocks)
Journals:                  2
Resource Groups:          160
Locking Protocol:          "lock_dlm"
Lock Table:                "an-cluster-03:shared"
UUID:                      e96dbbec-add4-c291-083b-381a866a773d
</syntaxhighlight>
</syntaxhighlight>
 
|-
Create the mount points and mount the new file system on both nodes;
!<span class="code">an-a04n02</span>
 
|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
mkdir /shared
mount /dev/an-c03n01_vg0/shared /shared
</syntaxhighlight>
 
 
 
= Odds and Sods =
 
This is a section for random notes. The stuff here will be integrated into the finished tutorial or removed.
 
== Determine multicast Address ==
 
Useful if you need to ensure that your switch has persistent multicast addresses set.
 
<syntaxhighlight lang="bash">
corosync-cmapctl | grep mcastaddr
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
totem.interface.0.mcastaddr (str) = 239.192.122.199
</syntaxhighlight>
</syntaxhighlight>
|}


 
<span class="highlight_warning"></span>
<span class="field"></span>
<span class="button"></span>
<span class="code"></span>
<span class="code"></span>
<syntaxhighlight lang="bash">
</syntaxhighlight>
<syntaxhighlight lang="text">
</syntaxhighlight>
<syntaxhighlight lang="diff">
</syntaxhighlight>


= Notes =
= Notes =


* [http://blog.clusterlabs.org/blog/2013/pacemaker-logging/ Pacemaker Logging]
* [http://blog.clusterlabs.org/blog/2013/pacemaker-logging/ Pacemaker Logging]
* Editing cib.xml offline is possible with: <span class="code">CIB_file=/path/to/real/cib.xml cibadmin ....</span> and sync to other nodes when done.


= Thanks =
= Thanks =

Latest revision as of 16:48, 19 November 2016

 AN!Wiki :: How To :: Anvil! Tutorial 3

Warning: This tutorial is incomplete, flawed and generally sucks at this time. Do not follow this and expect anything to work. In large part, it's a dumping ground for notes and little else. This warning will be removed when the tutorial is completed.

This is the third Anvil! tutorial built on Red Hat's Enterprise Linux 7. It marks the third generation of the Anvil! High-Availability Platform.

As with the previous tutorials, the end goal of this tutorial is an Anvil! platform for high-availability virtual servers. It's design attempts to remove all single points of failure from the system. Power and networking are made fully redundant in this version, along with minimizing the node failures which would lead to service interruption. This tutorial also covers the Striker dashboard and ScanCore monitoring and self-healing tools.

As it the previous tutorial, KVM will be the hypervisor used for facilitating virtual machines. The old cman and rgmanager tools are replaced in favour of pacemaker for resource management.

Before We Begin

This tutorial does not require prior Anvil! experience (or any clustering experience), but it does expect a certain familiarity with Linux and a low-intermediate understanding of networking. Where possible, steps are explained in detail and rationale is provided for why certain decisions are made.

For those with Anvil! experience;

Please be careful not to skip too much. There are some major and some subtle changes from previous tutorials.

OS Setup

This tutorial assumes a minimal install of either RHEL or CentOS version 7.

Post OS Install

 Note: With RHEL7, biosdevname tries to give network devices predictable names. It's very likely that your initial device names will differ from those in this tutorial.

If you are running RHEL

Before you can download any packages, you will need to register your nodes with Red Hat's subscription manager;

an-a04n01 
subscription-manager register --username $username --password $password --auto-attach
subscription-manager repos --enable=rhel-ha-for-rhel-7-server-rpms
subscription-manager repos --enable=rhel-7-server-optional-rpms

The system has been registered with ID: 9c578d87-bd80-4637-9f41-6076efb9e20e

Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed
an-a04n02 
subscription-manager register --username $username --password $password --auto-attach
subscription-manager repos --enable=rhel-ha-for-rhel-7-server-rpms
subscription-manager repos --enable=rhel-7-server-optional-rpms

The system has been registered with ID: a55c83e5-e4ec-4fcf-b7b7-b9455b3e07cf

Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

Adding LINBIT Repos

If you purchased full LINBIT support, you can add their repos in order to get DRBD 9 and associated tools.

First, download their registration tool.

an-a04n01 
cd /root
wget https://my.linbit.com/linbit-manage-node.py

--2016-11-19 10:22:21--  https://my.linbit.com/linbit-manage-node.py
Resolving my.linbit.com (my.linbit.com)... 212.69.166.235
Connecting to my.linbit.com (my.linbit.com)|212.69.166.235|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26797 (26K) [application/x-python-script]
Saving to: ‘linbit-manage-node.py’

100%[========================================================================================>] 26,797      --.-K/s   in 0.1s    

2016-11-19 10:22:21 (175 KB/s) - ‘linbit-manage-node.py’ saved [26797/26797]
an-a04n02 
cd /root
wget https://my.linbit.com/linbit-manage-node.py

--2016-11-19 10:26:52--  https://my.linbit.com/linbit-manage-node.py
Resolving my.linbit.com (my.linbit.com)... 212.69.166.235
Connecting to my.linbit.com (my.linbit.com)|212.69.166.235|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26797 (26K) [application/x-python-script]
Saving to: ‘linbit-manage-node.py’

100%[========================================================================================>] 26,797      --.-K/s   in 0.1s    

2016-11-19 10:26:53 (182 KB/s) - ‘linbit-manage-node.py’ saved [26797/26797]

Make it executable.

an-a04n01 
chmod 755 linbit-manage-node.py
ls -lah linbit-manage-node.py

-rwxr-xr-x. 1 root root 27K Oct 11 05:54 linbit-manage-node.py
an-a04n02 
chmod 755 linbit-manage-node.py

-rwxr-xr-x. 1 root root 27K Oct 11 05:54 linbit-manage-node.py
Note: If you get the error: 'ERR: Could not detect MAC addresses of your node', then the version of 'linbit-manage-node.py' does not yet recognise bridges or slaved interfaces in bonds. For now, you can download a modified version from Alteeve instead.

Now run the tool interactively.

an-a04n01 
/root/linbit-manage-node.py

linbit-manage-node.py (Version: 1.11)
Checking if version is up to date
[OK] Your version is up to date
Username:
an-a04n02 
/root/linbit-manage-node.py

linbit-manage-node.py (Version: 1.11)
Checking if version is up to date
[OK] Your version is up to date
Username:

Enter the user name and password given to you by LINBIT when you registered with them.

an-a04n01 
Username: xxxxxx
Credential (will not be echoed):

[OK] Login successful
The following contracts are available:
Will this node form a cluster with...

1) Contract: silver 2017-01-07 (ID: xxxx)

--> Please enter a number in range and press return:
an-a04n02 
Username: xxxxxx
Credential (will not be echoed):

[OK] Login successful
The following contracts are available:
Will this node form a cluster with...

1) Contract: silver 2017-01-07 (ID: xxxx)

--> Please enter a number in range and press return:

If you have multiple contracts, select the number to the left of the contract identification. Otherwise, select '1'.

an-a04n01 
--> Please enter a number in range and press return: 1

Writing registration data:
an-a04n02 
--> Please enter a number in range and press return: 1

Writing registration data:

Confirm that you want to write out the license file. Once you accept, you will be presented with a menu of which repositories you want to use from LINBIT. We're only going to enable the 'drbd-9.0' repo and leave the pacemaker repos disabled as we'll pull them from Red Hat.

an-a04n01 
--> Write to file (/var/lib/drbd-support/registration.json)? [y/N]

  Here are the repositories you can enable:

    1) pacemaker-1.1.15(Disabled)
    2) pacemaker-1.1.12(Disabled)
    3) pacemaker-1.1(Disabled)
    4) drbd-9.0(Disabled)
    5) drbd-8.4(Disabled)

  Enter the number of the repository you wish to enable/disable. Hit 0 when you are done.

  Enable/Disable: 4

  Here are the repositories you can enable:

    1) pacemaker-1.1.15(Disabled)
    2) pacemaker-1.1.12(Disabled)
    3) pacemaker-1.1(Disabled)
    4) drbd-9.0(Enabled)
    5) drbd-8.4(Disabled)

  Enter the number of the repository you wish to enable/disable. Hit 0 when you are done.

  Enable/Disable: 0
an-a04n02 
--> Write to file (/var/lib/drbd-support/registration.json)? [y/N]

  Here are the repositories you can enable:

    1) pacemaker-1.1.15(Disabled)
    2) pacemaker-1.1.12(Disabled)
    3) pacemaker-1.1(Disabled)
    4) drbd-9.0(Disabled)
    5) drbd-8.4(Disabled)

  Enter the number of the repository you wish to enable/disable. Hit 0 when you are done.

  Enable/Disable: 4

  Here are the repositories you can enable:

    1) pacemaker-1.1.15(Disabled)
    2) pacemaker-1.1.12(Disabled)
    3) pacemaker-1.1(Disabled)
    4) drbd-9.0(Enabled)
    5) drbd-8.4(Disabled)

  Enter the number of the repository you wish to enable/disable. Hit 0 when you are done.

  Enable/Disable: 0
Warning: The repository will include a node-specific hash string in the 'baseurl'. Keep this private!

Once you select '0' to exit that menu, a summary of the repo will be displayed and you will be asked if you want to save it or not.

an-a04n01 
Writing repository config:
Content:
[drbd-8.4]
name=LINBIT Packages for drbd-8.4 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/drbd-8.4/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1

[pacemaker-1.1]
name=LINBIT Packages for pacemaker-1.1 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1

[pacemaker-1.1.15]
name=LINBIT Packages for pacemaker-1.1.15 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1.15/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1

[pacemaker-1.1.12]
name=LINBIT Packages for pacemaker-1.1.12 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1.12/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1

[drbd-9.0]
name=LINBIT Packages for drbd-9.0 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/drbd-9.0/$basearch
enabled=1
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1

--> Write to file (/etc/yum.repos.d/linbit.repo)? [y/N] y
an-a04n02 
Writing repository config:
Content:
[drbd-8.4]
name=LINBIT Packages for drbd-8.4 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/drbd-8.4/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1

[pacemaker-1.1]
name=LINBIT Packages for pacemaker-1.1 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1

[pacemaker-1.1.15]
name=LINBIT Packages for pacemaker-1.1.15 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1.15/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1

[pacemaker-1.1.12]
name=LINBIT Packages for pacemaker-1.1.12 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/pacemaker-1.1.12/$basearch
enabled=0
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1

[drbd-9.0]
name=LINBIT Packages for drbd-9.0 - $basearch
baseurl=http://packages.linbit.com/xxxxxx/yum/rhel7/drbd-9.0/$basearch
enabled=1
gpgkey=https://packages.linbit.com/package-signing-pubkey.asc
gpgcheck=1

--> Write to file (/etc/yum.repos.d/linbit.repo)? [y/N] y

When you accept, it will download the yum plugins and then ask you if you want to save their PGP key.

an-a04n01 
[OK] Repository configuration written
Downloading LINBIT yum plugin
Downloading LINBIT yum plugin config
Final Notes:

--> Add linbit signing key to keyring now? [y/N] y

Now update your package information and install
LINBIT's kernel module and/or user space utilities
[OK] Congratulations! Your node was successfully configured.
an-a04n02 
[OK] Repository configuration written
Downloading LINBIT yum plugin
Downloading LINBIT yum plugin config
Final Notes:

--> Add linbit signing key to keyring now? [y/N] y

Now update your package information and install
LINBIT's kernel module and/or user space utilities
[OK] Congratulations! Your node was successfully configured.

Done!

Install

Not all of these are required, but most are used at one point or another in this tutorial.

Note: The fence-agents-virsh package is not available in RHEL 7 beta. Further, it's only needed if you're building your Anvil! using VMs.
an-a04n01 
yum install rsync pacemaker bridge-utils ntp corosync pcs wget gpm man vim screen mlocate syslinux bzip2 \
            openssh-clients fence-agents-all fence-agents-virsh policycoreutils-python drbd drbd-bash-completion \
            drbd-pacemaker drbd-udev drbd-utils drbdmanage
an-a04n02 
<same>


Making ssh faster when the net is down

By default, the nodes will try to resolve the host name of an incoming ssh connection. When the internet connection is down, DNS lookups have to time out, which can make login times quite slow. When something goes wrong, seconds count and waiting for up to a minute for an SSH password prompt can be maddening.

For this reason, we will make two changes to /etc/ssh/sshd_config that disable this login delay.

Please be aware that this can reduce security. If this is a concern, skip this step.

an-a04n01 
sed -i.anvil 's/#GSSAPIAuthentication no/GSSAPIAuthentication no/' /etc/ssh/sshd_config
sed -i 's/GSSAPIAuthentication yes/#GSSAPIAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
systemctl restart sshd.service
diff -u /etc/ssh/sshd_config.anvil /etc/ssh/sshd_config

--- /etc/ssh/sshd_config.anvil	2014-06-09 21:15:52.000000000 -0400
+++ /etc/ssh/sshd_config	2014-07-27 08:41:03.296760761 -0400
@@ -89,8 +89,8 @@
 #KerberosUseKuserok yes
 
 # GSSAPI options
-#GSSAPIAuthentication no
-GSSAPIAuthentication yes
+GSSAPIAuthentication no
+#GSSAPIAuthentication yes
 #GSSAPICleanupCredentials yes
 GSSAPICleanupCredentials yes
 #GSSAPIStrictAcceptorCheck yes
@@ -127,7 +127,7 @@
 #ClientAliveInterval 0
 #ClientAliveCountMax 3
 #ShowPatchLevel no
-#UseDNS yes
+UseDNS no
 #PidFile /var/run/sshd.pid
 #MaxStartups 10:30:100
 #PermitTunnel no
an-a04n02 
same

same

Subsequent logins when the net is down should be quick.

Configuring the network

If you want to make any other changes, like configuring the interface to have a static IP, do so now. Once you're done editing; 

nmcli connection reload
systemctl restart NetworkManager.service
ip addr show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:a7:9d:17 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.201/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fea7:9d17/64 scope link 
       valid_lft forever preferred_lft forever

The interface should now start on boot properly.

Setting the Hostname

Fedora 19 is very different from EL6.

Note: The '--pretty' line currently doesn't work as there is a bug (rhbz#895299) with single-quotes.
Note: The '--static' option is currently needed to prevent the '.' from being removed. See this bug (rhbz#896756).

Use a format that works for you. For the tutorial, node names are based on the following;

  • A two-letter prefix identifying the company/user (an, for "Alteeve's Niche!")
  • A sequential Anvil! ID number in the form of aXX (a01 for "Anvil! 01", a02 for Anvil! 02, etc)
  • A sequential node ID number in the form of nYY

In our case, this is my third Anvil! and we use the company prefix an, so these two nodes will be;

  • an-a04n01 - node 1
  • an-a04n02 - node 2
hostnamectl set-hostname an-a04n01.alteeve.ca --static
hostnamectl set-hostname --pretty "Alteeve's Niche! - Anvil! 03, Node 01"

If you want the new host name to take effect immediately, you can use the traditional hostname command: 

hostname an-a04n01.alteeve.ca

The "pretty" host name is stored in /etc/machine-info as the unquoted value for the PRETTY_HOSTNAME value. 

vim /etc/machine-info

PRETTY_HOSTNAME=Alteeves Niche! - Anvil! 03, Node 01

If you can't get the hostname command to work for some reason, you can reboot to have the system read the new values.

Network

 Note: (Note for myself) - Consider using 'primary_reselect=1.

We want static, named network devices. Follow this;

Then, use these configuration files;

Build the bridge; 

vim /etc/sysconfig/network-scripts/ifcfg-ifn_bridge1

# Internet-Facing Network - Bridge
DEVICE="ifn_bridge1"
TYPE="Bridge"
BOOTPROTO="none"
IPADDR="10.255.40.1"
NETMASK="255.255.0.0"
GATEWAY="10.255.255.254"
DNS1="8.8.8.8"
DNS2="8.8.4.4"
DEFROUTE="yes"

Now build the bonds; 

vim /etc/sysconfig/network-scripts/ifcfg-ifn_bond1

# Internet-Facing Network - Bond
DEVICE="ifn_bond1"
BRIDGE="ifn_bridge1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 primary=ifn_link1 updelay=120000 downdelay=0 fail_over_mac=none miimon=100 primary_reselect=better resend_igmp=5"

vim /etc/sysconfig/network-scripts/ifcfg-sn_bond1

# Storage Network - Bond
DEVICE="sn_bond1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 primary=sn_link1 updelay=120000 downdelay=0 fail_over_mac=none miimon=100 primary_reselect=better resend_igmp=5"
IPADDR="10.10.40.1"
NETMASK="255.255.0.0"

vim /etc/sysconfig/network-scripts/ifcfg-bcn_bond1

# Back-Channel Network - Bond
DEVICE="bcn_bond1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 primary=bcn_link1 updelay=120000 downdelay=0 fail_over_mac=none miimon=100 primary_reselect=better resend_igmp=5"
IPADDR="10.20.40.1"
NETMASK="255.255.0.0"

Now tell the interfaces to be slaves to their bonds;

Internet-Facing Network; 

vim /etc/sysconfig/network-scripts/ifcfg-ifn_link1

# Internet-Facing Network - Link 1
DEVICE="ifn_link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="ifn_bond1"

vim /etc/sysconfig/network-scripts/ifcfg-ifn_link2

# Internet-Facing Network - Link 2
DEVICE="ifn_link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="ifn_bond1"

Storage Network; 

vim /etc/sysconfig/network-scripts/ifcfg-sn_link1

# Storage Network - Link 1
DEVICE="sn_link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="sn_bond1"

vim /etc/sysconfig/network-scripts/ifcfg-sn_link2

# Storage Network - Link 2
DEVICE="sn_link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="sn_bond1"

Back-Channel Network 

vim /etc/sysconfig/network-scripts/ifcfg-bcn_link1

# Back-Channel Network - Link 1
DEVICE="bcn_link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="bcn_bond1"

vim /etc/sysconfig/network-scripts/ifcfg-bcn_link2

# Back-Channel Network - Link 2
DEVICE="bcn_link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="bcn_bond1"

Now restart the network, confirm that the bonds and bridge are up and you are ready to proceed.

Setup The hosts File

You can use DNS if you prefer. For now, lets use /etc/hosts for node name resolution. 

vim /etc/hosts

127.0.0.1	localhost localhost.localdomain localhost4 localhost4.localdomain4
::1		localhost localhost.localdomain localhost6 localhost6.localdomain6

# Anvil! 03, Node 01
10.255.40.1	an-a04n01.ifn
10.10.40.1	an-a04n01.sn
10.20.40.1	an-a04n01.bcn an-a04n01 an-a04n01.alteeve.ca
10.20.41.1	an-a04n01.ipmi

# Anvil! 03, Node 02
10.255.40.2	an-a04n02.ifn
10.10.40.2	an-a04n02.sn
10.20.40.2	an-a04n02.bcn an-a04n02 an-a04n02.alteeve.ca
10.20.41.2	an-a04n02.ipmi

# Foundation Pack
### Foundation Pack
# Network Switches
10.20.1.1	an-switch01 an-switch01.alteeve.ca
10.20.1.2	an-switch02 an-switch02.alteeve.ca	# Only accessible when out of the stack
 
# Switched PDUs
10.20.2.1	an-pdu01 an-pdu01.alteeve.ca
10.20.2.2	an-pdu02 an-pdu02.alteeve.ca
 
# Network-monitored UPSes
10.20.3.1	an-ups01 an-ups01.alteeve.ca
10.20.3.2	an-ups02 an-ups02.alteeve.ca
 
### Monitor Packs
10.20.4.1	an-striker01 an-striker01.alteeve.ca
10.255.4.1	an-striker01.ifn
10.20.4.2	an-striker02 an-striker02.alteeve.ca
10.255.4.2	an-striker02.ifn

Setup SSH

Same as before.

Populating And Pushing ~/.ssh/known_hosts

an-a04n01 
ssh-keygen -t rsa -N "" -b 8191 -f ~/.ssh/id_rsa

Generating public/private rsa key pair.

Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
be:17:cc:23:8e:b1:b4:76:a1:e4:2a:91:cb:cd:d8:3a root@an-a04n01.alteeve.ca
The key's randomart image is:
+--[ RSA 8191]----+
|                 |
|                 |
|                 |
|                 |
|   .    So       |
|  o   +.o =      |
| . B + B.o o     |
|  E + B o..      |
|  .+.o ...       |
+-----------------+
an-a04n01 
ssh-keygen -t rsa -N "" -b 8191 -f ~/.ssh/id_rsa

Generating public/private rsa key pair.
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
71:b1:9d:31:9f:7a:c9:10:74:e0:4c:69:53:8f:e4:70 root@an-a04n02.alteeve.ca
The key's randomart image is:
+--[ RSA 8191]----+
|          ..O+E  |
|           B+% + |
|        . o.*.= .|
|         o   + . |
|        S   . +  |
|             .   |
|                 |
|                 |
|                 |
+-----------------+

Setup autorized_keys:

an-a04n01 
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh root@an-a04n02 "cat /root/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys 
rsync -av ~/.ssh/authorized_keys root@an-a04n02:/root/.ssh/
ssh-keyscan an-a04n01.alteeve.ca >> ~/.ssh/known_hosts
ssh-keyscan an-a04n01 >> ~/.ssh/known_hosts
ssh-keyscan an-a04n01.bcn >> ~/.ssh/known_hosts
ssh-keyscan an-a04n01.sn >> ~/.ssh/known_hosts
ssh-keyscan an-a04n01.ifn >> ~/.ssh/known_hosts
ssh-keyscan an-a04n02.alteeve.ca >> ~/.ssh/known_hosts
ssh-keyscan an-a04n02 >> ~/.ssh/known_hosts
ssh-keyscan an-a04n02.bcn >> ~/.ssh/known_hosts
ssh-keyscan an-a04n02.sn >> ~/.ssh/known_hosts
ssh-keyscan an-a04n02.ifn >> ~/.ssh/known_hosts
rsync -av ~/.ssh/known_hosts root@an-a04n02:/root/.ssh/
rsync -av /etc/hosts root@an-a04n02:/etc/
an-a04n01

Keeping Time in Sync

It's not as critical as it used to be to keep the clocks on the nodes in sync, but it's still a good idea. 

ln -sf /usr/share/zoneinfo/America/Toronto /etc/localtime
systemctl start ntpd.service
systemctl enable ntpd.service

Configuring IPMI

F19 specifics based on the IPMI tutorial. 

yum -y install ipmitools OpenIPMI
systemctl start ipmi.service
systemctl enable ipmi.service

ln -s '/usr/lib/systemd/system/ipmi.service' '/etc/systemd/system/multi-user.target.wants/ipmi.service'

Our servers use lan channel 2, yours might be 1 or something else. Experiment. 

ipmitool lan print 2

Set in Progress         : Set Complete
Auth Type Support       : NONE MD5 PASSWORD 
Auth Type Enable        : Callback : NONE MD5 PASSWORD 
                        : User     : NONE MD5 PASSWORD 
                        : Operator : NONE MD5 PASSWORD 
                        : Admin    : NONE MD5 PASSWORD 
                        : OEM      : NONE MD5 PASSWORD 
IP Address Source       : BIOS Assigned Address
IP Address              : 10.20.41.1
Subnet Mask             : 255.255.0.0
MAC Address             : 00:19:99:9a:d8:e8
SNMP Community String   : public
IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP      : 10.20.255.254
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3,6,7,8,17
Cipher Suite Priv Max   : OOOOOOOOXXXXXXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM

I need to set the IPs to 10.20.41.1/16 and 10.20.41.2/16 for nodes 1 and 2, respectively. I also want to set the password to secret for the admin user.

Node 01 IP; 

ipmitool lan set 2 ipsrc static
ipmitool lan set 2 ipaddr 10.20.41.1
ipmitool lan set 2 netmask 255.255.0.0
ipmitool lan set 2 defgw ipaddr 10.20.255.254
ipmitool lan print 2

Set in Progress         : Set Complete
Auth Type Support       : NONE MD5 PASSWORD 
Auth Type Enable        : Callback : NONE MD5 PASSWORD 
                        : User     : NONE MD5 PASSWORD 
                        : Operator : NONE MD5 PASSWORD 
                        : Admin    : NONE MD5 PASSWORD 
                        : OEM      : NONE MD5 PASSWORD 
IP Address Source       : Static Address
IP Address              : 10.20.41.1
Subnet Mask             : 255.255.0.0
MAC Address             : 00:19:99:9a:d8:e8
SNMP Community String   : public
IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP      : 10.20.255.254
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3,6,7,8,17
Cipher Suite Priv Max   : OOOOOOOOXXXXXXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM

Node 01 IP; 

ipmitool lan set 2 ipsrc static
ipmitool lan set 2 ipaddr 10.20.41.2
ipmitool lan set 2 netmask 255.255.0.0
ipmitool lan set 2 defgw ipaddr 10.20.255.254
ipmitool lan print 2

Set in Progress         : Set Complete
Auth Type Support       : NONE MD5 PASSWORD 
Auth Type Enable        : Callback : NONE MD5 PASSWORD 
                        : User     : NONE MD5 PASSWORD 
                        : Operator : NONE MD5 PASSWORD 
                        : Admin    : NONE MD5 PASSWORD 
                        : OEM      : NONE MD5 PASSWORD 
IP Address Source       : Static Address
IP Address              : 10.20.41.2
Subnet Mask             : 255.255.0.0
MAC Address             : 00:19:99:9a:b1:78
SNMP Community String   : public
IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP      : 10.20.255.254
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3,6,7,8,17
Cipher Suite Priv Max   : OOOOOOOOXXXXXXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM

Set the password. 

ipmitool user list 2

ID  Name	     Callin  Link Auth	IPMI Msg   Channel Priv Limit
1                    true    true       true       Unknown (0x00)
2   admin            true    true       true       OEM
Get User Access command failed (channel 2, user 3): Unknown (0x32)

(ignore the error, it's harmless... *BOOM*)

We want to set admin's password, so we do:

Note: The 2 below is the ID number, not the LAN channel. 
ipmitool user set password 2 secret

Done!

Configuring the Anvil!

Now we're getting down to business!

For this section, we will be working on an-a04n01 and using ssh to perform tasks on an-a04n02.

Note: TODO: explain what this is and how it works.

Enable the pcs Daemon

 Note: Most of this section comes more or less verbatim from the main Clusters from Scratch tutorial.

We will use pcs, the Pacemaker Configuration System, to configure our Anvil!.

Note that pcsd uses TCP port 2224. 

systemctl start pcsd.service
systemctl enable pcsd.service

ln -s '/usr/lib/systemd/system/pcsd.service' '/etc/systemd/system/multi-user.target.wants/pcsd.service'

Now we need to set a password for the hacluster user. This is the account used by pcs on one node to talk to the pcs daemon on the other node. For this tutorial, we will use the password secret. You will want to use a stronger password, of course. 

echo "super secret password" | passwd --stdin hacluster

Changing password for user hacluster.
passwd: all authentication tokens updated successfully.

Open up the firewall: 

firewall-cmd --permanent --add-service=high-availability
firewall-cmd --reload

Initializing the Cluster

One of the biggest reasons we're using the pcs tool, over something like crm, is that it has been written to simplify the setup of clusters on Red Hat style operating systems. It will configure corosync automatically.

First, we need to know what hostname we will need to use for pcs.

Node 01: 

hostname
an-a04n01.alteeve.ca

Node 02: 

hostname
an-a04n02.alteeve.ca

Next, authenticate against the cluster nodes.

Both nodes: 

pcs cluster auth an-a04n01.alteeve.ca an-a04n02.alteeve.ca -u hacluster

This will ask you for the user name and password. The default user name is hacluster and we set the password to secret. 

Password: 
an-a04n01.alteeve.ca: 6e9f7e98-dfb7-4305-b8e0-d84bf4f93ce3
an-a04n01.alteeve.ca: Authorized
an-a04n02.alteeve.ca: ffee6a85-ddac-4d03-9b97-f136d532b478
an-a04n02.alteeve.ca: Authorized

Do this on one node only:

Now to initialize the cluster's communication and membership layer. 

pcs cluster setup --name an-anvil-03 an-a04n01.alteeve.ca an-a04n02.alteeve.ca
an-a04n01.alteeve.ca: Succeeded
an-a04n02.alteeve.ca: Succeeded

This will create the corosync configuration file /etc/corosync/corosync.conf; 

cat /etc/corosync/corosync.conf
totem {
version: 2
secauth: off
cluster_name: an-anvil-03
transport: udpu
}

nodelist {
  node {
        ring0_addr: an-a04n01.alteeve.ca
        nodeid: 1
       }
  node {
        ring0_addr: an-a04n02.alteeve.ca
        nodeid: 2
       }
}

quorum {
provider: corosync_votequorum
two_node: 1
}

logging {
to_syslog: yes
}

Start the Cluster For the First Time

This starts the cluster communication and membership layer for the first time.

On one node only; 

pcs cluster start --all
an-a04n01.alteeve.ca: Starting Cluster...
an-a04n02.alteeve.ca: Starting Cluster...

After a few moments, you should be able to check the status; 

pcs status
Cluster name: an-anvil-04
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Mon Jun 24 23:28:29 2013
Last change: Mon Jun 24 23:28:10 2013 via crmd on an-a04n01.alteeve.ca
Current DC: NONE
2 Nodes configured, unknown expected votes
0 Resources configured.


Node an-a04n01.alteeve.ca (1): UNCLEAN (offline)
Node an-a04n02.alteeve.ca (2): UNCLEAN (offline)

Full list of resources:

The other node should show almost the identical output.

Disabling Quorum

 Note: Show the math.

With quorum enabled, a two node cluster will lose quorum once either node fails. So we have to disable quorum.

By default, pacemaker uses quorum. You don't see this initially though; 

pcs property
Cluster Properties:
 dc-version: 1.1.9-0.1318.a7966fb.git.fc18-a7966fb
 cluster-infrastructure: corosync

To disable it, we set no-quorum-policy=ignore. 

pcs property set no-quorum-policy=ignore
pcs property
Cluster Properties:
 dc-version: 1.1.9-0.1318.a7966fb.git.fc18-a7966fb
 cluster-infrastructure: corosync
 no-quorum-policy: ignore

Enabling and Configuring Fencing

We will use IPMI and PDU based fence devices for redundancy.

You can see the list of available fence agents here. You will need to find the one for your hardware fence devices. 

pcs stonith list
fence_alom - Fence agent for Sun ALOM
fence_apc - Fence agent for APC over telnet/ssh
fence_apc_snmp - Fence agent for APC over SNMP
fence_baytech - I/O Fencing agent for Baytech RPC switches in combination with a Cyclades Terminal
                Server
fence_bladecenter - Fence agent for IBM BladeCenter
fence_brocade - Fence agent for Brocade over telnet
fence_bullpap - I/O Fencing agent for Bull FAME architecture controlled by a PAP management console.
fence_cisco_mds - Fence agent for Cisco MDS
fence_cisco_ucs - Fence agent for Cisco UCS
fence_cpint - I/O Fencing agent for GFS on s390 and zSeries VM clusters
fence_drac - fencing agent for Dell Remote Access Card
fence_drac5 - Fence agent for Dell DRAC CMC/5
fence_eaton_snmp - Fence agent for Eaton over SNMP
fence_egenera - I/O Fencing agent for the Egenera BladeFrame
fence_eps - Fence agent for ePowerSwitch
fence_hpblade - Fence agent for HP BladeSystem
fence_ibmblade - Fence agent for IBM BladeCenter over SNMP
fence_idrac - Fence agent for IPMI over LAN
fence_ifmib - Fence agent for IF MIB
fence_ilo - Fence agent for HP iLO
fence_ilo2 - Fence agent for HP iLO
fence_ilo3 - Fence agent for IPMI over LAN
fence_ilo_mp - Fence agent for HP iLO MP
fence_imm - Fence agent for IPMI over LAN
fence_intelmodular - Fence agent for Intel Modular
fence_ipdu - Fence agent for iPDU over SNMP
fence_ipmilan - Fence agent for IPMI over LAN
fence_kdump - Fence agent for use with kdump
fence_ldom - Fence agent for Sun LDOM
fence_lpar - Fence agent for IBM LPAR
fence_mcdata - I/O Fencing agent for McData FC switches
fence_rackswitch - fence_rackswitch - I/O Fencing agent for RackSaver RackSwitch
fence_rhevm - Fence agent for RHEV-M REST API
fence_rsa - Fence agent for IBM RSA
fence_rsb - I/O Fencing agent for Fujitsu-Siemens RSB
fence_sanbox2 - Fence agent for QLogic SANBox2 FC switches
fence_scsi - fence agent for SCSI-3 persistent reservations
fence_virsh - Fence agent for virsh
fence_vixel - I/O Fencing agent for Vixel FC switches
fence_vmware - Fence agent for VMWare
fence_vmware_soap - Fence agent for VMWare over SOAP API
fence_wti - Fence agent for WTI
fence_xcat - I/O Fencing agent for xcat environments
fence_xenapi - XenAPI based fencing for the Citrix XenServer virtual machines.
fence_zvm - I/O Fencing agent for GFS on s390 and zSeries VM clusters

We will use fence_ipmilan and fence_apc_snmp.

Configuring IPMI Fencing

Every fence agent has a possibly unique subset of options that can be used. You can see a brief description of these options with the pcs stonith describe fence_X command. Let's look at the options available for fence_ipmilan. 

pcs stonith describe fence_ipmilan
Stonith options for: fence_ipmilan
  auth: IPMI Lan Auth type (md5, password, or none)
  ipaddr: IPMI Lan IP to talk to
  passwd: Password (if required) to control power on IPMI device
  passwd_script: Script to retrieve password (if required)
  lanplus: Use Lanplus
  login: Username/Login (if required) to control power on IPMI device
  action: Operation to perform. Valid operations: on, off, reboot, status, list, diag, monitor or metadata
  timeout: Timeout (sec) for IPMI operation
  cipher: Ciphersuite to use (same as ipmitool -C parameter)
  method: Method to fence (onoff or cycle)
  power_wait: Wait X seconds after on/off operation
  delay: Wait X seconds before fencing is started
  privlvl: Privilege level on IPMI device
  verbose: Verbose mode

One of the nice things about pcs is that it allows us to create a test file to prepare all our changes in. Then, when we're happy with the changes, merge them into the running cluster. So let's make a copy called stonith_cfg 

pcs cluster cib stonith_cfg

Now add IPMI fencing. 

#                  unique name    fence agent   target node                           device addr             options
pcs stonith create fence_n01_ipmi fence_ipmilan pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-a04n01.ipmi" action="reboot" login="admin" passwd="secret" delay=15 op monitor interval=60s
pcs stonith create fence_n02_ipmi fence_ipmilan pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-a04n02.ipmi" action="reboot" login="admin" passwd="secret" op monitor interval=60s

Note that fence_n01_ipmi has a delay=15 set but fence_n02_ipmi does not. If the network connection breaks between the two nodes, they will both try to fence each other at the same time. If acpid is running, the slower node will not die right away. It will continue to run for up to four more seconds, ample time for it to also initiate a fence against the faster node. The end result is that both nodes get fenced. The ten-second delay protects against this by causing an-a04n02 to pause for 10 seconds before initiating a fence against an-a04n01. If both nodes are alive, an-a04n02 will power off before the 10 seconds pass, so it will never fence an-a04n01. However, if an-a04n01 really is dead, after the ten seconds have elapsed, fencing will proceed as normal.

Note: At the time of writing, pcmk_reboot_action is needed to override pacemaker's global fence action and pcmk_reboot_action is not recognized by pcs. Both of these issues will be resolved shortly; Pacemaker will honour action="..." in v1.1.10 and pcs will recognize pcmk_* special attributes "real soon now". Until then, the --force switch is needed.

Next, add the PDU fencing. This requires distinct "off" and "on" actions for each outlet on each PDU. With two nodes, each with two PSUs, this translates to eight commands. The "off" commands will be monitored to alert us if the PDU fails for some reason. There is no reason to monitor the "on" actions (it would be redundant). Note also that we don't bother using a "delay". The IPMI fence method will go first, before the PDU actions, so the PDU is already delayed. 

# Node 1 - off
pcs stonith create fence_n01_pdu1_off fence_apc_snmp pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-pdu01" action="off" port="1" op monitor interval="60s"
pcs stonith create fence_n01_pdu2_off fence_apc_snmp pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-pdu02" action="off" port="1" power_wait="5" op monitor interval="60s"

# Node 1 - on
pcs stonith create fence_n01_pdu1_on fence_apc_snmp pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-pdu01" action="on" port="1"
pcs stonith create fence_n01_pdu2_on fence_apc_snmp pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-pdu02" action="on" port="1"

# Node 2 - off
pcs stonith create fence_n02_pdu1_off fence_apc_snmp pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-pdu01" action="off" port="2" op monitor interval="60s"
pcs stonith create fence_n02_pdu2_off fence_apc_snmp pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-pdu02" action="off" port="2" power_wait="5" op monitor interval="60s"

# Node 2 - on
pcs stonith create fence_n02_pdu1_on fence_apc_snmp pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-pdu01" action="on" port="2"
pcs stonith create fence_n02_pdu2_on fence_apc_snmp pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-pdu02" action="on" port="2"

We can check the new configuration now; 

pcs status
Cluster name: an-anvil-04
Last updated: Tue Jul  2 16:41:55 2013
Last change: Tue Jul  2 16:41:44 2013 via cibadmin on an-a04n01.alteeve.ca
Stack: corosync
Current DC: an-a04n01.alteeve.ca (1) - partition with quorum
Version: 1.1.9-3.fc19-781a388
2 Nodes configured, unknown expected votes
10 Resources configured.


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca 
 fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca 
 fence_n01_pdu1_off	(stonith:fence_apc_snmp):	Started an-a04n01.alteeve.ca 
 fence_n01_pdu2_off	(stonith:fence_apc_snmp):	Started an-a04n02.alteeve.ca 
 fence_n02_pdu1_off	(stonith:fence_apc_snmp):	Started an-a04n01.alteeve.ca 
 fence_n02_pdu2_off	(stonith:fence_apc_snmp):	Started an-a04n02.alteeve.ca 
 fence_n01_pdu1_on	(stonith:fence_apc_snmp):	Started an-a04n01.alteeve.ca 
 fence_n01_pdu2_on	(stonith:fence_apc_snmp):	Started an-a04n02.alteeve.ca 
 fence_n02_pdu1_on	(stonith:fence_apc_snmp):	Started an-a04n01.alteeve.ca 
 fence_n02_pdu2_on	(stonith:fence_apc_snmp):	Started an-a04n02.alteeve.ca

Before we proceed, we need to tell pacemaker to use fencing; 

pcs property set stonith-enabled=true
pcs property
Cluster Properties:
Cluster Properties:
 cluster-infrastructure: corosync
 dc-version: 1.1.9-3.fc19-781a388
 no-quorum-policy: ignore
 stonith-enabled: true

Excellent!

Configuring Fence Levels

The goal of fence levels is to tell pacemaker that there are "fence methods" to try and to impose an order on those methods. Each method composes one or more fence primitives and, when 2 or more primitives are tied together, that all primitives must succeed for the overall method to succeed.

So in our case; the order we want is;

  • IPMI -> PDUs

The reason is that when IPMI fencing succeeds, we can be very certain the node is truly fenced. When PDU fencing succeeds, it only confirms that the power outlets were cycled. If someone moved a node's power cables to another outlet, we'll get a false positive. On that topic, tie-down the node's PSU cables to the PDU's cable tray when possible, clearly label the power cables and wrap the fingers of anyone who might move them around.

The PDU fencing needs to be implemented using four steps;

  • PDU 1, outlet X -> off
  • PDU 2, outlet X -> off
    • The power_wait="5" setting for the fence_n0X_pdu2_off primitives will cause a 5 second delay here, giving ample time to ensure the nodes lose power
  • PDU 1, outlet X -> on
  • PDU 2, outlet X -> on

This is to ensure that both outlets are off at the same time, ensuring that the node loses power. This works because fencing_topology acts serially.

Putting all this together, we issue this command; 

pcs stonith level add 1 an-a04n01.alteeve.ca fence_n01_ipmi
pcs stonith level add 1 an-a04n02.alteeve.ca fence_n02_ipmi

The 1 tells pacemaker that this is our highest priority fence method. We can see that this was set using pcs; 

pcs stonith level
 Node: an-a04n01.alteeve.ca
  Level 1 - fence_n01_ipmi
 Node: an-a04n02.alteeve.ca
  Level 1 - fence_n02_ipmi

Now we'll tell pacemaker to use the PDUs as the second fence method. Here we tie together the two off calls and the two on calls into a single method. 

pcs stonith level add 2 an-a04n01.alteeve.ca fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on
pcs stonith level add 2 an-a04n02.alteeve.ca fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on

Check again and we'll see that the new methods were added. 

pcs stonith level
 Node: an-a04n01.alteeve.ca
  Level 1 - fence_n01_ipmi
  Level 2 - fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on
 Node: an-a04n02.alteeve.ca
  Level 1 - fence_n02_ipmi
  Level 2 - fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on

For those of us who are XML fans, this is what the cib looks like now: 

cat /var/lib/pacemaker/cib/cib.xml
<cib epoch="18" num_updates="0" admin_epoch="0" validate-with="pacemaker-1.2" cib-last-written="Thu Jul 18 13:15:53 2013" update-origin="an-a04n01.alteeve.ca" update-client="cibadmin" crm_feature_set="3.0.7" have-quorum="1" dc-uuid="1">
  <configuration>
    <crm_config>
      <cluster_property_set id="cib-bootstrap-options">
        <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.9-dde1c52"/>
        <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
        <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/>
      </cluster_property_set>
    </crm_config>
    <nodes>
      <node id="1" uname="an-a04n01.alteeve.ca"/>
      <node id="2" uname="an-a04n02.alteeve.ca"/>
    </nodes>
    <resources>
      <primitive class="stonith" id="fence_n01_ipmi" type="fence_ipmilan">
        <instance_attributes id="fence_n01_ipmi-instance_attributes">
          <nvpair id="fence_n01_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n01.alteeve.ca"/>
          <nvpair id="fence_n01_ipmi-instance_attributes-ipaddr" name="ipaddr" value="an-a04n01.ipmi"/>
          <nvpair id="fence_n01_ipmi-instance_attributes-action" name="action" value="reboot"/>
          <nvpair id="fence_n01_ipmi-instance_attributes-login" name="login" value="admin"/>
          <nvpair id="fence_n01_ipmi-instance_attributes-passwd" name="passwd" value="secret"/>
          <nvpair id="fence_n01_ipmi-instance_attributes-delay" name="delay" value="15"/>
        </instance_attributes>
        <operations>
          <op id="fence_n01_ipmi-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n02_ipmi" type="fence_ipmilan">
        <instance_attributes id="fence_n02_ipmi-instance_attributes">
          <nvpair id="fence_n02_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n02.alteeve.ca"/>
          <nvpair id="fence_n02_ipmi-instance_attributes-ipaddr" name="ipaddr" value="an-a04n02.ipmi"/>
          <nvpair id="fence_n02_ipmi-instance_attributes-action" name="action" value="reboot"/>
          <nvpair id="fence_n02_ipmi-instance_attributes-login" name="login" value="admin"/>
          <nvpair id="fence_n02_ipmi-instance_attributes-passwd" name="passwd" value="secret"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_ipmi-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n01_pdu1_off" type="fence_apc_snmp">
        <instance_attributes id="fence_n01_pdu1_off-instance_attributes">
          <nvpair id="fence_n01_pdu1_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n01.alteeve.ca"/>
          <nvpair id="fence_n01_pdu1_off-instance_attributes-ipaddr" name="ipaddr" value="an-pdu01"/>
          <nvpair id="fence_n01_pdu1_off-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_n01_pdu1_off-instance_attributes-port" name="port" value="1"/>
        </instance_attributes>
        <operations>
          <op id="fence_n01_pdu1_off-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n01_pdu2_off" type="fence_apc_snmp">
        <instance_attributes id="fence_n01_pdu2_off-instance_attributes">
          <nvpair id="fence_n01_pdu2_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n01.alteeve.ca"/>
          <nvpair id="fence_n01_pdu2_off-instance_attributes-ipaddr" name="ipaddr" value="an-pdu02"/>
          <nvpair id="fence_n01_pdu2_off-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_n01_pdu2_off-instance_attributes-port" name="port" value="1"/>
          <nvpair id="fence_n01_pdu2_off-instance_attributes-power_wait" name="power_wait" value="5"/>
        </instance_attributes>
        <operations>
          <op id="fence_n01_pdu2_off-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n01_pdu1_on" type="fence_apc_snmp">
        <instance_attributes id="fence_n01_pdu1_on-instance_attributes">
          <nvpair id="fence_n01_pdu1_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n01.alteeve.ca"/>
          <nvpair id="fence_n01_pdu1_on-instance_attributes-ipaddr" name="ipaddr" value="an-pdu01"/>
          <nvpair id="fence_n01_pdu1_on-instance_attributes-action" name="action" value="on"/>
          <nvpair id="fence_n01_pdu1_on-instance_attributes-port" name="port" value="1"/>
        </instance_attributes>
        <operations>
          <op id="fence_n01_pdu1_on-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n01_pdu2_on" type="fence_apc_snmp">
        <instance_attributes id="fence_n01_pdu2_on-instance_attributes">
          <nvpair id="fence_n01_pdu2_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n01.alteeve.ca"/>
          <nvpair id="fence_n01_pdu2_on-instance_attributes-ipaddr" name="ipaddr" value="an-pdu02"/>
          <nvpair id="fence_n01_pdu2_on-instance_attributes-action" name="action" value="on"/>
          <nvpair id="fence_n01_pdu2_on-instance_attributes-port" name="port" value="1"/>
        </instance_attributes>
        <operations>
          <op id="fence_n01_pdu2_on-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n02_pdu1_off" type="fence_apc_snmp">
        <instance_attributes id="fence_n02_pdu1_off-instance_attributes">
          <nvpair id="fence_n02_pdu1_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n02.alteeve.ca"/>
          <nvpair id="fence_n02_pdu1_off-instance_attributes-ipaddr" name="ipaddr" value="an-pdu01"/>
          <nvpair id="fence_n02_pdu1_off-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_n02_pdu1_off-instance_attributes-port" name="port" value="2"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_pdu1_off-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n02_pdu2_off" type="fence_apc_snmp">
        <instance_attributes id="fence_n02_pdu2_off-instance_attributes">
          <nvpair id="fence_n02_pdu2_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n02.alteeve.ca"/>
          <nvpair id="fence_n02_pdu2_off-instance_attributes-ipaddr" name="ipaddr" value="an-pdu02"/>
          <nvpair id="fence_n02_pdu2_off-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_n02_pdu2_off-instance_attributes-port" name="port" value="2"/>
          <nvpair id="fence_n02_pdu2_off-instance_attributes-power_wait" name="power_wait" value="5"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_pdu2_off-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n02_pdu1_on" type="fence_apc_snmp">
        <instance_attributes id="fence_n02_pdu1_on-instance_attributes">
          <nvpair id="fence_n02_pdu1_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n02.alteeve.ca"/>
          <nvpair id="fence_n02_pdu1_on-instance_attributes-ipaddr" name="ipaddr" value="an-pdu01"/>
          <nvpair id="fence_n02_pdu1_on-instance_attributes-action" name="action" value="on"/>
          <nvpair id="fence_n02_pdu1_on-instance_attributes-port" name="port" value="2"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_pdu1_on-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="stonith" id="fence_n02_pdu2_on" type="fence_apc_snmp">
        <instance_attributes id="fence_n02_pdu2_on-instance_attributes">
          <nvpair id="fence_n02_pdu2_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a04n02.alteeve.ca"/>
          <nvpair id="fence_n02_pdu2_on-instance_attributes-ipaddr" name="ipaddr" value="an-pdu02"/>
          <nvpair id="fence_n02_pdu2_on-instance_attributes-action" name="action" value="on"/>
          <nvpair id="fence_n02_pdu2_on-instance_attributes-port" name="port" value="2"/>
        </instance_attributes>
        <operations>
          <op id="fence_n02_pdu2_on-monitor-interval-60s" interval="60s" name="monitor"/>
        </operations>
      </primitive>
    </resources>
    <constraints/>
    <fencing-topology>
      <fencing-level devices="fence_n01_ipmi" id="fl-an-a04n01.alteeve.ca-1" index="1" target="an-a04n01.alteeve.ca"/>
      <fencing-level devices="fence_n02_ipmi" id="fl-an-a04n02.alteeve.ca-1" index="1" target="an-a04n02.alteeve.ca"/>
      <fencing-level devices="fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on" id="fl-an-a04n01.alteeve.ca-2" index="2" target="an-a04n01.alteeve.ca"/>
      <fencing-level devices="fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on" id="fl-an-a04n02.alteeve.ca-2" index="2" target="an-a04n02.alteeve.ca"/>
    </fencing-topology>
  </configuration>
</cib>

Fencing using fence_virsh

 Note: To write this section, I used two virtual machines called pcmk1 and pcmk2.

If you are trying to learn fencing using KVM or Xen virtual machines, you can use the fence_virsh. You can also use fence_virtd, which is actually recommended by many, but I have found it to be rather unreliable.

To use fence_virsh, first install it. 

yum -y install fence-agents-virsh
<lots of yum output>

Now test it from the command line. To do this, we need to know a few things;

  • The VM host is at IP 192.168.122.1
  • The username and password (-l and -p respectively) are the credentials used to log into VM host over SSH.
    • If you don't want your password to be shown, create a little shell script that simply prints your password and then use -S /path/to/script instead of -p "secret".
  • The name of the target VM, as shown by virsh list --all on the host, is the node (-n) value. For me, the nodes are called an-a04n01 and an-a04n02.

Create the Password Script

In my case, the host is called 'lemass', so I want to create a password script called '/root/lemass.pw'. The name of the script is entirely up to you.

an-a04n01 
vim /root/lemass.pw
echo "my secret password"
chmod 755 /root/lemass.pw
/root/lemass.pw
my secret password
rsync -av /root/lemass.pw root@an-a04n02:/root/
sending incremental file list
lemass.pw

sent 102 bytes  received 31 bytes  266.00 bytes/sec
total size is 25  speedup is 0.19
an-a04n02 
/root/lemass.pw
my secret password

Done.

Test fence_virsh Status from the Command Line

an-a04n01 
fence_virsh -a 192.168.122.1 -l root -S /root/lemass.pw -n an-a04n02 -o status
Status: ON
an-a04n02 
fence_virsh -a 192.168.122.1 -l root -S /root/lemass.pw -n an-a04n01 -o status
Status: ON

Excellent! Now to configure it in pacemaker;

an-a04n01 
pcs stonith create fence_n01_virsh fence_virsh pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="192.168.122.1" action="reboot" login="root" passwd_script="/root/lemass.pw" port="an-a04n01" delay=15 op monitor interval=60s
pcs stonith create fence_n02_virsh fence_virsh pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="192.168.122.1" action="reboot" login="root" passwd_script="/root/lemass.pw" port="an-a04n02" op monitor interval=60s
pcs cluster status
Cluster Status:
 Last updated: Sun Jan 26 15:45:31 2014
 Last change: Sun Jan 26 15:06:14 2014 via crmd on an-a04n01.alteeve.ca
 Stack: corosync
 Current DC: an-a04n02.alteeve.ca (2) - partition with quorum
 Version: 1.1.10-19.el7-368c726
 2 Nodes configured
 2 Resources configured

PCSD Status:
an-a04n01.alteeve.ca: 
  an-a04n01.alteeve.ca: Online
an-a04n02.alteeve.ca: 
  an-a04n02.alteeve.ca: Online

Test Fencing

ToDo: Kill each node with echo c > /proc/sysrq-trigger and make sure the other node fences it.

Shared Storage

DRBD

We will use DRBD 8.4.

Install DRBD 8.4.4 from AN!

 Warning: this doesn't work.

ToDo: Make a proper repo

an-a04n01 
rpm -Uvh https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-8.4.4-4.el7.x86_64.rpm \
         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-bash-completion-8.4.4-4.el7.x86_64.rpm \
         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-pacemaker-8.4.4-4.el7.x86_64.rpm \
         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-udev-8.4.4-4.el7.x86_64.rpm \
         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-utils-8.4.4-4.el7.x86_64.rpm \
         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-heartbeat-8.4.4-4.el7.x86_64.rpm \
         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-xen-8.4.4-4.el7.x86_64.rpm
an-a04n02


Install DRBD 8.4.4 From Source

At this time, no EPEL repo exists for RHEL7, and the Fedora RPMs don't work, so we will install DRBD 8.4.4 from source.

Install dependencies: 

yum -y install gcc flex rpm-build wget kernel-devel
wget -c http://oss.linbit.com/drbd/8.4/drbd-8.4.4.tar.gz
tar -xvzf drbd-8.4.4.tar.gz 
cd drbd-8.4.4
./configure \
  --prefix=/usr \
  --localstatedir=/var \
  --sysconfdir=/etc \
  --with-km \
  --with-udev \
  --with-pacemaker \
  --with-bashcompletion \
  --with-utils \
  --without-xen \
  --without-rgmanager \
  --without-heartbeat
make
make install

Don't let DRBD start on boot (pacemaker will handle it for us). 

systemctl disable drbd.service
drbd.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig drbd off

Done.

Optional; Make RPMs

 Warning: I've not been able to get the RPMs genreated here to install yet. I'd recommend skipping this, unless you want to help sort out the problems. :)

After ./configure above, you can make RPMs instead of installing directly.

Dependencies: 

yum install rpmdevtools redhat-rpm-config kernel-devel
<install text>

Setup RPM dev tree: 

cd ~
rpmdev-setuptree
ls -lah ~/rpmbuild/
wget -c http://oss.linbit.com/drbd/8.4/drbd-8.4.4.tar.gz
tar -xvzf drbd-8.4.4.tar.gz
cd drbd-8.4.4
./configure \
  --prefix=/usr \
  --localstatedir=/var \
  --sysconfdir=/etc \
  --with-km \
  --with-udev \
  --with-pacemaker \
  --with-bashcompletion \
  --with-utils \
  --without-xen \
  --without-heartbeat
total 4.0K
drwxr-xr-x. 7 root root   67 Dec 23 20:06 .
dr-xr-x---. 6 root root 4.0K Dec 23 20:06 ..
drwxr-xr-x. 2 root root    6 Dec 23 20:06 BUILD
drwxr-xr-x. 2 root root    6 Dec 23 20:06 RPMS
drwxr-xr-x. 2 root root    6 Dec 23 20:06 SOURCES
drwxr-xr-x. 2 root root    6 Dec 23 20:06 SPECS
drwxr-xr-x. 2 root root    6 Dec 23 20:06 SRPMS

Userland tools: 

make rpm
checking for presence of 8\.4\.4 in various changelog files
<snip>
+ exit 0
You have now:
/root/rpmbuild/RPMS/x86_64/drbd-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-utils-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-xen-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-udev-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-pacemaker-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-heartbeat-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-bash-completion-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-debuginfo-8.4.4-4.el7.x86_64.rpm

Kernel module: 

make kmp-rpm
checking for presence of 8\.4\.4 in various changelog files
<snip>
+ exit 0
You have now:
/root/rpmbuild/RPMS/x86_64/drbd-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-utils-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-xen-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-udev-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-pacemaker-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-heartbeat-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-bash-completion-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-debuginfo-8.4.4-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/kmod-drbd-8.4.4_3.10.0_54.0.1-4.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/drbd-kernel-debuginfo-8.4.4-4.el7.x86_64.rpm

Configure DRBD

Configure global-common.conf; 

vim /etc/drbd.d/global_common.conf
# These are options to set for the DRBD daemon sets the default values for
# resources.
global {
	# This tells DRBD that you allow it to report this installation to 
	# LINBIT for statistical purposes. If you have privacy concerns, set
	# this to 'no'. The default is 'ask' which will prompt you each time
	# DRBD is updated. Set to 'yes' to allow it without being prompted.
	usage-count no;

	# minor-count dialog-refresh disable-ip-verification
}

common {
	handlers {
		pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
		pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
		local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
		# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
		# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
		# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
		# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
		
		# Hook into Pacemaker's fencing.
		fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
	}

	startup {
		# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
	}

	options {
		# cpu-mask on-no-data-accessible
	}

	disk {
		# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
		# disk-drain md-flushes resync-rate resync-after al-extents
                # c-plan-ahead c-delay-target c-fill-target c-max-rate
                # c-min-rate disk-timeout
                fencing resource-and-stonith;
	}

	net {
		# protocol timeout max-epoch-size max-buffers unplug-watermark
		# connect-int ping-int sndbuf-size rcvbuf-size ko-count
		# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
		# after-sb-1pri after-sb-2pri always-asbp rr-conflict
		# ping-timeout data-integrity-alg tcp-cork on-congestion
		# congestion-fill congestion-extents csums-alg verify-alg
		# use-rle

		# Protocol "C" tells DRBD not to tell the operating system that
		# the write is complete until the data has reach persistent
		# storage on both nodes. This is the slowest option, but it is
		# also the only one that guarantees consistency between the
		# nodes. It is also required for dual-primary, which we will 
		# be using.
		protocol C;

		# Tell DRBD to allow dual-primary. This is needed to enable 
		# live-migration of our servers.
		allow-two-primaries yes;

		# This tells DRBD what to do in the case of a split-brain when
		# neither node was primary, when one node was primary and when
		# both nodes are primary. In our case, we'll be running
		# dual-primary, so we can not safely recover automatically. The
		# only safe option is for the nodes to disconnect from one
		# another and let a human decide which node to invalidate. Of 
		after-sb-0pri discard-zero-changes;
		after-sb-1pri discard-secondary;
		after-sb-2pri disconnect;
	}
}

And now configure the first resource; 

vim /etc/drbd.d/r0.res
# This is the first DRBD resource. If will store the shared file systems and
# the servers designed to run on node 01.
resource r0 {
	# These options here are common to both nodes. If for some reason you
	# need to set unique values per node, you can move these to the
	# 'on <name> { ... }' section.
	
	# This sets the device name of this DRBD resouce.
	device /dev/drbd0;

	# This tells DRBD what the backing device is for this resource.
	disk /dev/sda5;

	# This controls the location of the metadata. When "internal" is used,
	# as we use here, a little space at the end of the backing devices is
	# set aside (roughly 32 MB per 1 TB of raw storage). External metadata
	# can be used to put the metadata on another partition when converting
	# existing file systems to be DRBD backed, when there is no extra space
	# available for the metadata.
	meta-disk internal;

	# NOTE: this is not required or even recommended with pacemaker. remove
	# 	this options as soon as pacemaker is setup.
	startup {
		# This tells DRBD to promote both nodes to 'primary' when this
		# resource starts. However, we will let pacemaker control this
		# so we comment it out, which tells DRBD to leave both nodes
		# as secondary when drbd starts.
		#become-primary-on both;
	}

	# NOTE: Later, make it an option in the dashboard to trigger a manual
	# 	verify and/or schedule periodic automatic runs
	net {
		# TODO: Test performance differences between sha1 and md5
		# This tells DRBD how to do a block-by-block verification of
		# the data stored on the backing devices. Any verification
		# failures will result in the effected block being marked
		# out-of-sync.
		verify-alg md5;

		# TODO: Test the performance hit of this being enabled.
		# This tells DRBD to generate a checksum for each transmitted
		# packet. If the data received data doesn't generate the same
		# sum, a retransmit request is generated. This protects against
		# otherwise-undetected errors in transmission, like 
		# bit-flipping. See:
		# http://www.drbd.org/users-guide/s-integrity-check.html
		data-integrity-alg md5;
	}

	# WARNING: Confirm that these are safe when the controller's BBU is
	#          depleted/failed and the controller enters write-through 
	#          mode.
	disk {
		# TODO: Test the real-world performance differences gained with
		#       these options.
		# This tells DRBD not to bypass the write-back caching on the
		# RAID controller. Normally, DRBD forces the data to be flushed
		# to disk, rather than allowing the write-back cachine to 
		# handle it. Normally this is dangerous, but with BBU-backed
		# caching, it is safe. The first option disables disk flushing
		# and the second disabled metadata flushes.
		disk-flushes no;
		md-flushes no;
	}

	# This sets up the resource on node 01. The name used below must be the
	# named returned by "uname -n".
	on an-a04n01.alteeve.ca {
		# This is the address and port to use for DRBD traffic on this
		# node. Multiple resources can use the same IP but the ports
		# must differ. By convention, the first resource uses 7788, the
		# second uses 7789 and so on, incrementing by one for each
		# additional resource. 
		address 10.10.40.1:7788;
	}
	on an-a04n02.alteeve.ca {
		address 10.10.40.2:7788;
	}
}

Disable drbd from starting on boot. 

systemctl disable drbd.service
drbd.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig drbd off

Load the config; 

modprobe drbd

Now check the config; 

drbdadm dump
  --==  Thank you for participating in the global usage survey  ==--
The server's response is:

you are the 69th user to install this version
/etc/drbd.d/r0.res:3: in resource r0:
become-primary-on is set to both, but allow-two-primaries is not set.

Ignore that error. It has been reported and does not effect operation.

Create the metadisk; 

drbdadm create-md r0
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success

Start the DRBD resource on both nodes; 

drbdadm up r0

Once /proc/drbd shows both nodes connected, force one to primary and it will sync over the second. 

drbdadm primary --force r0

You should see the resource syncing now. Push both nodes to primary; 

drbdadm primary r0

DLM, Clustered LVM and GFS2

an-a04n01 
sed -i.anvil 's^filter = \[ "a/\.\*/" \]^filter = \[ "a|/dev/drbd*|", "r/.*/" \]^' /etc/lvm/lvm.conf
sed -i 's/locking_type = 1$/locking_type = 3/' /etc/lvm/lvm.conf
sed -i 's/fallback_to_local_locking = 1$/fallback_to_local_locking = 0/' /etc/lvm/lvm.conf 
sed -i 's/use_lvmetad = 1$/use_lvmetad = 0/' /etc/lvm/lvm.conf
--- /etc/lvm/lvm.conf.anvil	2013-11-27 03:28:08.000000000 -0500
+++ /etc/lvm/lvm.conf	2014-01-26 18:57:41.026928464 -0500
@@ -84,7 +84,7 @@
     # lvmetad is used" comment that is attached to global/use_lvmetad setting.
 
     # By default we accept every block device:
-    filter = [ "a/.*/" ]
+    filter = [ "a|/dev/drbd*|", "r/.*/" ]
 
     # Exclude the cdrom drive
     # filter = [ "r|/dev/cdrom|" ]
@@ -451,7 +451,7 @@
     # supported in clustered environment. If use_lvmetad=1 and locking_type=3
     # is set at the same time, LVM always issues a warning message about this
     # and then it automatically disables lvmetad use.
-    locking_type = 1
+    locking_type = 3
 
     # Set to 0 to fail when a lock request cannot be satisfied immediately.
     wait_for_locks = 1
@@ -467,7 +467,7 @@
     # to 1 an attempt will be made to use local file-based locking (type 1).
     # If this succeeds, only commands against local volume groups will proceed.
     # Volume Groups marked as clustered will be ignored.
-    fallback_to_local_locking = 1
+    fallback_to_local_locking = 0
 
     # Local non-LV directory that holds file-based locks while commands are
     # in progress.  A directory like /tmp that may get wiped on reboot is OK.
@@ -594,7 +594,7 @@
     # supported in clustered environment. If use_lvmetad=1 and locking_type=3
     # is set at the same time, LVM always issues a warning message about this
     # and then it automatically disables lvmetad use.
-    use_lvmetad = 1
+    use_lvmetad = 0
 
     # Full path of the utility called to check that a thin metadata device
     # is in a state that allows it to be used.
rsync -av /etc/lvm/lvm.conf* root@an-a04n02:/etc/lvm/
sending incremental file list
lvm.conf
lvm.conf.anvil

sent 48536 bytes  received 440 bytes  97952.00 bytes/sec
total size is 90673  speedup is 1.85
an-a04n02 
diff -u /etc/lvm/lvm.conf.anvil /etc/lvm/lvm.conf
--- /etc/lvm/lvm.conf.anvil	2013-11-27 03:28:08.000000000 -0500
+++ /etc/lvm/lvm.conf	2014-01-26 18:57:41.000000000 -0500
@@ -84,7 +84,7 @@
     # lvmetad is used" comment that is attached to global/use_lvmetad setting.
 
     # By default we accept every block device:
-    filter = [ "a/.*/" ]
+    filter = [ "a|/dev/drbd*|", "r/.*/" ]
 
     # Exclude the cdrom drive
     # filter = [ "r|/dev/cdrom|" ]
@@ -451,7 +451,7 @@
     # supported in clustered environment. If use_lvmetad=1 and locking_type=3
     # is set at the same time, LVM always issues a warning message about this
     # and then it automatically disables lvmetad use.
-    locking_type = 1
+    locking_type = 3
 
     # Set to 0 to fail when a lock request cannot be satisfied immediately.
     wait_for_locks = 1
@@ -467,7 +467,7 @@
     # to 1 an attempt will be made to use local file-based locking (type 1).
     # If this succeeds, only commands against local volume groups will proceed.
     # Volume Groups marked as clustered will be ignored.
-    fallback_to_local_locking = 1
+    fallback_to_local_locking = 0
 
     # Local non-LV directory that holds file-based locks while commands are
     # in progress.  A directory like /tmp that may get wiped on reboot is OK.
@@ -594,7 +594,7 @@
     # supported in clustered environment. If use_lvmetad=1 and locking_type=3
     # is set at the same time, LVM always issues a warning message about this
     # and then it automatically disables lvmetad use.
-    use_lvmetad = 1
+    use_lvmetad = 0
 
     # Full path of the utility called to check that a thin metadata device
     # is in a state that allows it to be used.

Disable lvmetad as it's not cluster-aware.

an-a04n01 
systemctl disable lvm2-lvmetad.service
systemctl disable lvm2-lvmetad.socket
systemctl stop lvm2-lvmetad.service
rm '/etc/systemd/system/sockets.target.wants/lvm2-lvmetad.socket'
an-a04n02 
systemctl disable lvm2-lvmetad.service
systemctl disable lvm2-lvmetad.socket
systemctl stop lvm2-lvmetad.service
rm '/etc/systemd/system/sockets.target.wants/lvm2-lvmetad.socket'
 Note: This will be moved to pacemaker shortly. We're enabling it here just long enough to configure pacemaker.

Start DLM and clvmd;

an-a04n01 
systemctl start dlm.service
systemctl start clvmd.service
an-a04n02 
systemctl start dlm.service
systemctl start clvmd.service

Create the PV, VG and the /shared LV;

an-a04n01 
pvcreate /dev/drbd0
  Physical volume "/dev/drbd0" successfully created
vgcreate an-a04n01_vg0 /dev/drbd0
  /proc/devices: No entry for device-mapper found
  Clustered volume group "an-a04n01_vg0" successfully created
lvcreate -L 10G -n shared an-a04n01_vg0
  Logical volume "shared" created
an-a04n02 
pvscan
  PV /dev/drbd0   VG an-a04n01_vg0   lvm2 [20.00 GiB / 20.00 GiB free]
  Total: 1 [20.00 GiB] / in use: 1 [20.00 GiB] / in no VG: 0 [0   ]
vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "an-a04n01_vg0" using metadata type lvm2
lvscan
  ACTIVE            '/dev/an-a04n01_vg0/shared' [10.00 GiB] inherit

Format the /dev/an-a04n01_vg0/shared;

an-a04n01 
mkfs.gfs2 -j 2 -p lock_dlm -t an-anvil-04:shared /dev/an-a04n01_vg0/shared
/dev/an-a04n01_vg0/shared is a symbolic link to /dev/dm-0
This will destroy any data on /dev/dm-0
Are you sure you want to proceed? [y/n]y
Device:                    /dev/an-a04n01_vg0/shared
Block size:                4096
Device size:               10.00 GB (2621440 blocks)
Filesystem size:           10.00 GB (2621438 blocks)
Journals:                  2
Resource groups:           40
Locking protocol:          "lock_dlm"
Lock table:                "an-anvil-04:shared"
UUID:                      20bafdb0-1f86-f424-405b-9bf608c0c486
mkdir /shared
mount /dev/an-a04n01_vg0/shared /shared
df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/vda3                           18G  5.6G   12G  32% /
devtmpfs                           932M     0  932M   0% /dev
tmpfs                              937M   61M  877M   7% /dev/shm
tmpfs                              937M  1.9M  935M   1% /run
tmpfs                              937M     0  937M   0% /sys/fs/cgroup
/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
/dev/vda1                          484M   83M  401M  18% /boot
/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared
an-a04n02 
Filesystem                         Size  Used Avail Use% Mounted on
/dev/vda3                           18G  5.6G   12G  32% /
devtmpfs                           932M     0  932M   0% /dev
tmpfs                              937M   76M  862M   9% /dev/shm
tmpfs                              937M  2.0M  935M   1% /run
tmpfs                              937M     0  937M   0% /sys/fs/cgroup
/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
/dev/vda1                          484M   83M  401M  18% /boot
/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared

Shut down gfs2, clvmd and drbd now.

an-a04n01 
umount /shared/
systemctl stop clvmd.service
drbdadm down r0
an-a04n02 
umount /shared/
systemctl stop clvmd.service
drbdadm down r0

Done.

Add Storage to Pacemaker

Configure Dual-Primary DRBD

Setup DRBD as a dual-primary resource.

an-a04n01 
pcs cluster cib drbd_cfg
pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 op monitor interval=60s
pcs -f drbd_cfg resource master drbd_r0_Clone drbd_r0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs cluster cib-push drbd_cfg
CIB updated

Give it a couple minutes to promote both nodes to Master on both nodes. Initially, it will appear as Master on one node only.

Once updated, you should see this:

an-a04n01 
pcs status
Cluster name: an-anvil-04
Last updated: Sun Jan 26 20:26:33 2014
Last change: Sun Jan 26 20:23:23 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: corosync
Current DC: an-a04n02.alteeve.ca (2) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
4 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_virsh	(stonith:fence_virsh):	Started an-a04n01.alteeve.ca 
 fence_n02_virsh	(stonith:fence_virsh):	Started an-a04n02.alteeve.ca 
 Master/Slave Set: drbd_r0_Clone [drbd_r0]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

PCSD Status:
an-a04n01.alteeve.ca: 
  an-a04n01.alteeve.ca: Online
an-a04n02.alteeve.ca: 
  an-a04n02.alteeve.ca: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled
an-a04n02 
pcs status
Cluster name: an-anvil-04
Last updated: Sun Jan 26 20:26:58 2014
Last change: Sun Jan 26 20:23:23 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: corosync
Current DC: an-a04n02.alteeve.ca (2) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
4 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_virsh	(stonith:fence_virsh):	Started an-a04n01.alteeve.ca 
 fence_n02_virsh	(stonith:fence_virsh):	Started an-a04n02.alteeve.ca 
 Master/Slave Set: drbd_r0_Clone [drbd_r0]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

PCSD Status:
an-a04n01.alteeve.ca: 
  an-a04n01.alteeve.ca: Online
an-a04n02.alteeve.ca: 
  an-a04n02.alteeve.ca: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

Configure DLM

an-a04n01 
pcs cluster cib dlm_cfg
pcs -f dlm_cfg resource create dlm ocf:pacemaker:controld op monitor interval=60s
pcs -f dlm_cfg resource clone dlm clone-max=2 clone-node-max=1
pcs cluster cib-push dlm_cfg
CIB updated
an-a04n02 
pcs status
Cluster name: an-anvil-04
Last updated: Sun Jan 26 20:34:36 2014
Last change: Sun Jan 26 20:33:31 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: corosync
Current DC: an-a04n02.alteeve.ca (2) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
6 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_virsh	(stonith:fence_virsh):	Started an-a04n01.alteeve.ca 
 fence_n02_virsh	(stonith:fence_virsh):	Started an-a04n02.alteeve.ca 
 Master/Slave Set: drbd_r0_Clone [drbd_r0]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Clone Set: dlm-clone [dlm]
     Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

PCSD Status:
an-a04n01.alteeve.ca: 
  an-a04n01.alteeve.ca: Online
an-a04n02.alteeve.ca: 
  an-a04n02.alteeve.ca: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

Configure Cluster LVM

an-a04n01 
pcs cluster cib clvmd_cfg
pcs -f clvmd_cfg resource create clvmd lsb:clvmd params daemon_timeout=30s op monitor interval=60s
pcs -f clvmd_cfg resource clone clvmd clone-max=2 clone-node-max=1
pcs -f clvmd_cfg constraint colocation add dlm-clone clvmd-clone INFINITY
pcs -f clvmd_cfg constraint order start dlm then start clvmd-clone
pcs cluster cib-push clvmd_cfg
CIB updated
an-a04n02 
pcs status
Cluster name: an-anvil-04
Last updated: Mon Jan 27 19:00:33 2014
Last change: Mon Jan 27 19:00:19 2014 via crm_resource on an-a04n01.alteeve.ca
Stack: corosync
Current DC: an-a04n01.alteeve.ca (1) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
8 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_virsh        (stonith:fence_virsh):  Started an-a04n01.alteeve.ca
 fence_n02_virsh        (stonith:fence_virsh):  Started an-a04n02.alteeve.ca
 Master/Slave Set: drbd_r0_Clone [drbd_r0]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Clone Set: dlm-clone [dlm]
     Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Clone Set: clvmd-clone [clvmd]
     Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

PCSD Status:
an-a04n01.alteeve.ca:
  an-a04n01.alteeve.ca: Online
an-a04n02.alteeve.ca:
  an-a04n02.alteeve.ca: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

Configure the /shared GFS2 Partition

an-a04n01 
pcs cluster cib fs_cfg
pcs -f fs_cfg resource create sharedFS Filesystem device="/dev/an-a04n01_vg0/shared" directory="/shared" fstype="gfs2"
pcs -f fs_cfg resource clone sharedFS
pcs cluster cib-push fs_cfg
CIB updated
df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/vda3                           18G  5.6G   12G  32% /
devtmpfs                           932M     0  932M   0% /dev
tmpfs                              937M   61M  877M   7% /dev/shm
tmpfs                              937M  2.2M  935M   1% /run
tmpfs                              937M     0  937M   0% /sys/fs/cgroup
/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
/dev/vda1                          484M   83M  401M  18% /boot
/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared
an-a04n02 
df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/vda3                           18G  5.6G   12G  32% /
devtmpfs                           932M     0  932M   0% /dev
tmpfs                              937M   76M  862M   9% /dev/shm
tmpfs                              937M  2.6M  935M   1% /run
tmpfs                              937M     0  937M   0% /sys/fs/cgroup
/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
/dev/vda1                          484M   83M  401M  18% /boot
/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared

Configuring Constraints

an-a04n01 
pcs cluster cib cst_cfg
pcs -f cst_cfg constraint order start dlm then promote drbd_r0_Clone
pcs -f cst_cfg constraint order promote drbd_r0_Clone then start clvmd-clone
pcs -f cst_cfg constraint order promote clvmd-clone then start sharedFS-clone
pcs cluster cib-push cst_cfg
CIB updated
pcs constraint show
Location Constraints:
Ordering Constraints:
  start dlm then promote drbd_r0_Clone
  promote drbd_r0_Clone then start clvmd-clone
  start clvmd-clone then start sharedFS-clone
Colocation Constraints:
an-a04n02 
pcs constraint show
Location Constraints:
Ordering Constraints:
  start dlm then promote drbd_r0_Clone
  promote drbd_r0_Clone then start clvmd-clone
  start clvmd-clone then start sharedFS-clone
Colocation Constraints:

Odds and Sods

This is a section for random notes. The stuff here will be integrated into the finished tutorial or removed.

Determine multicast Address

Useful if you need to ensure that your switch has persistent multicast addresses set. 

corosync-cmapctl | grep mcastaddr
totem.interface.0.mcastaddr (str) = 239.192.122.199



an-a04n01
an-a04n02

Notes

  • Pacemaker Logging
  • Editing cib.xml offline is possible with: CIB_file=/path/to/real/cib.xml cibadmin .... and sync to other nodes when done.

Thanks

This list will certainly grow as this tutorial progresses;

 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche! Enterprise Support:
Alteeve Support 		Community Support
© Alteeve's Niche! Inc. 1997-2024   Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.
Retrieved from ‘https://alteeve.com/w/index.php?title=Anvil!_Tutorial_3&oldid=7226’