Latest revision as of 12:23, 21 July 2014

AN!Wiki :: How To :: Anvil! Tutorial 3 on EL6

Warning: This tutorial is incomplete, flawed and generally sucks at this time. Do not follow this and expect anything to work. In large part, it's a dumping ground for notes and little else. This warning will be removed when the tutorial is completed.

This is the third Anvil! tutorial built on Red Hat's Enterprise Linux 6.5 and newer. It is meant to be a stop-gap / learning cluster before RHEL 7 is released and stabilized.

Before We Begin

This tutorial does not require prior Anvil! experience (or any clustering experience), but it does expect a certain familiarity with Linux and a low-intermediate understanding of networking. Where possible, steps are explained in detail and rationale is provided for why certain decisions are made.

For those with Anvil! experience;

Please be careful not to skip too much. There are some major and some subtle changes from previous tutorials.

OS Setup

Warning: RHEL v6.5 or newer is required.

Post OS Install

Stuff.

If you're using RHEL proper, register your nodes with RHN.

Note: You need to replace $user and $pass with your RHN account details.

an-a04n01

rhnreg_ks --username "$user" --password "$pass" --force --profilename "an-a04n01.alteeve.ca"
rhn-channel --add --user "$user" --password "$pass" --channel=rhel-x86_64-server-rs-6

an-a04n02

rhnreg_ks --username "$user" --password "$pass" --force --profilename "an-a04n02.alteeve.ca"
rhn-channel --add --user "$user" --password "$pass" --channel=rhel-x86_64-server-rs-6

Adding AN! Repo

AN! offers a new repo with a few RPMs not in stock EL 6 distros.

an-a04n01

an-a04n02

cat <<-END>/etc/yum.repos.d/an.repo
[an-repo]
name=AN! Repo for Anvil! stuff
baseurl=https://alteeve.ca/repo/el6/
enabled=1
gpgcheck=0
protect=1
END

yum clean all

cat <<-END>/etc/yum.repos.d/an.repo
[an-repo]
name=AN! Repo for Anvil! stuff
baseurl=https://alteeve.ca/repo/el6/
enabled=1
gpgcheck=0
protect=1
END

yum clean all

Done.

Install

Not all of these are required, but most are used at one point or another in this tutorial.

Note: The fence-agents-virsh package is not available in RHEL 7 beta. Further, it's only needed if you're building your Anvil! using VMs.

an-a04n01

an-a04n02

yum -y update
yum -y install bridge-utils vim pacemaker corosync cman gfs2-utils \
               ccs pcs ipmitool OpenIPMI lvm2-cluster drbd84-utils \
               drbd84-kmod
chkconfig ipmi on
chkconfig acpid off
chkconfig kdump off
chkconfig drbd off
/etc/init.d/ipmi start
/etc/init.d/acpid stop
/etc/init.d/kdump stop
/etc/init.d/drbd stop

# same as an-a04n01

Setup Networking

TODO: Explain this.

Remap all NICs to have purpose-based names.

IFN Bridge

an-a04n01

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-ifn-bridge1

# Internet-Facing Network - Bridge
DEVICE="ifn-bridge1"
TYPE="Bridge"
BOOTPROTO="none"
IPADDR="10.255.40.1"
NETMASK="255.255.0.0"
GATEWAY="10.255.255.254"
DNS1="8.8.8.8"
DNS2="8.8.4.4"
DEFROUTE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-ifn-bridge1

# Internet-Facing Network - Bridge
DEVICE="ifn-bridge1"
TYPE="Bridge"
BOOTPROTO="none"
IPADDR="10.255.40.2"
NETMASK="255.255.0.0"
GATEWAY="10.255.255.254"
DNS1="8.8.8.8"
DNS2="8.8.4.4"
DEFROUTE="yes"

IFN Bond

an-a04n01

vim /etc/sysconfig/network-scripts/ifcfg-ifn-bond1

# Internet-Facing Network - Bond
DEVICE="ifn-bond1"
BRIDGE="ifn-bridge1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=ifn-link1"

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-ifn-bond1

# Internet-Facing Network - Bond
DEVICE="ifn-bond1"
BRIDGE="ifn-bridge1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=ifn-link1"

IFN Links

an-a04n01

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-ifn-link1

# Internet-Facing Network - Link 1
HWADDR="00:1B:21:81:C3:34"
DEVICE="ifn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="ifn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-ifn-link2

# Internet-Facing Network - Link 2
HWADDR="A0:36:9F:02:E0:05"
DEVICE="ifn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="ifn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-ifn-link1

# Internet-Facing Network - Link 1
HWADDR="00:1B:21:81:C2:EA"
DEVICE="ifn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="ifn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-ifn-link2

# Internet-Facing Network - Link 2
HWADDR="A0:36:9F:07:D6:2F"
DEVICE="ifn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="ifn-bond1"
SLAVE="yes"

SN Bond

an-a04n01

vim /etc/sysconfig/network-scripts/ifcfg-sn-bond1

# Storage Network - Bond
DEVICE="sn-bond1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=sn-link1"
IPADDR="10.10.40.1"
NETMASK="255.255.0.0"

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-sn-bond1

# Storage Network - Bond
DEVICE="sn-bond1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=sn-link1"
IPADDR="10.10.40.2"
NETMASK="255.255.0.0"

SN Links

an-a04n01

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-sn-link1

# Storage Network - Link 1
HWADDR="00:19:99:9C:9B:9F"
DEVICE="sn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="sn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-sn-link2

# Storage Network - Link 2
HWADDR="A0:36:9F:02:E0:04"
DEVICE="sn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="sn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-sn-link1

# Storage Network - Link 1
HWADDR="00:19:99:9C:A0:6D"
DEVICE="sn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="sn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-sn-link2

# Storage Network - Link 2
HWADDR="A0:36:9F:07:D6:2E"
DEVICE="sn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="sn-bond1"
SLAVE="yes"

BCN Bond

an-a04n01

vim /etc/sysconfig/network-scripts/ifcfg-bcn-bond1

# Back-Channel Network - Bond
DEVICE="bcn-bond1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=bcn-link1"
IPADDR="10.20.40.1"
NETMASK="255.255.0.0"

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-bcn-bond1

# Back-Channel Network - Bond
DEVICE="bcn-bond1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=bcn-link1"
IPADDR="10.20.40.2"
NETMASK="255.255.0.0"

BCN Links

an-a04n01

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-bcn-link1

# Back-Channel Network - Link 1
HWADDR="00:19:99:9C:9B:9E"
DEVICE="bcn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="bcn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-bcn-link2

# Back-Channel Network - Link 2
HWADDR="00:1B:21:81:C3:35"
DEVICE="bcn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="bcn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-bcn-link1

# Back-Channel Network - Link 1
HWADDR="00:19:99:9C:A0:6C"
DEVICE="bcn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="bcn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-bcn-link2

# Back-Channel Network - Link 2
HWADDR="00:1B:21:81:C2:EB"
DEVICE="bcn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="bcn-bond1"
SLAVE="yes"

Making ssh faster when the net is down

By default, the nodes will try to resolve the host name of an incoming ssh connection. When the internet connection is down, DNS lookups have to time out, which can make login times quite slow. When something goes wrong, seconds count and waiting for up to a minute for an SSH password prompt can be maddening.

For this reason, we will make two changes to /etc/ssh/sshd_config that disable this login delay.

Please be aware that this can reduce security. If this is a concern, skip this step.

an-a04n01

sed -i.anvil 's/#GSSAPIAuthentication no/GSSAPIAuthentication no/' /etc/ssh/sshd_config
sed -i 's/GSSAPIAuthentication yes/#GSSAPIAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
systemctl restart sshd.service
diff -u /etc/ssh/sshd_config.anvil /etc/ssh/sshd_config

--- /etc/ssh/sshd_config.anvil	2013-09-30 03:08:17.000000000 -0400
+++ /etc/ssh/sshd_config	2014-05-28 00:35:30.954000741 -0400
@@ -77,8 +77,8 @@
 #KerberosUseKuserok yes
 
 # GSSAPI options
-#GSSAPIAuthentication no
-GSSAPIAuthentication yes
+GSSAPIAuthentication no
+#GSSAPIAuthentication yes
 #GSSAPICleanupCredentials yes
 GSSAPICleanupCredentials yes
 #GSSAPIStrictAcceptorCheck yes
@@ -119,7 +119,7 @@
 #ClientAliveInterval 0
 #ClientAliveCountMax 3
 #ShowPatchLevel no
-#UseDNS yes
+UseDNS no
 #PidFile /var/run/sshd.pid
 #MaxStartups 10:30:100
 #PermitTunnel no

an-a04n02

sed -i.anvil 's/#GSSAPIAuthentication no/GSSAPIAuthentication no/' /etc/ssh/sshd_config
sed -i 's/GSSAPIAuthentication yes/#GSSAPIAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
systemctl restart sshd.service
diff -u /etc/ssh/sshd_config.anvil /etc/ssh/sshd_config

--- /etc/ssh/sshd_config.anvil	2013-09-30 03:08:17.000000000 -0400
+++ /etc/ssh/sshd_config	2014-05-28 00:35:33.016999110 -0400
@@ -77,8 +77,8 @@
 #KerberosUseKuserok yes
 
 # GSSAPI options
-#GSSAPIAuthentication no
-GSSAPIAuthentication yes
+GSSAPIAuthentication no
+#GSSAPIAuthentication yes
 #GSSAPICleanupCredentials yes
 GSSAPICleanupCredentials yes
 #GSSAPIStrictAcceptorCheck yes
@@ -119,7 +119,7 @@
 #ClientAliveInterval 0
 #ClientAliveCountMax 3
 #ShowPatchLevel no
-#UseDNS yes
+UseDNS no
 #PidFile /var/run/sshd.pid
 #MaxStartups 10:30:100
 #PermitTunnel no

Subsequent logins when the net is down should be quick.

Setting the Hostname

TODO

Setup The hosts File

You can use DNS if you prefer. For now, lets use /etc/hosts for node name resolution.

an-a04n01

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

### Anvil! systems
# Anvil! 03, Node 01
10.20.40.1	an-a04n01.bcn an-a04n01 an-a04n01.alteeve.ca
10.20.41.1	an-a04n01.ipmi
10.10.40.1	an-a04n01.sn
10.255.40.1	an-a04n01.ifn

# Anvil! 03, Node 02
10.20.40.2	an-a04n02.bcn an-a04n02 an-a04n02.alteeve.ca
10.20.41.2	an-a04n02.ipmi
10.10.40.2	an-a04n02.sn
10.255.40.2	an-a04n02.ifn

### Foundation Pack
# Network Switches
10.20.1.1	an-s01 an-s01.alteeve.ca
10.20.1.2	an-s02 an-s02.alteeve.ca	# Only accessible when out of the stack
 
# Switched PDUs
10.20.2.1	an-p01 an-p01.alteeve.ca
10.20.2.2	an-p02 an-p02.alteeve.ca
 
# Network-monitored UPSes
10.20.3.1	an-u01 an-u01.alteeve.ca
10.20.3.2	an-u02 an-u02.alteeve.ca
 
### Monitor Packs
10.20.4.1	an-m01 an-m01.alteeve.ca
10.255.4.1	an-m01.ifn
10.20.4.2	an-m02 an-m02.alteeve.ca
10.255.4.2	an-m02.ifn

an-a04n02

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

### Anvil! systems
# Anvil! 03, Node 01
10.20.40.1	an-a04n01.bcn an-a04n01 an-a04n01.alteeve.ca
10.20.41.1	an-a04n01.ipmi
10.10.40.1	an-a04n01.sn
10.255.40.1	an-a04n01.ifn

# Anvil! 03, Node 02
10.20.40.2	an-a04n02.bcn an-a04n02 an-a04n02.alteeve.ca
10.20.41.2	an-a04n02.ipmi
10.10.40.2	an-a04n02.sn
10.255.40.2	an-a04n02.ifn

### Foundation Pack
# Network Switches
10.20.1.1	an-s01 an-s01.alteeve.ca
10.20.1.2	an-s02 an-s02.alteeve.ca	# Only accessible when out of the stack
 
# Switched PDUs
10.20.2.1	an-p01 an-p01.alteeve.ca
10.20.2.2	an-p02 an-p02.alteeve.ca
 
# Network-monitored UPSes
10.20.3.1	an-u01 an-u01.alteeve.ca
10.20.3.2	an-u02 an-u02.alteeve.ca
 
### Monitor Packs
10.20.4.1	an-m01 an-m01.alteeve.ca
10.255.4.1	an-m01.ifn
10.20.4.2	an-m02 an-m02.alteeve.ca
10.255.4.2	an-m02.ifn

Setup SSH

Same as before.

Populating And Pushing ~/.ssh/known_hosts

an-a04n01

ssh-keygen -t rsa -N "" -b 8191 -f ~/.ssh/id_rsa

Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
f9:41:7e:aa:96:8e:fa:47:79:f5:3a:33:89:c3:9a:4b root@an-a04n01.alteeve.ca
The key's randomart image is:
+--[ RSA 8191]----+
|                 |
|                 |
|          .      |
|         +  .    |
|        S.o...   |
|        o..+  .  |
|       .E+o. o   |
|       o+o+ *    |
|    .oo+*o . +   |
+-----------------+

an-a04n01

ssh-keygen -t rsa -N "" -b 8191 -f ~/.ssh/id_rsa

Generating public/private rsa key pair.
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
3f:1a:02:17:44:10:5e:6f:2b:98:44:09:e5:e0:ea:4b root@an-a04n02.alteeve.ca
The key's randomart image is:
+--[ RSA 8191]----+
|  oo==+          |
| . =.o .         |
|  . + . o        |
| . . o o .       |
|.   + o S        |
|.    o . .       |
| E    . . o      |
|. .    . o .     |
| .      .        |
+-----------------+

Setup autorized_keys:

an-a04n01

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh root@an-a04n02 "cat /root/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys

The authenticity of host 'an-a04n02 (10.20.40.2)' can't be established.
RSA key fingerprint is 22:09:7b:0c:8b:d8:80:08:80:6d:0e:bc:fb:5a:e1:de.
Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'an-a04n02,10.20.40.2' (RSA) to the list of known hosts.

root@an-a04n02's password:

Populate ~/.ssh/known_hosts:

an-a04n01

ssh-keyscan an-a04n01.alteeve.ca >> ~/.ssh/known_hosts

# an-a04n01.alteeve.ca SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n01 >> ~/.ssh/known_hosts

# an-a04n01 SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n01.bcn >> ~/.ssh/known_hosts

# an-a04n01.bcn SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n01.sn >> ~/.ssh/known_hosts

# an-a04n01.sn SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n01.ifn >> ~/.ssh/known_hosts

# an-a04n01.ifn SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n02.alteeve.ca >> ~/.ssh/known_hosts

# an-a04n02.alteeve.ca SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n02 >> ~/.ssh/known_hosts

# an-a04n02 SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n02.bcn >> ~/.ssh/known_hosts

# an-a04n02.bcn SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n02.sn >> ~/.ssh/known_hosts

# an-a04n02.sn SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n02.ifn >> ~/.ssh/known_hosts

# an-a04n02.ifn SSH-2.0-OpenSSH_5.3

Now copy the files to the second node:

an-a04n01

rsync -av ~/.ssh/authorized_keys root@an-a04n02:/root/.ssh/

root@an-a04n02's password:

sending incremental file list
authorized_keys

sent 2937 bytes  received 31 bytes  1187.20 bytes/sec
total size is 2854  speedup is 0.96

rsync -av ~/.ssh/known_hosts root@an-a04n02:/root/.ssh/

sending incremental file list
known_hosts

sent 4829 bytes  received 31 bytes  9720.00 bytes/sec
total size is 4750  speedup is 0.98

Note that there was no password prompt the second time. Hoozah!

Configuring the Firewall

an-a04n01

# cman (corosync's totem)
iptables -I INPUT -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 -d 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
iptables -I INPUT -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 --dports 5404,5405 -j ACCEPT

# dlm
iptables -I INPUT -m state --state NEW -p tcp -s 10.20.0.0/16 -d 10.20.0.0/16 --dport 21064 -j ACCEPT 

# DRBD resource 0 and 1 - on the SN
iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7788 -j ACCEPT
iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7789 -j ACCEPT

# Make the new rules persistent.
/etc/init.d/iptables save

iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]

an-a04n01

# cman (corosync's totem)
iptables -I INPUT -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 -d 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
iptables -I INPUT -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 --dports 5404,5405 -j ACCEPT

# dlm
iptables -I INPUT -m state --state NEW -p tcp -s 10.20.0.0/16 -d 10.20.0.0/16 --dport 21064 -j ACCEPT 

# DRBD resource 0 and 1 - on the SN
iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7788 -j ACCEPT
iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7789 -j ACCEPT

# Make the new rules persistent.
/etc/init.d/iptables save

iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]

Keeping Time in Sync

It's not as critical as it used to be to keep the clocks on the nodes in sync, but it's still a good idea.

an-a04n01	chkconfig ntpd on /etc/init.d/ntpd start Starting ntpd: [ OK ]
an-a04n01	chkconfig ntpd on /etc/init.d/ntpd start Starting ntpd: [ OK ]

Configuring the Anvil!

Now we're getting down to business!

For this section, we will be working on an-a04n01 and using ssh to perform tasks on an-a04n02.

Note: TODO: explain what this is and how it works.

Configuring cman

With RHEL 6, we do not need to configure corosync directly. We will create a "skeleton" cluster.conf file which will, in turn, handle corosync for us. Once configured and the configuration has been copied to the peer, we will start pacemaker and it will handle starting (and stopping) pacemaker and corosync for us.

We will use 'ccs' to configure the skeleton cluster.conf file.

an-a04n01

ccs -f /etc/cluster/cluster.conf --createcluster an-anvil-04
ccs -f /etc/cluster/cluster.conf --setcman two_node="1" expected_votes="1"
ccs -f /etc/cluster/cluster.conf --addnode an-a04n01.alteeve.ca
ccs -f /etc/cluster/cluster.conf --addnode an-a04n02.alteeve.ca
ccs -f /etc/cluster/cluster.conf --addfencedev pcmk agent=fence_pcmk 
ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect an-a04n01.alteeve.ca
ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect an-a04n02.alteeve.ca
ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk an-a04n01.alteeve.ca pcmk-redirect port=an-a04n01.alteeve.ca
ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk an-a04n02.alteeve.ca pcmk-redirect port=an-a04n02.alteeve.ca
ccs -f /etc/cluster/cluster.conf --setfencedaemon post_join_delay="30"
cat /etc/cluster/cluster.conf

<cluster config_version="10" name="an-anvil-04">
  <fence_daemon post_join_delay="30"/>
  <clusternodes>
    <clusternode name="an-a04n01.alteeve.ca" nodeid="1">
      <fence>
        <method name="pcmk-redirect">
          <device name="pcmk" port="an-a04n01.alteeve.ca"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="an-a04n02.alteeve.ca" nodeid="2">
      <fence>
        <method name="pcmk-redirect">
          <device name="pcmk" port="an-a04n02.alteeve.ca"/>
        </method>
      </fence>
    </clusternode>
  </clusternodes>
  <cman expected_votes="1" two_node="1"/>
  <fencedevices>
    <fencedevice agent="fence_pcmk" name="pcmk"/>
  </fencedevices>
  <rm>
    <failoverdomains/>
    <resources/>
  </rm>
</cluster>

Copy it to an-a04n02;

an-a04n01

rsync -av /etc/cluster/cluster.conf root@an-a04n02:/etc/cluster/

sending incremental file list
cluster.conf

sent 838 bytes  received 31 bytes  579.33 bytes/sec
total size is 758  speedup is 0.87

an-a04n02

cat /etc/cluster/cluster.conf

<cluster config_version="10" name="an-anvil-04">
  <fence_daemon post_join_delay="30"/>
  <clusternodes>
    <clusternode name="an-a04n01.alteeve.ca" nodeid="1">
      <fence>
        <method name="pcmk-redirect">
          <device name="pcmk" port="an-a04n01.alteeve.ca"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="an-a04n02.alteeve.ca" nodeid="2">
      <fence>
        <method name="pcmk-redirect">
          <device name="pcmk" port="an-a04n02.alteeve.ca"/>
        </method>
      </fence>
    </clusternode>
  </clusternodes>
  <cman expected_votes="1" two_node="1"/>
  <fencedevices>
    <fencedevice agent="fence_pcmk" name="pcmk"/>
  </fencedevices>
  <rm>
    <failoverdomains/>
    <resources/>
  </rm>
</cluster>

Starting Pacemaker

Now start pacemaker proper.

an-a04n01

/etc/init.d/pacemaker start

Starting cluster: 
   Checking if cluster has been disabled at boot...        [  OK  ]
   Checking Network Manager...                             [  OK  ]
   Global setup...                                         [  OK  ]
   Loading kernel modules...                               [  OK  ]
   Mounting configfs...                                    [  OK  ]
   Starting cman...                                        [  OK  ]
   Waiting for quorum...                                   [  OK  ]
   Starting fenced...                                      [  OK  ]
   Starting dlm_controld...                                [  OK  ]
   Tuning DLM kernel config...                             [  OK  ]
   Starting gfs_controld...                                [  OK  ]
   Unfencing self...                                       [  OK  ]
   Joining fence domain...                                 [  OK  ]
Starting Pacemaker Cluster Manager                         [  OK  ]

an-a04n02

/etc/init.d/pacemaker start

Starting cluster: 
   Checking if cluster has been disabled at boot...        [  OK  ]
   Checking Network Manager...                             [  OK  ]
   Global setup...                                         [  OK  ]
   Loading kernel modules...                               [  OK  ]
   Mounting configfs...                                    [  OK  ]
   Starting cman...                                        [  OK  ]
   Waiting for quorum...                                   [  OK  ]
   Starting fenced...                                      [  OK  ]
   Starting dlm_controld...                                [  OK  ]
   Tuning DLM kernel config...                             [  OK  ]
   Starting gfs_controld...                                [  OK  ]
   Unfencing self...                                       [  OK  ]
   Joining fence domain...                                 [  OK  ]
Starting Pacemaker Cluster Manager                         [  OK  ]

Verify pacemaker proper started as expected.

an-a04n01

pcs status

Cluster name: an-anvil-04
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Wed May 28 20:59:33 2014
Last change: Wed May 28 20:59:18 2014 via crmd on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
0 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

an-a04n02

pcs status

WARNING: no stonith devices and stonith-enabled is not false
Last updated: Wed May 28 20:59:29 2014
Last change: Wed May 28 20:59:18 2014 via crmd on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
0 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

Note the error about stonith. We will address that momentarily.

Configure and test stonith (aka fencing)

We will use IPMI and PDU based fence devices with STONITH levels.

You can see the list of available fence agents here. You will need to find the one for your hardware fence devices.

Note: Ignore the errors.

an-a04n01

pcs stonith list

fence_apc - Fence agent for APC over telnet/ssh
fence_apc_snmp - Fence agent for APC over SNMP
fence_bladecenter - Fence agent for IBM BladeCenter
fence_bladecenter_snmp - Fence agent for IBM BladeCenter over SNMP
fence_brocade - Fence agent for Brocade over telnet
Error: no metadata for /usr/sbin/fence_check
fence_cisco_mds - Fence agent for Cisco MDS
fence_cisco_ucs - Fence agent for Cisco UCS
fence_drac - fencing agent for Dell Remote Access Card
fence_drac5 - Fence agent for Dell DRAC CMC/5
fence_eaton_snmp - Fence agent for Eaton over SNMP
fence_egenera - I/O Fencing agent for the Egenera BladeFrame
fence_eps - Fence agent for ePowerSwitch
fence_hpblade - Fence agent for HP BladeSystem
fence_ibmblade - Fence agent for IBM BladeCenter over SNMP
fence_idrac - Fence agent for IPMI over LAN
fence_ifmib - Fence agent for IF MIB
fence_ilo - Fence agent for HP iLO
fence_ilo2 - Fence agent for HP iLO
fence_ilo3 - Fence agent for IPMI over LAN
fence_ilo4 - Fence agent for IPMI over LAN
fence_ilo_mp - Fence agent for HP iLO MP
fence_imm - Fence agent for IPMI over LAN
fence_intelmodular - Fence agent for Intel Modular
fence_ipdu - Fence agent for iPDU over SNMP
fence_ipmilan - Fence agent for IPMI over LAN
fence_kdump - Fence agent for use with kdump
Error: no metadata for /usr/sbin/fence_node
fence_rhevm - Fence agent for RHEV-M REST API
fence_rsa - Fence agent for IBM RSA
fence_rsb - I/O Fencing agent for Fujitsu-Siemens RSB
fence_sanbox2 - Fence agent for QLogic SANBox2 FC switches
fence_scsi - fence agent for SCSI-3 persistent reservations
Error: no metadata for /usr/sbin/fence_tool
fence_virsh - Fence agent for virsh
fence_virt - Fence agent for virtual machines
fence_vmware - Fence agent for VMWare
fence_vmware_soap - Fence agent for VMWare over SOAP API
fence_wti - Fence agent for WTI
fence_xvm - Fence agent for virtual machines

We will use fence_ipmilan and fence_apc_snmp.

Configuring IPMI Fencing

Setup out IPMI BMCs (on LAN channel 2 and using user ID 2).

an-a04n01

ipmitool lan set 2 ipsrc static
ipmitool lan set 2 ipaddr 10.20.41.1
ipmitool lan set 2 netmask 255.255.0.0
ipmitool lan set 2 defgw ipaddr 10.20.255.254
ipmitool user set password 2 Initial1

an-a04n02

ipmitool lan set 2 ipsrc static
ipmitool lan set 2 ipaddr 10.20.41.2
ipmitool lan set 2 netmask 255.255.0.0
ipmitool lan set 2 defgw ipaddr 10.20.255.254
ipmitool user set password 2 Initial1

Test the new settings (using the hostnames we set in /etc/hosts):

an-a04n01

fence_ipmilan -a an-a04n02.ipmi -l admin -p Initial1 -o status

Getting status of IPMI:an-a04n02.ipmi...Chassis power = On
Done

an-a04n02

fence_ipmilan -a an-a04n01.ipmi -l admin -p Initial1 -o status

Getting status of IPMI:an-a04n01.ipmi...Chassis power = On
Done

Good, now we can configure IPMI fencing.

Every fence agent has a possibly unique subset of options that can be used. You can see a brief description of these options with the pcs stonith describe fence_X command. Let's look at the options available for fence_ipmilan.

an-a04n01

pcs stonith describe fence_ipmilan

Stonith options for: fence_ipmilan
  auth: IPMI Lan Auth type (md5, password, or none)
  ipaddr: IPMI Lan IP to talk to
  passwd: Password (if required) to control power on IPMI device
  passwd_script: Script to retrieve password (if required)
  lanplus: Use Lanplus to improve security of connection
  login: Username/Login (if required) to control power on IPMI device
  action: Operation to perform. Valid operations: on, off, reboot, status, list, diag, monitor or metadata
  timeout: Timeout (sec) for IPMI operation
  cipher: Ciphersuite to use (same as ipmitool -C parameter)
  method: Method to fence (onoff or cycle)
  power_wait: Wait X seconds after on/off operation
  delay: Wait X seconds before fencing is started
  privlvl: Privilege level on IPMI device
  verbose: Verbose mode
  stonith-timeout: How long to wait for the STONITH action to complete per a stonith device.
  priority: The priority of the stonith resource. Devices are tried in order of highest priority to lowest.
  pcmk_host_map: A mapping of host names to ports numbers for devices that do not support host names.
  pcmk_host_list: A list of machines controlled by this device (Optional unless pcmk_host_check=static-list).
  pcmk_host_check: How to determin which machines are controlled by the device.

One of the nice things about pcs is that it allows us to create a test file to prepare all our changes in. Then, when we're happy with the changes, merge them into the running cluster. So let's make a copy called stonith_cfg

Now add IPMI fencing.

an-a04n01

pcs cluster cib stonith_cfg
#   work in our temp file         unique name    fence agent   target node                           device addr             options
pcs -f stonith_cfg stonith create fence_n01_ipmi fence_ipmilan pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-a04n01.ipmi" action="reboot" login="admin" passwd="Initial1" delay=15 op monitor interval=10s
pcs -f stonith_cfg stonith create fence_n02_ipmi fence_ipmilan pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-a04n02.ipmi" action="reboot" login="admin" passwd="Initial1" op monitor interval=10s
pcs cluster cib-push stonith_cfg

Note that fence_n01_ipmi has a delay=15 set but fence_n02_ipmi does not. If the network connection breaks between the two nodes, they will both try to fence each other at the same time. If acpid is running, the slower node will not die right away. It will continue to run for up to four more seconds, ample time for it to also initiate a fence against the faster node. The end result is that both nodes get fenced. The ten-second delay protects against this by causing an-a04n02 to pause for 10 seconds before initiating a fence against an-a04n01. If both nodes are alive, an-a04n02 will power off before the 10 seconds pass, so it will never fence an-a04n01. However, if an-a04n01 really is dead, after the ten seconds have elapsed, fencing will proceed as normal.

NOTE: Get my PDUs back and use them here!

We can check the new configuration now;

an-a04n01

pcs status

Cluster name: an-anvil-04
Last updated: Wed May 28 22:01:14 2014
Last change: Wed May 28 21:55:59 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
2 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca 
 fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca

Tell pacemaker to use fencing;

an-a04n01

pcs property set stonith-enabled=true
pcs property set no-quorum-policy=ignore
pcs property

Cluster Properties:
 cluster-infrastructure: cman
 dc-version: 1.1.10-14.el6_5.3-368c726
 no-quorum-policy: ignore
 stonith-enabled: true

Excellent!

Configuring Fence Levels

TODO...

Test Fencing

ToDo: Kill each node with echo c > /proc/sysrq-trigger and make sure the other node fences it.

Shared Storage

DRBD -> Clustered LVM -> GFS2

DRBD

We will use DRBD 8.4.

Partition Storage

How you do this will depend a lot on your storage (local disks, md software RAID, hardware RAID, 1 or multiple arrays, etc). It will also depend on how you plan to divy up your servers; you need two partitions; One for servers that will run on node 1 and another for node 2. It also depends on how much space you want for the /shared partition.

In our case, we're using a single hardware RAID array, we'll set aside 40 GB of space for /shared and we're going to divide the remaining free space evenly.

an-a04n01

parted -a opt /dev/sda "print free"

Model: LSI RAID 5/6 SAS 6G (scsi)
Disk /dev/sda: 898GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system     Flags
        32.3kB  1049kB  1016kB           Free Space
 1      1049kB  538MB   537MB   primary  ext4            boot
 2      538MB   4833MB  4295MB  primary  linux-swap(v1)
 3      4833MB  26.3GB  21.5GB  primary  ext4
        26.3GB  898GB   872GB            Free Space

an-a04n01

# same as an-a04n01

So 872 GB of free space, less 40 for /shared leaves 832 GB for servers. Divided evenly in 2 gives us 416 GB per server pool. Our first partition will then be 446 GB (40 for /shared) and the second will be 416 GB.

The free space starts at 26.3 GB, so our first partition will start at 26.3 GB and end at 492 GB (rounding off the .3). The second partition will then start at 492 GB and end at 898 GB, the end of the disk. Both of these new partitions will be contained in an extended partition.

Note: After each change, we will get an error saying "Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy). As a result, it may not reflect all of your changes until after reboot.". Will reboot once done to address this.

an-a04n01

parted -a opt /dev/sda "mkpart extended 26.3GB 898GB"
parted -a opt /dev/sda "mkpart logical 26.3GB 492GB"
parted -a opt /dev/sda "mkpart logical 492GB 898GB"
parted -a opt /dev/sda "print free"

Model: LSI RAID 5/6 SAS 6G (scsi)
Disk /dev/sda: 898GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type      File system     Flags
        32.3kB  1049kB  1016kB            Free Space
 1      1049kB  538MB   537MB   primary   ext4            boot
 2      538MB   4833MB  4295MB  primary   linux-swap(v1)
 3      4833MB  26.3GB  21.5GB  primary   ext4
 4      26.3GB  898GB   872GB   extended                  lba
 5      26.3GB  492GB   466GB   logical
 6      492GB   898GB   406GB   logical

an-a04n01

# same as an-a04n01

Reboot

an-a04n01	reboot
an-a04n01	reboot

Configure DRBD

Configure global-common.conf;

an-a04n01

vim /etc/drbd.d/global_common.conf

# These are options to set for the DRBD daemon sets the default values for
# resources.
global {
	# This tells DRBD that you allow it to report this installation to 
	# LINBIT for statistical purposes. If you have privacy concerns, set
	# this to 'no'. The default is 'ask' which will prompt you each time
	# DRBD is updated. Set to 'yes' to allow it without being prompted.
	usage-count yes;
 
	# minor-count dialog-refresh disable-ip-verification
}
 
common {
	handlers {
		# pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
		# pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
		# local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
		# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
		# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
		# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
		# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
 
		# Hook into Pacemaker's fencing.
		fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
		before-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
	}
 
	startup {
		# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
	}
 
	options {
		# cpu-mask on-no-data-accessible
	}
 
	disk {
		# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
		# disk-drain md-flushes resync-rate resync-after al-extents
                # c-plan-ahead c-delay-target c-fill-target c-max-rate
                # c-min-rate disk-timeout
                fencing resource-and-stonith;
	}
 
	net {
		# protocol timeout max-epoch-size max-buffers unplug-watermark
		# connect-int ping-int sndbuf-size rcvbuf-size ko-count
		# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
		# after-sb-1pri after-sb-2pri always-asbp rr-conflict
		# ping-timeout data-integrity-alg tcp-cork on-congestion
		# congestion-fill congestion-extents csums-alg verify-alg
		# use-rle
 
		# Protocol "C" tells DRBD not to tell the operating system that
		# the write is complete until the data has reach persistent
		# storage on both nodes. This is the slowest option, but it is
		# also the only one that guarantees consistency between the
		# nodes. It is also required for dual-primary, which we will 
		# be using.
		protocol C;
 
		# Tell DRBD to allow dual-primary. This is needed to enable 
		# live-migration of our servers.
		allow-two-primaries yes;
 
		# This tells DRBD what to do in the case of a split-brain when
		# neither node was primary, when one node was primary and when
		# both nodes are primary. In our case, we'll be running
		# dual-primary, so we can not safely recover automatically. The
		# only safe option is for the nodes to disconnect from one
		# another and let a human decide which node to invalidate. Of 
		after-sb-0pri discard-zero-changes;
		after-sb-1pri discard-secondary;
		after-sb-2pri disconnect;
	}
}

And now configure the first resource;

an-a04n01

vim /etc/drbd.d/r0.res

# This is the first DRBD resource. It will store the shared file systems and
# the servers designed to run on node 01.
resource r0 {
	# These options here are common to both nodes. If for some reason you
	# need to set unique values per node, you can move these to the
	# 'on <name> { ... }' section.
 
	# This sets the device name of this DRBD resouce.
	device /dev/drbd0;
 
	# This tells DRBD what the backing device is for this resource.
	disk /dev/sda5;
 
	# This controls the location of the metadata. When "internal" is used,
	# as we use here, a little space at the end of the backing devices is
	# set aside (roughly 32 MB per 1 TB of raw storage). External metadata
	# can be used to put the metadata on another partition when converting
	# existing file systems to be DRBD backed, when there is no extra space
	# available for the metadata.
	meta-disk internal;
 
	# NOTE: this is not required or even recommended with pacemaker. remove
	# 	this options as soon as pacemaker is setup.
	startup {
		# This tells DRBD to promote both nodes to 'primary' when this
		# resource starts. However, we will let pacemaker control this
		# so we comment it out, which tells DRBD to leave both nodes
		# as secondary when drbd starts.
		#become-primary-on both;
	}
 
	# NOTE: Later, make it an option in the dashboard to trigger a manual
	# 	verify and/or schedule periodic automatic runs
	net {
		# TODO: Test performance differences between sha1 and md5
		# This tells DRBD how to do a block-by-block verification of
		# the data stored on the backing devices. Any verification
		# failures will result in the effected block being marked
		# out-of-sync.
		verify-alg md5;
 
		# TODO: Test the performance hit of this being enabled.
		# This tells DRBD to generate a checksum for each transmitted
		# packet. If the data received data doesn't generate the same
		# sum, a retransmit request is generated. This protects against
		# otherwise-undetected errors in transmission, like 
		# bit-flipping. See:
		# http://www.drbd.org/users-guide/s-integrity-check.html
		data-integrity-alg md5;
	}
 
	# WARNING: Confirm that these are safe when the controller's BBU is
	#          depleted/failed and the controller enters write-through 
	#          mode.
	disk {
		# TODO: Test the real-world performance differences gained with
		#       these options.
		# This tells DRBD not to bypass the write-back caching on the
		# RAID controller. Normally, DRBD forces the data to be flushed
		# to disk, rather than allowing the write-back cachine to 
		# handle it. Normally this is dangerous, but with BBU-backed
		# caching, it is safe. The first option disables disk flushing
		# and the second disabled metadata flushes.
		disk-flushes no;
		md-flushes no;
	}
 
	# This sets up the resource on node 01. The name used below must be the
	# named returned by "uname -n".
	on an-a04n01.alteeve.ca {
		# This is the address and port to use for DRBD traffic on this
		# node. Multiple resources can use the same IP but the ports
		# must differ. By convention, the first resource uses 7788, the
		# second uses 7789 and so on, incrementing by one for each
		# additional resource. 
		address 10.10.40.1:7788;
	}
	on an-a04n02.alteeve.ca {
		address 10.10.40.2:7788;
	}
}

And the second.

an-a04n01

vim /etc/drbd.d/r1.res

# This is the first DRBD resource. It will store the servers designed 
# to run on node 02.
resource r1 {
	device /dev/drbd1;
	disk /dev/sda6;
	meta-disk internal;
 
	net {
		verify-alg md5;
		data-integrity-alg md5;
	}
 
	disk {
		disk-flushes no;
		md-flushes no;
	}
 
	on an-a04n01.alteeve.ca {
		address 10.10.40.1:7789;
	}
	on an-a04n02.alteeve.ca {
		address 10.10.40.2:7789;
	}
}

Test the config;

an-a04n01

drbdadm dump

# /etc/drbd.conf
common {
}

# resource r0 on an-a04n01.alteeve.ca: not ignored, not stacked
# defined at /etc/drbd.d/r0.res:3
resource r0 {
    on an-a04n01.alteeve.ca {
        volume 0 {
            device       /dev/drbd0 minor 0;
            disk         /dev/sda5;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.1:7788;
    }
    on an-a04n02.alteeve.ca {
        volume 0 {
            device       /dev/drbd0 minor 0;
            disk         /dev/sda5;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.2:7788;
    }
    net {
        verify-alg       md5;
        data-integrity-alg md5;
    }
    disk {
        disk-flushes      no;
        md-flushes        no;
    }
}

# resource r1 on an-a04n01.alteeve.ca: not ignored, not stacked
# defined at /etc/drbd.d/r1.res:3
resource r1 {
    on an-a04n01.alteeve.ca {
        volume 0 {
            device       /dev/drbd1 minor 1;
            disk         /dev/sda6;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.1:7789;
    }
    on an-a04n02.alteeve.ca {
        volume 0 {
            device       /dev/drbd1 minor 1;
            disk         /dev/sda6;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.2:7789;
    }
    net {
        verify-alg       md5;
        data-integrity-alg md5;
    }
    disk {
        disk-flushes      no;
        md-flushes        no;
    }
}

Good, copy it to the other node and test it there.

an-a04n01

rsync -av /etc/drbd.* root@an-a04n02:/etc/

sending incremental file list
drbd.d/
drbd.d/global_common.conf
drbd.d/r0.res
drbd.d/r1.res

sent 5738 bytes  received 73 bytes  11622.00 bytes/sec
total size is 5618  speedup is 0.97

an-a04n01

drbdadm dump

# /etc/drbd.conf
common {
}

# resource r0 on an-a04n02.alteeve.ca: not ignored, not stacked
# defined at /etc/drbd.d/r0.res:3
resource r0 {
    on an-a04n01.alteeve.ca {
        volume 0 {
            device       /dev/drbd0 minor 0;
            disk         /dev/sda5;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.1:7788;
    }
    on an-a04n02.alteeve.ca {
        volume 0 {
            device       /dev/drbd0 minor 0;
            disk         /dev/sda5;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.2:7788;
    }
    net {
        verify-alg       md5;
        data-integrity-alg md5;
    }
    disk {
        disk-flushes      no;
        md-flushes        no;
    }
}

# resource r1 on an-a04n02.alteeve.ca: not ignored, not stacked
# defined at /etc/drbd.d/r1.res:3
resource r1 {
    on an-a04n01.alteeve.ca {
        volume 0 {
            device       /dev/drbd1 minor 1;
            disk         /dev/sda6;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.1:7789;
    }
    on an-a04n02.alteeve.ca {
        volume 0 {
            device       /dev/drbd1 minor 1;
            disk         /dev/sda6;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.2:7789;
    }
    net {
        verify-alg       md5;
        data-integrity-alg md5;
    }
    disk {
        disk-flushes      no;
        md-flushes        no;
    }
}

This isn't a plain dump of your configs, you will notice things have been shifted around. The point is that it dumped the configuration without errors, so we're good to go.

Start DRBD for the first time

Load the config;

an-a04n01	modprobe drbd lsmod \| grep drbd drbd 333723 0 libcrc32c 1246 1 drbd
an-a04n01	modprobe drbd lsmod \| grep drbd drbd 333723 0 libcrc32c 1246 1 drbd

Note: If you have used these partitions before, drbd may see an FS and refuse to create the MD. If that happens, use 'dd' to zero out the partition.

Create the metadisk;

an-a04n01

drbdadm create-md r{0,1}

Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success

an-a04n01

drbdadm create-md r{0,1}

Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success

Bring up the new resources.

an-a04n01

drbdadm up r{0,1}
cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
 0: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/Outdated C r----s
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:454762916
 1: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/Outdated C r----s
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396782732

an-a04n01

drbdadm up r{0,1}
cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:454762916
 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396782732

Neither node has data, so we'll arbitrarily force node 01 to become primary, then normally promote node 02 to primary.

an-a04n01

drbdadm primary --force r{0,1}
cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:2136 nr:0 dw:0 dr:2800 al:0 bm:0 lo:0 pe:3 ua:0 ap:0 ep:1 wo:d oos:454760880
        [>....................] sync'ed:  0.1% (444100/444104)M
        finish: 421:04:29 speed: 252 (252) K/sec
 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:24696 nr:0 dw:0 dr:25360 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396758036
        [>....................] sync'ed:  0.1% (387456/387480)M
        finish: 35:33:06 speed: 3,084 (3,084) K/sec

an-a04n01

drbdadm primary r{0,1}
cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
 0: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:859488 dw:859432 dr:608 al:0 bm:52 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:453903484
        [>....................] sync'ed:  0.2% (443264/444104)M
        finish: 71:24:53 speed: 1,752 (4,428) want: 440 K/sec
 1: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:1140588 dw:1140532 dr:608 al:0 bm:69 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:395642200
        [>....................] sync'ed:  0.3% (386368/387480)M
        finish: 70:30:41 speed: 1,548 (5,876) want: 4,400 K/sec

The sync rate starts low, but it will continue to climb, you can keep an eye on it if you wish. DRBD 8.4 is smarter than 8.3 in that it will adjust the sync rate automatically based on load.

We can proceed now, we do not have to wait for the sync to complete.

Clustered LVM and GFS2

Clustered LVM provides the logical volumes that will back our /shared GFS2 partition and the storage for the HA servers.

Configure lvm.conf

Configure clustered LVM.

an-a04n01

sed -i.anvil 's^filter = \[ "a/\.\*/" \]^filter = \[ "a|/dev/drbd*|", "r/.*/" \]^' /etc/lvm/lvm.conf
sed -i 's/locking_type = 1$/locking_type = 3/' /etc/lvm/lvm.conf
sed -i 's/fallback_to_local_locking = 1$/fallback_to_local_locking = 0/' /etc/lvm/lvm.conf 
diff -u /etc/lvm/lvm.conf.anvil /etc/lvm/lvm.conf

--- /etc/lvm/lvm.conf.anvil	2013-10-30 04:10:42.000000000 -0400
+++ /etc/lvm/lvm.conf	2014-06-04 18:38:15.545166869 -0400
@@ -82,7 +82,7 @@
 
 
     # By default we accept every block device:
-    filter = [ "a/.*/" ]
+    filter = [ "a|/dev/drbd*|", "r/.*/" ]
 
     # Exclude the cdrom drive
     # filter = [ "r|/dev/cdrom|" ]
@@ -459,7 +459,7 @@
     # Type 3 uses built-in clustered locking.
     # Type 4 uses read-only locking which forbids any operations that might 
     # change metadata.
-    locking_type = 1
+    locking_type = 3
 
     # Set to 0 to fail when a lock request cannot be satisfied immediately.
     wait_for_locks = 1
@@ -475,7 +475,7 @@
     # to 1 an attempt will be made to use local file-based locking (type 1).
     # If this succeeds, only commands against local volume groups will proceed.
     # Volume Groups marked as clustered will be ignored.
-    fallback_to_local_locking = 1
+    fallback_to_local_locking = 0
 
     # Local non-LV directory that holds file-based locks while commands are
     # in progress.  A directory like /tmp that may get wiped on reboot is OK.

rsync -av /etc/lvm/lvm.conf* root@an-a04n02:/etc/lvm/

sending incremental file list
lvm.conf
lvm.conf.anvil

sent 47499 bytes  received 440 bytes  95878.00 bytes/sec
total size is 89999  speedup is 1.88

an-a04n02

diff -u /etc/lvm/lvm.conf.anvil /etc/lvm/lvm.conf

--- /etc/lvm/lvm.conf.anvil	2013-10-30 04:10:42.000000000 -0400
+++ /etc/lvm/lvm.conf	2014-06-04 18:38:15.000000000 -0400
@@ -82,7 +82,7 @@
 
 
     # By default we accept every block device:
-    filter = [ "a/.*/" ]
+    filter = [ "a|/dev/drbd*|", "r/.*/" ]
 
     # Exclude the cdrom drive
     # filter = [ "r|/dev/cdrom|" ]
@@ -459,7 +459,7 @@
     # Type 3 uses built-in clustered locking.
     # Type 4 uses read-only locking which forbids any operations that might 
     # change metadata.
-    locking_type = 1
+    locking_type = 3
 
     # Set to 0 to fail when a lock request cannot be satisfied immediately.
     wait_for_locks = 1
@@ -475,7 +475,7 @@
     # to 1 an attempt will be made to use local file-based locking (type 1).
     # If this succeeds, only commands against local volume groups will proceed.
     # Volume Groups marked as clustered will be ignored.
-    fallback_to_local_locking = 1
+    fallback_to_local_locking = 0
 
     # Local non-LV directory that holds file-based locks while commands are
     # in progress.  A directory like /tmp that may get wiped on reboot is OK.

Start clvmd

Note: This will be moved to pacemaker shortly. We're enabling it here just long enough to configure pacemaker.

Make sure the cluster is up (you could use 'pcs status', 'cman_tool status', etc):

an-a04n01

dlm_tool dump | grep node

1401921044 cluster node 1 added seq 68
1401921044 set_configfs_node 1 10.20.40.1 local 1
1401921044 cluster node 2 added seq 68
1401921044 set_configfs_node 2 10.20.40.2 local 0
1401921044 run protocol from nodeid 1

Make sure DRBD is up as primary on both nodes:

an-a04n01

cat /proc/drbd

version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
 0: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r-----
    ns:1519672 nr:0 dw:0 dr:1520336 al:0 bm:93 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:448214308
	[>....................] sync'ed:  0.4% (437708/439192)M
	finish: 6:20:02 speed: 19,652 (15,992) K/sec
 1: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r-----
    ns:1896504 nr:0 dw:0 dr:1897168 al:0 bm:115 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:390577164
	[>....................] sync'ed:  0.5% (381420/383272)M
	finish: 2:33:17 speed: 42,440 (19,960) K/sec

Note that we don't have to wait for the sync to finish.

Start clvmd;

an-a04n01	/etc/init.d/clvmd start Starting clvmd: Activating VG(s): No volume groups found [ OK ]
an-a04n02	/etc/init.d/clvmd start Starting clvmd: Activating VG(s): No volume groups found [ OK ]

Note: If this fails, showing a timeout or simply never returning, make sure that TCP port 21064 is opened in your firewall on both nodes.

From here on, pacemaker will start clvmd when pacemaker itself start, *if* clvmd is set to start on boot. So lets set that.

an-a04n01	chkconfig clvmd on chkconfig --list clvmd clvmd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
an-a04n01	chkconfig clvmd on chkconfig --list clvmd clvmd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Create Initial PVs, VGs and the /shared LV

Create the PV, VG and the /shared LV;

an-a04n01

pvcreate /dev/drbd{0,1}

  Physical volume "/dev/drbd0" successfully created
  Physical volume "/dev/drbd1" successfully created

vgcreate an-a04n01_vg0 /dev/drbd0

  Clustered volume group "an-a04n01_vg0" successfully created

vgcreate an-a04n02_vg0 /dev/drbd1

  Clustered volume group "an-a04n02_vg0" successfully created

lvcreate -L 40GiB -n shared an-a04n01_vg0

  Logical volume "shared" created

an-a04n02

pvdisplay

  --- Physical volume ---
  PV Name               /dev/drbd1
  VG Name               an-a04n02_vg0
  PV Size               378.40 GiB / not usable 3.14 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              96870
  Free PE               96870
  Allocated PE          0
  PV UUID               TpEXBC-7822-UGz0-ICz1-AJdg-v5eS-lyB7C5
   
  --- Physical volume ---
  PV Name               /dev/drbd0
  VG Name               an-a04n01_vg0
  PV Size               433.70 GiB / not usable 4.41 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              111025
  Free PE               100785
  Allocated PE          10240
  PV UUID               RoHAJQ-qrsO-Ofwz-f8W7-jIXd-2cvG-oPgfFR

vgdisplay

  --- Volume group ---
  VG Name               an-a04n02_vg0
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  Clustered             yes
  Shared                no
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               378.40 GiB
  PE Size               4.00 MiB
  Total PE              96870
  Alloc PE / Size       0 / 0   
  Free  PE / Size       96870 / 378.40 GiB
  VG UUID               9bTBDu-JSma-kwKR-4oBI-sxi1-YT6i-1uIM4C
   
  --- Volume group ---
  VG Name               an-a04n01_vg0
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  Clustered             yes
  Shared                no
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               433.69 GiB
  PE Size               4.00 MiB
  Total PE              111025
  Alloc PE / Size       10240 / 40.00 GiB
  Free  PE / Size       100785 / 393.69 GiB
  VG UUID               hLnvle-EScm-cP1t-xodO-cKyv-5EyC-TyIpj5

lvdisplay

  --- Logical volume ---
  LV Path                /dev/an-a04n01_vg0/shared
  LV Name                shared
  VG Name                an-a04n01_vg0
  LV UUID                tvolRF-cb3L-29Dn-Vgqd-e4rf-Qq2e-JFIcbA
  LV Write Access        read/write
  LV Creation host, time an-a04n01.alteeve.ca, 2014-06-07 18:54:41 -0400
  LV Status              available
  # open                 0
  LV Size                40.00 GiB
  Current LE             10240
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

Create the /shared GFS2 filesystem

Format the /dev/an-a04n01_vg0/shared logical volume as a GFS2 filesystem;

an-a04n01

mkfs.gfs2 -j 2 -p lock_dlm -t an-anvil-04:shared /dev/an-a04n01_vg0/shared

This will destroy any data on /dev/an-a04n01_vg0/shared.
It appears to contain: symbolic link to `../dm-0'

Are you sure you want to proceed? [y/n] y

Device:                    /dev/an-a04n01_vg0/shared
Blocksize:                 4096
Device Size                40.00 GB (10485760 blocks)
Filesystem Size:           40.00 GB (10485758 blocks)
Journals:                  2
Resource Groups:           160
Locking Protocol:          "lock_dlm"
Lock Table:                "an-anvil-04:shared"
UUID:                      e07d35fe-6860-f790-38cd-af075366c27b

mkdir /shared
mount /dev/an-a04n01_vg0/shared /shared
df -hP

Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           20G  1.5G   18G   8% /
tmpfs                               12G   67M   12G   1% /dev/shm
/dev/sda1                          504M   72M  407M  16% /boot
/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared

an-a04n02

mkdir /shared
mount /dev/an-a04n01_vg0/shared /shared
df -hP

Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           20G  1.5G   18G   8% /
tmpfs                               12G   52M   12G   1% /dev/shm
/dev/sda1                          504M   72M  407M  16% /boot
/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared

Add Storage to Pacemaker

Configure Dual-Primary DRBD

Setup DRBD as a dual-primary resource.

Notes:

Clones allow for a given service to run on multiple nodes.
- master-max is how many copies of the resource can be promoted to master at the same time across the cluster.
- master-node-max is how many copies of the resource can be promoted to master on a given node.
- clone-max is how many copies can run in the cluster, default is to the number of nodes in the cluster.
- clone-node-max is the number of instances of the resource that can run on each node.
- notify controls whether other nodes are notified before and after a resource is started or stopped on a given node.

an-a04n01

pcs cluster cib drbd_cfg
pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 op monitor interval=10s
pcs -f drbd_cfg resource create drbd_r1 ocf:linbit:drbd drbd_resource=r1 op monitor interval=10s
### Ignore this for now.
#pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 \
#                op monitor interval=29s role=Master \
#                op monitor interval=31s role=Slave \
#                op promote interval=0 timeout=90s start-delay=2s \
#                op start interval=0 timeout=240s \
#                op stop interval=0 timeout=120s
pcs -f drbd_cfg resource master drbd_r0_Clone drbd_r0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs -f drbd_cfg resource master drbd_r1_Clone drbd_r1 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs cluster cib-push drbd_cfg

CIB updated

Give it a couple minutes to promote both nodes to Master on both nodes. Initially, it will appear as Master on one node only.

Once updated, you should see this:

an-a04n01

pcs status

Cluster name: an-anvil-04
Last updated: Sat Jun  7 20:29:09 2014
Last change: Sat Jun  7 20:28:36 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
6 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca 
 fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca 
 Master/Slave Set: drbd_r0_Clone [drbd_r0]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Master/Slave Set: drbd_r1_Clone [drbd_r1]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

an-a04n02

pcs status

Cluster name: an-anvil-04
Last updated: Sat Jun  7 20:29:36 2014
Last change: Sat Jun  7 20:28:36 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
6 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca 
 fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca 
 Master/Slave Set: drbd_r0_Clone [drbd_r0]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Master/Slave Set: drbd_r1_Clone [drbd_r1]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Configure LVM

We need to have pacemaker activate our clustered LVM LVs on start, and deactivate them when stopping. We don't start/stop clvmd directly because of stop timing issues that can lead to stray fencing.

Note: This will throw errors if there are no LVs on a given VG... Do not add a volume group until at least one logical volume has been created.

an-a04n01

pcs cluster cib lvm_cfg
pcs -f lvm_cfg resource create lvm_n01_vg0 ocf:heartbeat:lvm volgrpname=an-a04n01_vg0 op monitor interval=10s
pcs -f lvm_cfg resource master lvm_n01_vg0_Clone lvm_n01_vg0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs cluster cib-push lvm_cfg

CIB updated

Configure LVM to start after the DRBD PV is Primary

It we stopped here, there is a good chance that on future starts of pacemaker, LVM and DRBD would start in parallel, DRBD would take too long, LVM would error out and stonith's would start to fly. To prevent this, we will tell Pacemaker not to start the LVM resource until after the DRBD resource that is behind the volume group has been promoted to primary.

an-a04n01

pcs cluster cib cst_cfg 
pcs -f cst_cfg constraint order promote drbd_r0_Clone then start lvm_n01_vg0_Clone

Adding drbd_r0_Clone lvm_n01_vg0_Clone (kind: Mandatory) (Options: first-action=promote then-action=start)

pcs cluster cib-push cst_cfg

CIB updated

pcs constraint show

Location Constraints:
Ordering Constraints:
  promote drbd_r0_Clone then start lvm_n01_vg0_Clone
Colocation Constraints:

Configure the /shared GFS2 Partition

an-a04n01

pcs cluster cib fs_cfg
pcs -f fs_cfg resource create sharedFS Filesystem device="/dev/an-a04n01_vg0/shared" directory="/shared" fstype="gfs2"
pcs -f fs_cfg resource clone sharedFS master-max=2 master-node-max=1 clone-max=2 clone-node-max=1
pcs cluster cib-push fs_cfg

CIB updated

pcs status

Cluster name: an-anvil-04
Last updated: Sat Jun  7 21:09:28 2014
Last change: Sat Jun  7 21:08:47 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
8 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca 
 fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca 
 Master/Slave Set: drbd_r0_Clone [drbd_r0]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Master/Slave Set: drbd_r1_Clone [drbd_r1]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Clone Set: sharedFS-clone [sharedFS]
     Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

df -hP

Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           20G  1.5G   18G   8% /
tmpfs                               12G   67M   12G   1% /dev/shm
/dev/sda1                          504M   72M  407M  16% /boot
/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared

an-a04n02

df -h

Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           20G  1.5G   18G   8% /
tmpfs                               12G   52M   12G   1% /dev/shm
/dev/sda1                          504M   72M  407M  16% /boot
/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared

Configur /shared to start after LVM

As we did before in making sure LVM started after DRBD, this time we will make sure LVM starts before /shared is mounted.

an-a04n01

pcs cluster cib cst_cfg
pcs -f cst_cfg constraint order start lvm_n01_vg0_Clone then start sharedFS

Adding lvm_n01_vg0_Clone sharedFS (kind: Mandatory) (Options: first-action=start then-action=start)

pcs cluster cib-push cst_cfg

CIB updated

pcs constraint show --full

Location Constraints:
Ordering Constraints:
  promote drbd_r0_Clone then start lvm_n01_vg0_Clone (Mandatory) (id:order-drbd_r0_Clone-lvm_n01_vg0_Clone-mandatory)
  start lvm_n01_vg0_Clone then start sharedFS-clone (Mandatory) (id:order-lvm_n01_vg0_Clone-sharedFS-clone-mandatory)
Colocation Constraints:

Note that this time we added '--full'. If you ever need to delete a constraint, you would use 'pcs constraint delete <id>'.

Notes

Pacemaker Logging

Thanks

This list will certainly grow as this tutorial progresses;

Olivier Allart, RCHE for doing a lot of the heavy lifting on the fencing_topology configuration.

Any questions, feedback, advice, complaints or meanderings are welcome.
`Alteeve's Niche!`	`Enterprise Support: Alteeve Support`	`Community Support`
© Alteeve's Niche! Inc. 1997-2024		Anvil! "Intelligent Availability®" Platform
`legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.`

@@ Line 43: / Line 43: @@
 AN! offers a new repo with a few RPMs not in stock EL 6 distros.
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+!<span class="code">an-a04n02</span>
+|-
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+cat <<-END>/etc/yum.repos.d/an.repo
+[an-repo]
+name=AN! Repo for Anvil! stuff
+baseurl=https://alteeve.ca/repo/el6/
+enabled=1
+gpgcheck=0
+protect=1
+END
+</syntaxhighlight>
 <syntaxhighlight lang="bash">
-vi /etc/yum.repos.d/an.repo
+yum clean all
 </syntaxhighlight>
-<syntaxhighlight lang="text">
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+cat <<-END>/etc/yum.repos.d/an.repo
 [an-repo]
 name=AN! Repo for Anvil! stuff
@@ Line 53: / Line 68: @@
 gpgcheck=0
 protect=1
+END
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
 yum clean all
 </syntaxhighlight>
-<syntaxhighlight lang="text">
+|}
-</syntaxhighlight>
 Done.
@@ Line 69: / Line 83: @@
 {{note|1=The <span class="code">fence-agents-virsh</span> package is not available in RHEL 7 beta. Further, it's only needed if you're building your Anvil! using VMs.}}
-<syntaxhighlight lang="bash">
+{|class="wikitable"
-yum install bridge-utils
+!<span class="code">an-a04n01</span>
+!<span class="code">an-a04n02</span>
+|-
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+yum -y update
+yum -y install bridge-utils vim pacemaker corosync cman gfs2-utils \
+               ccs pcs ipmitool OpenIPMI lvm2-cluster drbd84-utils \
+               drbd84-kmod
+chkconfig ipmi on
+chkconfig acpid off
+chkconfig kdump off
+chkconfig drbd off
+/etc/init.d/ipmi start
+/etc/init.d/acpid stop
+/etc/init.d/kdump stop
+/etc/init.d/drbd stop
+</syntaxhighlight>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+# same as an-a04n01
 </syntaxhighlight>
+|}
 === Setup Networking ===
@@ Line 77: / Line 110: @@
 TODO: Explain this.
+Remap all NICs to have purpose-based names.
+* [[IFN]] Bridge
-=== Making ssh faster when the net is down ===
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
-By default, the nodes will try to resolve the host name of an incoming ssh connection. When the internet connection is down, DNS lookups have to time out, which can make login times quite slow. When something goes wrong, seconds count and waiting for up to a minute for an SSH password prompt can be maddening.
+!<span class="code">an-a04n02</span>
+|-
-For this reason, we will make two changes to <span class="code">/etc/ssh/sshd_config</span> that disable this login delay.
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/sysconfig/network-scripts/ifcfg-ifn-bridge1
-Please be aware that this can reduce security. If this is a concern, skip this step.
+</syntaxhighlight>
 <syntaxhighlight lang="bash">
-sed -i.anvil 's/#GSSAPIAuthentication no/GSSAPIAuthentication no/' /etc/ssh/sshd_config
+# Internet-Facing Network - Bridge
-sed -i 's/GSSAPIAuthentication yes/#GSSAPIAuthentication yes/' /etc/ssh/sshd_config
+DEVICE="ifn-bridge1"
-sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
+TYPE="Bridge"
-systemctl restart sshd.service
+BOOTPROTO="none"
-diff -u /etc/ssh/sshd_config.anvil /etc/ssh/sshd_config
+IPADDR="10.255.40.1"
+NETMASK="255.255.0.0"
+GATEWAY="10.255.255.254"
+DNS1="8.8.8.8"
+DNS2="8.8.4.4"
+DEFROUTE="yes"
 </syntaxhighlight>
-<syntaxhighlight lang="diff">
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
---- /etc/ssh/sshd_config.anvil	2013-11-08 09:17:23.000000000 -0500
+vim /etc/sysconfig/network-scripts/ifcfg-ifn-bridge1
-+++ /etc/ssh/sshd_config	2014-04-03 00:01:40.980951975 -0400
-@@ -89,8 +89,8 @@
- #KerberosUseKuserok yes
- # GSSAPI options
--#GSSAPIAuthentication no
--GSSAPIAuthentication yes
-+GSSAPIAuthentication no
-+#GSSAPIAuthentication yes
- #GSSAPICleanupCredentials yes
- GSSAPICleanupCredentials yes
- #GSSAPIStrictAcceptorCheck yes
-@@ -127,7 +127,7 @@
- #ClientAliveInterval 0
- #ClientAliveCountMax 3
- #ShowPatchLevel no
--#UseDNS yes
-+UseDNS no
- #PidFile /var/run/sshd.pid
- #MaxStartups 10:30:100
- #PermitTunnel no
 </syntaxhighlight>
-Subsequent logins when the net is down should be quick.
-=== Configuring the network ===
-<span class="code"></span>
 <syntaxhighlight lang="bash">
+# Internet-Facing Network - Bridge
+DEVICE="ifn-bridge1"
+TYPE="Bridge"
+BOOTPROTO="none"
+IPADDR="10.255.40.2"
+NETMASK="255.255.0.0"
+GATEWAY="10.255.255.254"
+DNS1="8.8.8.8"
+DNS2="8.8.4.4"
+DEFROUTE="yes"
 </syntaxhighlight>
+|}
+* [[IFN]] Bond
-Enable the <span class="code">eth0</span> interface on boot.
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/sysconfig/network-scripts/ifcfg-ifn-bond1
+</syntaxhighlight>
 <syntaxhighlight lang="bash">
-sed -i.bak 's/ONBOOT=.*/ONBOOT="yes"/' /etc/sysconfig/network-scripts/ifcfg-eth0
+# Internet-Facing Network - Bond
-diff -U0 /etc/sysconfig/network-scripts/ifcfg-eth0.bak /etc/sysconfig/network-scripts/ifcfg-eth0
+DEVICE="ifn-bond1"
+BRIDGE="ifn-bridge1"
+NM_CONTROLLED="no"
+BOOTPROTO="none"
+ONBOOT="yes"
+BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=ifn-link1"
 </syntaxhighlight>
-<syntaxhighlight lang="diff">
+|-
---- /etc/sysconfig/network-scripts/ifcfg-eth0.bak	2014-01-23 16:15:45.008085032 -0500
+!<span class="code">an-a04n02</span>
-+++ /etc/sysconfig/network-scripts/ifcfg-eth0	2014-01-23 16:15:25.573009623 -0500
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-@@ -11 +11 @@
+vim /etc/sysconfig/network-scripts/ifcfg-ifn-bond1
--ONBOOT=no
-+ONBOOT="yes"
 </syntaxhighlight>
-If you want to make any other changes, like configuring the interface to have a static IP, do so now. Once you're done editing;
 <syntaxhighlight lang="bash">
-nmcli connection reload
+# Internet-Facing Network - Bond
-systemctl restart NetworkManager.service
+DEVICE="ifn-bond1"
-ip addr show
+BRIDGE="ifn-bridge1"
-</syntaxhighlight>
+NM_CONTROLLED="no"
-<syntaxhighlight lang="text">
+BOOTPROTO="none"
-: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
+ONBOOT="yes"
-    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=ifn-link1"
-    inet 127.0.0.1/8 scope host lo
-       valid_lft forever preferred_lft forever
-    inet6 ::1/128 scope host
-       valid_lft forever preferred_lft forever
-: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
-    link/ether 52:54:00:a7:9d:17 brd ff:ff:ff:ff:ff:ff
-    inet 192.168.122.201/24 scope global eth0
-       valid_lft forever preferred_lft forever
-    inet6 fe80::5054:ff:fea7:9d17/64 scope link
-       valid_lft forever preferred_lft forever
 </syntaxhighlight>
+|}
-The interface should now start on boot properly.
+* [[IFN]] Links
-== Setting the Hostname ==
-Fedora 19 is '''very''' different from [[EL6]].
-{{note|1=The '<span class="code">--pretty</span>' line currently doesn't work as there is [https://bugzilla.redhat.com/show_bug.cgi?id=895299 a bug (rhbz#895299)] with single-quotes.}}
-{{note|1=The '<span class="code">--static</span>' option is currently needed to prevent the '<span class="code">.</span>' from being removed. See [https://bugzilla.redhat.com/show_bug.cgi?id=896756 this bug (rhbz#896756)].}}
-Use a format that works for you. For the tutorial, node names are based on the following;
-* A two-letter prefix identifying the company/user (<span class="code">an</span>, for "Alteeve's Niche!")
-* A sequential Anvil! ID number in the form of <span class="code">aXX</span> (<span class="code">a01</span> for "Anvil! 01", <span class="code">a02</span> for Anvil! 02, etc)
-* A sequential node ID number in the form of <span class="code">nYY</span>
-In our case, this is my third Anvil! and we use the company prefix <span class="code">an</span>, so these two nodes will be;
-* <span class="code">an-a03n01</span> - node 1
-* <span class="code">an-a03n02</span> - node 2
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+!<span class="code">an-a04n02</span>
+|-
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/sysconfig/network-scripts/ifcfg-ifn-link1
+</syntaxhighlight>
 <syntaxhighlight lang="bash">
-hostnamectl set-hostname an-a03n01.alteeve.ca --static
+# Internet-Facing Network - Link 1
-hostnamectl set-hostname --pretty "Alteeve's Niche! - Anvil! 03, Node 01"
+HWADDR="00:1B:21:81:C3:34"
+DEVICE="ifn-link1"
+NM_CONTROLLED="no"
+BOOTPROTO="none"
+ONBOOT="yes"
+MASTER="ifn-bond1"
+SLAVE="yes"
 </syntaxhighlight>
-If you want the new host name to take effect immediately, you can use the traditional <span class="code">hostname</span> command:
 <syntaxhighlight lang="bash">
-hostname an-a03n01.alteeve.ca
+vim /etc/sysconfig/network-scripts/ifcfg-ifn-link2
 </syntaxhighlight>
-The "pretty" host name is stored in <span class="code">/etc/machine-info</span> as the unquoted value for the <span class="code">PRETTY_HOSTNAME</span> value.
 <syntaxhighlight lang="bash">
-vim /etc/machine-info
+# Internet-Facing Network - Link 2
+HWADDR="A0:36:9F:02:E0:05"
+DEVICE="ifn-link2"
+NM_CONTROLLED="no"
+BOOTPROTO="none"
+ONBOOT="yes"
+MASTER="ifn-bond1"
+SLAVE="yes"
+</syntaxhighlight>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/sysconfig/network-scripts/ifcfg-ifn-link1
 </syntaxhighlight>
-<syntaxhighlight lang="text">
+<syntaxhighlight lang="bash">
-PRETTY_HOSTNAME=Alteeves Niche! - Anvil! 01, Node 01
+# Internet-Facing Network - Link 1
+HWADDR="00:1B:21:81:C2:EA"
+DEVICE="ifn-link1"
+NM_CONTROLLED="no"
+BOOTPROTO="none"
+ONBOOT="yes"
+MASTER="ifn-bond1"
+SLAVE="yes"
 </syntaxhighlight>
-If you can't get the <span class="code">hostname</span> command to work for some reason, you can reboot to have the system read the new values.
-== Network ==
-We want static, named network devices. Follow this;
-* [[Changing Ethernet Device Names in EL7 and Fedora 15+]]
-Then, use these configuration files;
-Build the bridge;
 <syntaxhighlight lang="bash">
-vim /etc/sysconfig/network-scripts/ifcfg-ifn-vbr1
+vim /etc/sysconfig/network-scripts/ifcfg-ifn-link2
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-# Internet-Facing Network - Bridge
+# Internet-Facing Network - Link 2
-DEVICE="ifn-vbr1"
+HWADDR="A0:36:9F:07:D6:2F"
-TYPE="Bridge"
+DEVICE="ifn-link2"
+NM_CONTROLLED="no"
 BOOTPROTO="none"
-IPADDR="10.255.10.1"
+ONBOOT="yes"
-NETMASK="255.255.0.0"
+MASTER="ifn-bond1"
-GATEWAY="10.255.255.254"
+SLAVE="yes"
-DNS1="8.8.8.8"
-DNS2="8.8.4.4"
-DEFROUTE="yes"
 </syntaxhighlight>
+|}
-Now build the bonds;
+* [[SN]] Bond
-<syntaxhighlight lang="bash">
+{|class="wikitable"
-vim /etc/sysconfig/network-scripts/ifcfg-ifn-bond1
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/sysconfig/network-scripts/ifcfg-sn-bond1
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-# Internet-Facing Network - Bond
+# Storage Network - Bond
-DEVICE="ifn-bond1"
+DEVICE="sn-bond1"
-BRIDGE="ifn-vbr1"
+NM_CONTROLLED="no"
 BOOTPROTO="none"
-NM_CONTROLLED="no"
 ONBOOT="yes"
-BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=ifn1"
+BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=sn-link1"
+IPADDR="10.10.40.1"
+NETMASK="255.255.0.0"
 </syntaxhighlight>
+|-
-<syntaxhighlight lang="bash">
+!<span class="code">an-a04n02</span>
-vim /etc/sysconfig/network-scripts/ifcfg-sn-bond1
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/sysconfig/network-scripts/ifcfg-sn-bond1
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
 # Storage Network - Bond
 DEVICE="sn-bond1"
+NM_CONTROLLED="no"
 BOOTPROTO="none"
-NM_CONTROLLED="no"
 ONBOOT="yes"
-BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=sn1"
+BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=sn-link1"
-IPADDR="10.10.10.1"
+IPADDR="10.10.40.2"
 NETMASK="255.255.0.0"
 </syntaxhighlight>
+|}
+* [[SN]] Links
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+!<span class="code">an-a04n02</span>
+|-
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/sysconfig/network-scripts/ifcfg-sn-link1
+</syntaxhighlight>
 <syntaxhighlight lang="bash">
-vim /etc/sysconfig/network-scripts/ifcfg-bcn-bond1
+# Storage Network - Link 1
+HWADDR="00:19:99:9C:9B:9F"
+DEVICE="sn-link1"
+NM_CONTROLLED="no"
+BOOTPROTO="none"
+ONBOOT="yes"
+MASTER="sn-bond1"
+SLAVE="yes"
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-# Back-Channel Network - Bond
+vim /etc/sysconfig/network-scripts/ifcfg-sn-link2
-DEVICE="bcn-bond1"
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+# Storage Network - Link 2
+HWADDR="A0:36:9F:02:E0:04"
+DEVICE="sn-link2"
+NM_CONTROLLED="no"
 BOOTPROTO="none"
+ONBOOT="yes"
+MASTER="sn-bond1"
+SLAVE="yes"
+</syntaxhighlight>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/sysconfig/network-scripts/ifcfg-sn-link1
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+# Storage Network - Link 1
+HWADDR="00:19:99:9C:A0:6D"
+DEVICE="sn-link1"
 NM_CONTROLLED="no"
+BOOTPROTO="none"
 ONBOOT="yes"
-BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=bcn1"
+MASTER="sn-bond1"
-IPADDR="10.20.10.1"
+SLAVE="yes"
-NETMASK="255.255.0.0"
 </syntaxhighlight>
-Now tell the interfaces to be slaves to their bonds;
-Internet-Facing Network;
 <syntaxhighlight lang="bash">
-vim /etc/sysconfig/network-scripts/ifcfg-ifn1
+vim /etc/sysconfig/network-scripts/ifcfg-sn-link2
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-# Internet-Facing Network - Link 1
+# Storage Network - Link 2
-DEVICE="ifn1"
+HWADDR="A0:36:9F:07:D6:2E"
+DEVICE="sn-link2"
 NM_CONTROLLED="no"
 BOOTPROTO="none"
 ONBOOT="yes"
+MASTER="sn-bond1"
 SLAVE="yes"
-MASTER="ifn-bond1"
 </syntaxhighlight>
+|}
+* [[BCN]] Bond
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/sysconfig/network-scripts/ifcfg-bcn-bond1
+</syntaxhighlight>
 <syntaxhighlight lang="bash">
-vim /etc/sysconfig/network-scripts/ifcfg-ifn2
+# Back-Channel Network - Bond
+DEVICE="bcn-bond1"
+NM_CONTROLLED="no"
+BOOTPROTO="none"
+ONBOOT="yes"
+BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=bcn-link1"
+IPADDR="10.20.40.1"
+NETMASK="255.255.0.0"
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n02</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/sysconfig/network-scripts/ifcfg-bcn-bond1
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-# Back-Channel Network - Link 2
+# Back-Channel Network - Bond
-DEVICE="ifn2"
+DEVICE="bcn-bond1"
 NM_CONTROLLED="no"
 BOOTPROTO="none"
 ONBOOT="yes"
-SLAVE="yes"
+BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=bcn-link1"
-MASTER="ifn-bond1"
+IPADDR="10.20.40.2"
+NETMASK="255.255.0.0"
 </syntaxhighlight>
+|}
-Storage Network;
+* [[BCN]] Links
-<syntaxhighlight lang="bash">
+{|class="wikitable"
-vim /etc/sysconfig/network-scripts/ifcfg-sn1
+!<span class="code">an-a04n01</span>
+!<span class="code">an-a04n02</span>
+|-
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/sysconfig/network-scripts/ifcfg-bcn-link1
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-# Storage Network - Link 1
+# Back-Channel Network - Link 1
-DEVICE="sn1"
+HWADDR="00:19:99:9C:9B:9E"
+DEVICE="bcn-link1"
 NM_CONTROLLED="no"
 BOOTPROTO="none"
 ONBOOT="yes"
+MASTER="bcn-bond1"
 SLAVE="yes"
-MASTER="sn-bond1"
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-vim /etc/sysconfig/network-scripts/ifcfg-sn2
+vim /etc/sysconfig/network-scripts/ifcfg-bcn-link2
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-# Storage Network - Link 1
+# Back-Channel Network - Link 2
-DEVICE="sn2"
+HWADDR="00:1B:21:81:C3:35"
+DEVICE="bcn-link2"
 NM_CONTROLLED="no"
 BOOTPROTO="none"
 ONBOOT="yes"
+MASTER="bcn-bond1"
 SLAVE="yes"
-MASTER="sn-bond1"
 </syntaxhighlight>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-Back-Channel Network
+vim /etc/sysconfig/network-scripts/ifcfg-bcn-link1
-<syntaxhighlight lang="bash">
-vim /etc/sysconfig/network-scripts/ifcfg-bcn1
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
 # Back-Channel Network - Link 1
-DEVICE="bcn1"
+HWADDR="00:19:99:9C:A0:6C"
+DEVICE="bcn-link1"
 NM_CONTROLLED="no"
 BOOTPROTO="none"
 ONBOOT="yes"
+MASTER="bcn-bond1"
 SLAVE="yes"
-MASTER="bcn-bond1"
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-vim /etc/sysconfig/network-scripts/ifcfg-bcn2
+vim /etc/sysconfig/network-scripts/ifcfg-bcn-link2
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-# Storage Network - Link 1
+# Back-Channel Network - Link 2
-DEVICE="bcn2"
+HWADDR="00:1B:21:81:C2:EB"
+DEVICE="bcn-link2"
 NM_CONTROLLED="no"
 BOOTPROTO="none"
 ONBOOT="yes"
+MASTER="bcn-bond1"
 SLAVE="yes"
-MASTER="bcn-bond1"
 </syntaxhighlight>
+|}
+=== Making ssh faster when the net is down ===
+By default, the nodes will try to resolve the host name of an incoming ssh connection. When the internet connection is down, DNS lookups have to time out, which can make login times quite slow. When something goes wrong, seconds count and waiting for up to a minute for an SSH password prompt can be maddening.
+For this reason, we will make two changes to <span class="code">/etc/ssh/sshd_config</span> that disable this login delay.
+Please be aware that this can reduce security. If this is a concern, skip this step.
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+sed -i.anvil 's/#GSSAPIAuthentication no/GSSAPIAuthentication no/' /etc/ssh/sshd_config
+sed -i 's/GSSAPIAuthentication yes/#GSSAPIAuthentication yes/' /etc/ssh/sshd_config
+sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
+systemctl restart sshd.service
+diff -u /etc/ssh/sshd_config.anvil /etc/ssh/sshd_config
+</syntaxhighlight>
+<syntaxhighlight lang="diff">
+--- /etc/ssh/sshd_config.anvil	2013-09-30 03:08:17.000000000 -0400
++++ /etc/ssh/sshd_config	2014-05-28 00:35:30.954000741 -0400
+@@ -77,8 +77,8 @@
+ #KerberosUseKuserok yes
+ # GSSAPI options
+-#GSSAPIAuthentication no
+-GSSAPIAuthentication yes
++GSSAPIAuthentication no
++#GSSAPIAuthentication yes
+ #GSSAPICleanupCredentials yes
+ GSSAPICleanupCredentials yes
+ #GSSAPIStrictAcceptorCheck yes
+@@ -119,7 +119,7 @@
+ #ClientAliveInterval 0
+ #ClientAliveCountMax 3
+ #ShowPatchLevel no
+-#UseDNS yes
++UseDNS no
+ #PidFile /var/run/sshd.pid
+ #MaxStartups 10:30:100
+ #PermitTunnel no
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n02</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+sed -i.anvil 's/#GSSAPIAuthentication no/GSSAPIAuthentication no/' /etc/ssh/sshd_config
+sed -i 's/GSSAPIAuthentication yes/#GSSAPIAuthentication yes/' /etc/ssh/sshd_config
+sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
+systemctl restart sshd.service
+diff -u /etc/ssh/sshd_config.anvil /etc/ssh/sshd_config
+</syntaxhighlight>
+<syntaxhighlight lang="diff">
+--- /etc/ssh/sshd_config.anvil	2013-09-30 03:08:17.000000000 -0400
++++ /etc/ssh/sshd_config	2014-05-28 00:35:33.016999110 -0400
+@@ -77,8 +77,8 @@
+ #KerberosUseKuserok yes
+ # GSSAPI options
+-#GSSAPIAuthentication no
+-GSSAPIAuthentication yes
++GSSAPIAuthentication no
++#GSSAPIAuthentication yes
+ #GSSAPICleanupCredentials yes
+ GSSAPICleanupCredentials yes
+ #GSSAPIStrictAcceptorCheck yes
+@@ -119,7 +119,7 @@
+ #ClientAliveInterval 0
+ #ClientAliveCountMax 3
+ #ShowPatchLevel no
+-#UseDNS yes
++UseDNS no
+ #PidFile /var/run/sshd.pid
+ #MaxStartups 10:30:100
+ #PermitTunnel no
+</syntaxhighlight>
+|}
+Subsequent logins when the net is down should be quick.
+== Setting the Hostname ==
-Now restart the network, confirm that the bonds and bridge are up and you are ready to proceed.
+TODO
 == Setup The hosts File ==
@@ Line 365: / Line 517: @@
 You can use [[DNS]] if you prefer. For now, lets use <span class="code">/etc/hosts</span> for node name resolution.
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/hosts
+</syntaxhighlight>
 <syntaxhighlight lang="bash">
+.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
+::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
+### Anvil! systems
+# Anvil! 03, Node 01
+.20.40.1	an-a04n01.bcn an-a04n01 an-a04n01.alteeve.ca
+.20.41.1	an-a04n01.ipmi
+.10.40.1	an-a04n01.sn
+.255.40.1	an-a04n01.ifn
+# Anvil! 03, Node 02
+.20.40.2	an-a04n02.bcn an-a04n02 an-a04n02.alteeve.ca
+.20.41.2	an-a04n02.ipmi
+.10.40.2	an-a04n02.sn
+.255.40.2	an-a04n02.ifn
+### Foundation Pack
+# Network Switches
+.20.1.1	an-s01 an-s01.alteeve.ca
+.20.1.2	an-s02 an-s02.alteeve.ca	# Only accessible when out of the stack
+# Switched PDUs
+.20.2.1	an-p01 an-p01.alteeve.ca
+.20.2.2	an-p02 an-p02.alteeve.ca
+# Network-monitored UPSes
+.20.3.1	an-u01 an-u01.alteeve.ca
+.20.3.2	an-u02 an-u02.alteeve.ca
+### Monitor Packs
+.20.4.1	an-m01 an-m01.alteeve.ca
+.255.4.1	an-m01.ifn
+.20.4.2	an-m02 an-m02.alteeve.ca
+.255.4.2	an-m02.ifn
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n02</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 vim /etc/hosts
 </syntaxhighlight>
-<syntaxhighlight lang="text">
+<syntaxhighlight lang="bash">
 .0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
 ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
+### Anvil! systems
 # Anvil! 03, Node 01
-.255.30.1     an-a03n01.ifn
+.20.40.1	an-a04n01.bcn an-a04n01 an-a04n01.alteeve.ca
-.10.30.1      an-a03n01.sn
+.20.41.1	an-a04n01.ipmi
-.20.30.1      an-a03n01.bcn an-a03n01 an-a03n01.alteeve.ca
+.10.40.1	an-a04n01.sn
-.20.31.1      an-a03n01.ipmi
+.255.40.1	an-a04n01.ifn
 # Anvil! 03, Node 02
-.255.30.2     an-a03n02.ifn
+.20.40.2	an-a04n02.bcn an-a04n02 an-a04n02.alteeve.ca
-.10.30.2      an-a03n02.sn
+.20.41.2	an-a04n02.ipmi
-.20.30.2      an-a03n02.bcn an-a03n02 an-a03n02.alteeve.ca
+.10.40.2	an-a04n02.sn
-.20.31.2      an-a03n02.ipmi
+.255.40.2	an-a04n02.ifn
-# Foundation Pack
+### Foundation Pack
-.20.2.7       an-p03 an-p03.alteeve.ca
+# Network Switches
+.20.1.1	an-s01 an-s01.alteeve.ca
+.20.1.2	an-s02 an-s02.alteeve.ca	# Only accessible when out of the stack
+# Switched PDUs
+.20.2.1	an-p01 an-p01.alteeve.ca
+.20.2.2	an-p02 an-p02.alteeve.ca
+# Network-monitored UPSes
+.20.3.1	an-u01 an-u01.alteeve.ca
+.20.3.2	an-u02 an-u02.alteeve.ca
+### Monitor Packs
+.20.4.1	an-m01 an-m01.alteeve.ca
+.255.4.1	an-m01.ifn
+.20.4.2	an-m02 an-m02.alteeve.ca
+.255.4.2	an-m02.ifn
 </syntaxhighlight>
+|}
 == Setup SSH ==
@@ Line 396: / Line 609: @@
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 ssh-keygen -t rsa -N "" -b 8191 -f ~/.ssh/id_rsa
@@ Line 402: / Line 615: @@
 <syntaxhighlight lang="text">
 Generating public/private rsa key pair.
 Your identification has been saved in /root/.ssh/id_rsa.
 Your public key has been saved in /root/.ssh/id_rsa.pub.
 The key fingerprint is:
-be:17:cc:23:8e:b1:b4:76:a1:e4:2a:91:cb:cd:d8:3a root@an-a03n01.alteeve.ca
+f9:41:7e:aa:96:8e:fa:47:79:f5:3a:33:89:c3:9a:4b root@an-a04n01.alteeve.ca
 The key's randomart image is:
 +--[ RSA 8191]----+
 |                 |
 |                 |
-|                 |
+|          .      |
-|                 |
+|         +  .    |
-|   .    So       |
+|        S.o...   |
-|  o   +.o =      |
+|        o..+  .  |
-| . B + B.o o     |
+|       .E+o. o   |
-|  E + B o..      |
+|       o+o+ *    |
-|  .+.o ...       |
+|    .oo+*o . +   |
 +-----------------+
 </syntaxhighlight>
 |-
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 ssh-keygen -t rsa -N "" -b 8191 -f ~/.ssh/id_rsa
@@ Line 431: / Line 643: @@
 Your public key has been saved in /root/.ssh/id_rsa.pub.
 The key fingerprint is:
-:b1:9d:31:9f:7a:c9:10:74:e0:4c:69:53:8f:e4:70 root@an-a03n02.alteeve.ca
+f:1a:02:17:44:10:5e:6f:2b:98:44:09:e5:e0:ea:4b root@an-a04n02.alteeve.ca
 The key's randomart image is:
 +--[ RSA 8191]----+
-|          ..O+E  |
+|  oo==+          |
-|           B+% + |
+| . =.o .         |
-|        . o.*.= .|
+|  . + . o        |
-|         o   + . |
+| . . o o .       |
-|        S   . +  |
+|.   + o S        |
-|             .   |
+|.    o . .       |
-|                 |
+| E    . . o      |
-|                 |
+|. .    . o .     |
-|                 |
+| .      .        |
 +-----------------+
 </syntaxhighlight>
@@ Line 450: / Line 662: @@
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
-ssh root@an-a03n02 "cat /root/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys
+ssh root@an-a04n02 "cat /root/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys
-rsync -av ~/.ssh/authorized_keys root@an-a03n02:/root/.ssh/
+</syntaxhighlight>
-ssh root@an-a03n01
+<syntaxhighlight lang="text">
-ssh root@an-a03n01.alteeve.ca
+The authenticity of host 'an-a04n02 (10.20.40.2)' can't be established.
-ssh root@an-a03n02
+RSA key fingerprint is 22:09:7b:0c:8b:d8:80:08:80:6d:0e:bc:fb:5a:e1:de.
-ssh root@an-a03n02.alteeve.ca
+Are you sure you want to continue connecting (yes/no)? yes
-rsync -av ~/.ssh/known_hosts root@an-a03n02:/root/.ssh/
+</syntaxhighlight>
-rsync -av /etc/hosts root@an-a03n02:/etc/
+<syntaxhighlight lang="text">
+Warning: Permanently added 'an-a04n02,10.20.40.2' (RSA) to the list of known hosts.
 </syntaxhighlight>
-|-
+<syntaxhighlight lang="text">
-!<span class="code">an-a03n01</span>
+root@an-a04n02's password:
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-ssh root@an-a03n01
 </syntaxhighlight>
 |}
-== Keeping Time in Sync ==
+Populate <span class="code">~/.ssh/known_hosts</span>:
-It's not as critical as it used to be to keep the clocks on the nodes in sync, but it's still a good idea.
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+ssh-keyscan an-a04n01.alteeve.ca >> ~/.ssh/known_hosts
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+# an-a04n01.alteeve.ca SSH-2.0-OpenSSH_5.3
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+ssh-keyscan an-a04n01 >> ~/.ssh/known_hosts
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+# an-a04n01 SSH-2.0-OpenSSH_5.3
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+ssh-keyscan an-a04n01.bcn >> ~/.ssh/known_hosts
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+# an-a04n01.bcn SSH-2.0-OpenSSH_5.3
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+ssh-keyscan an-a04n01.sn >> ~/.ssh/known_hosts
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+# an-a04n01.sn SSH-2.0-OpenSSH_5.3
+</syntaxhighlight>
 <syntaxhighlight lang="bash">
-ln -sf /usr/share/zoneinfo/America/Toronto /etc/localtime
+ssh-keyscan an-a04n01.ifn >> ~/.ssh/known_hosts
-systemctl start ntpd.service
+</syntaxhighlight>
-systemctl enable ntpd.service
+<syntaxhighlight lang="text">
+# an-a04n01.ifn SSH-2.0-OpenSSH_5.3
 </syntaxhighlight>
-== Configuring IPMI ==
-F19 specifics based on the [[IPMI]] tutorial.
 <syntaxhighlight lang="bash">
-yum -y install ipmitools OpenIPMI
+ssh-keyscan an-a04n02.alteeve.ca >> ~/.ssh/known_hosts
-systemctl start ipmi.service
-systemctl enable ipmi.service
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-ln -s '/usr/lib/systemd/system/ipmi.service' '/etc/systemd/system/multi-user.target.wants/ipmi.service'
+# an-a04n02.alteeve.ca SSH-2.0-OpenSSH_5.3
 </syntaxhighlight>
-Our servers use lan channel 2, yours might be 1 or something else. Experiment.
 <syntaxhighlight lang="bash">
-ipmitool lan print 2
+ssh-keyscan an-a04n02 >> ~/.ssh/known_hosts
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Set in Progress         : Set Complete
+# an-a04n02 SSH-2.0-OpenSSH_5.3
-Auth Type Support       : NONE MD5 PASSWORD
-Auth Type Enable        : Callback : NONE MD5 PASSWORD
-                        : User     : NONE MD5 PASSWORD
-                        : Operator : NONE MD5 PASSWORD
-                        : Admin    : NONE MD5 PASSWORD
-                        : OEM      : NONE MD5 PASSWORD
-IP Address Source       : BIOS Assigned Address
-IP Address              : 10.20.51.1
-Subnet Mask             : 255.255.0.0
-MAC Address             : 00:19:99:9a:d8:e8
-SNMP Community String   : public
-IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
-Default Gateway IP      : 10.20.255.254
-.1q VLAN ID          : Disabled
-.1q VLAN Priority    : 0
-RMCP+ Cipher Suites     : 0,1,2,3,6,7,8,17
-Cipher Suite Priv Max   : OOOOOOOOXXXXXXX
-                        :     X=Cipher Suite Unused
-                        :     c=CALLBACK
-                        :     u=USER
-                        :     o=OPERATOR
-                        :     a=ADMIN
-                        :     O=OEM
 </syntaxhighlight>
-I need to set the IPs to <span class="code">10.20.31.1/16</span> and <span class="code">10.20.31.2/16</span> for nodes 1 and 2, respectively. I also want to set the password to <span class="code">secret</span> for the <span class="code">admin</span> user.
-'''Node 01''' IP;
 <syntaxhighlight lang="bash">
-ipmitool lan set 2 ipsrc static
+ssh-keyscan an-a04n02.bcn >> ~/.ssh/known_hosts
-ipmitool lan set 2 ipaddr 10.20.31.
-ipmitool lan set 2 netmask 255.255.0.0
-ipmitool lan set 2 defgw ipaddr 10.20.255.254
-ipmitool lan print 2
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Set in Progress         : Set Complete
+# an-a04n02.bcn SSH-2.0-OpenSSH_5.3
-Auth Type Support       : NONE MD5 PASSWORD
-Auth Type Enable        : Callback : NONE MD5 PASSWORD
-                        : User     : NONE MD5 PASSWORD
-                        : Operator : NONE MD5 PASSWORD
-                        : Admin    : NONE MD5 PASSWORD
-                        : OEM      : NONE MD5 PASSWORD
-IP Address Source       : Static Address
-IP Address              : 10.20.31.1
-Subnet Mask             : 255.255.0.0
-MAC Address             : 00:19:99:9a:d8:e8
-SNMP Community String   : public
-IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
-Default Gateway IP      : 10.20.255.254
-.1q VLAN ID          : Disabled
-.1q VLAN Priority    : 0
-RMCP+ Cipher Suites     : 0,1,2,3,6,7,8,17
-Cipher Suite Priv Max   : OOOOOOOOXXXXXXX
-                        :     X=Cipher Suite Unused
-                        :     c=CALLBACK
-                        :     u=USER
-                        :     o=OPERATOR
-                        :     a=ADMIN
-                        :     O=OEM
 </syntaxhighlight>
-'''Node 01''' IP;
 <syntaxhighlight lang="bash">
-ipmitool lan set 2 ipsrc static
+ssh-keyscan an-a04n02.sn >> ~/.ssh/known_hosts
-ipmitool lan set 2 ipaddr 10.20.31.2
-ipmitool lan set 2 netmask 255.255.0.0
-ipmitool lan set 2 defgw ipaddr 10.20.255.254
-ipmitool lan print 2
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Set in Progress         : Set Complete
+# an-a04n02.sn SSH-2.0-OpenSSH_5.3
-Auth Type Support       : NONE MD5 PASSWORD
-Auth Type Enable        : Callback : NONE MD5 PASSWORD
-                        : User     : NONE MD5 PASSWORD
-                        : Operator : NONE MD5 PASSWORD
-                        : Admin    : NONE MD5 PASSWORD
-                        : OEM      : NONE MD5 PASSWORD
-IP Address Source       : Static Address
-IP Address              : 10.20.31.2
-Subnet Mask             : 255.255.0.0
-MAC Address             : 00:19:99:9a:b1:78
-SNMP Community String   : public
-IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
-Default Gateway IP      : 10.20.255.254
-.1q VLAN ID          : Disabled
-.1q VLAN Priority    : 0
-RMCP+ Cipher Suites     : 0,1,2,3,6,7,8,17
-Cipher Suite Priv Max   : OOOOOOOOXXXXXXX
-                        :     X=Cipher Suite Unused
-                        :     c=CALLBACK
-                        :     u=USER
-                        :     o=OPERATOR
-                        :     a=ADMIN
-                        :     O=OEM
 </syntaxhighlight>
-Set the password.
 <syntaxhighlight lang="bash">
-ipmitool user list 2
+ssh-keyscan an-a04n02.ifn >> ~/.ssh/known_hosts
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-ID  Name	     Callin  Link Auth	IPMI Msg   Channel Priv Limit
+# an-a04n02.ifn SSH-2.0-OpenSSH_5.3
-                    true    true       true       Unknown (0x00)
-  admin            true    true       true       OEM
-Get User Access command failed (channel 2, user 3): Unknown (0x32)
 </syntaxhighlight>
+|}
-(ignore the error, it's harmless... *BOOM*)
+Now copy the files to the second node:
-We want to set <span class="code">admin</span>'s password, so we do:
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
-{{note|1=The <span class="code">2</span> below is the ID number, not the LAN channel.}}
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+rsync -av ~/.ssh/authorized_keys root@an-a04n02:/root/.ssh/
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+root@an-a04n02's password:
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+sending incremental file list
+authorized_keys
+sent 2937 bytes  received 31 bytes  1187.20 bytes/sec
+total size is 2854  speedup is 0.96
+</syntaxhighlight>
 <syntaxhighlight lang="bash">
-ipmitool user set password 2 secret
+rsync -av ~/.ssh/known_hosts root@an-a04n02:/root/.ssh/
 </syntaxhighlight>
+<syntaxhighlight lang="text">
+sending incremental file list
+known_hosts
-Done!
+sent 4829 bytes  received 31 bytes  9720.00 bytes/sec
+total size is 4750  speedup is 0.98
-= Configuring the Anvil! =
+</syntaxhighlight>
+|}
-Now we're getting down to business!
+Note that there was no password prompt the second time. Hoozah!
-For this section, we will be working on <span class="code">an-a03n01</span> and using [[ssh]] to perform tasks on <span class="code">an-a03n02</span>.
+== Configuring the Firewall ==
-{{note|1=TODO: explain what this is and how it works.}}
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
-== Enable the pcs Daemon ==
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+# cman (corosync's totem)
+iptables -I INPUT -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 -d 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
+iptables -I INPUT -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
-{{note|1=Most of this section comes more or less verbatim from the main [http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Clusters_from_Scratch/index.html Clusters from Scratch] tutorial.}}
+# dlm
+iptables -I INPUT -m state --state NEW -p tcp -s 10.20.0.0/16 -d 10.20.0.0/16 --dport 21064 -j ACCEPT
-We will use [[pcs]], the Pacemaker Configuration System, to configure our Anvil!.
+# DRBD resource 0 and 1 - on the SN
+iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7788 -j ACCEPT
+iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7789 -j ACCEPT
-<syntaxhighlight lang="bash">
+# Make the new rules persistent.
-systemctl start pcsd.service
+/etc/init.d/iptables save
-systemctl enable pcsd.service
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-ln -s '/usr/lib/systemd/system/pcsd.service' '/etc/systemd/system/multi-user.target.wants/pcsd.service'
+iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]
 </syntaxhighlight>
+|-
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+# cman (corosync's totem)
+iptables -I INPUT -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 -d 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
+iptables -I INPUT -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
-Now we need to set a password for the <span class="code">hacluster</span> user. This is the account used by <span class="code">pcs</span> on one node to talk to the <span class="code">pcs</span> [[daemon]] on the other node. For this tutorial, we will use the password <span class="code">secret</span>. You will want to use [https://xkcd.com/936/ a stronger password], of course.
+# dlm
+iptables -I INPUT -m state --state NEW -p tcp -s 10.20.0.0/16 -d 10.20.0.0/16 --dport 21064 -j ACCEPT
-<syntaxhighlight lang="bash">
+# DRBD resource 0 and 1 - on the SN
-echo secret | passwd --stdin hacluster
+iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7788 -j ACCEPT
+iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7789 -j ACCEPT
+# Make the new rules persistent.
+/etc/init.d/iptables save
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Changing password for user hacluster.
+iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]
-passwd: all authentication tokens updated successfully.
 </syntaxhighlight>
+|}
-== Initializing the Cluster ==
+== Keeping Time in Sync ==
-One of the biggest reasons we're using the [[pcs]] tool, over something like [[crm]], is that it has been written to simplify the setup of clusters on [[Red Hat]] style operating systems. It will configure [[corosync]] automatically.
+It's not as critical as it used to be to keep the clocks on the nodes in sync, but it's still a good idea.
-First, we need to know what <span class="code">hostname</span> we will need to use for <span class="code">[[pcs]]</span>.
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
-'''Node 01''':
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+chkconfig ntpd on
-<syntaxhighlight lang="bash">
+/etc/init.d/ntpd start
-hostname
 </syntaxhighlight>
-<syntaxhighlight lang="bash">
+<syntaxhighlight lang="text">
-an-a03n01.alteeve.ca
+Starting ntpd:                                             [  OK  ]
 </syntaxhighlight>
+|-
-'''Node 02''':
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-<syntaxhighlight lang="bash">
+chkconfig ntpd on
-hostname
+/etc/init.d/ntpd start
 </syntaxhighlight>
-<syntaxhighlight lang="bash">
+<syntaxhighlight lang="text">
-an-a03n02.alteeve.ca
+Starting ntpd:                                             [  OK  ]
 </syntaxhighlight>
+|}
-Next, authenticate against the cluster nodes.
+= Configuring the Anvil! =
-'''Both nodes''':
+Now we're getting down to business!
-<syntaxhighlight lang="bash">
+For this section, we will be working on <span class="code">an-a04n01</span> and using [[ssh]] to perform tasks on <span class="code">an-a04n02</span>.
-pcs cluster auth an-a03n01.alteeve.ca an-a03n02.alteeve.ca -u hacluster
-</syntaxhighlight>
-This will ask you for the user name and password. The default user name is <span class="code">hacluster</span> and we set the password to <span class="code">secret</span>.
+{{note|1=TODO: explain what this is and how it works.}}
-<syntaxhighlight lang="text">
+== Configuring cman ==
-Password:
-an-a03n01.alteeve.ca: 6e9f7e98-dfb7-4305-b8e0-d84bf4f93ce3
-an-a03n01.alteeve.ca: Authorized
-an-a03n02.alteeve.ca: ffee6a85-ddac-4d03-9b97-f136d532b478
-an-a03n02.alteeve.ca: Authorized
-</syntaxhighlight>
-'''Do this on one node only''':
+With RHEL 6, we do not need to configure corosync directly. We will create a "skeleton" cluster.conf file which will, in turn, handle corosync for us. Once configured and the configuration has been copied to the peer, we will start pacemaker and it will handle starting (and stopping) pacemaker and corosync for us.
-Now to initialize the cluster's communication and membership layer.
+We will use 'ccs' to configure the skeleton cluster.conf file.
-<syntaxhighlight lang="bash">
+{|class="wikitable"
-pcs cluster setup --name an-cluster-03 an-a03n01.alteeve.ca an-a03n02.alteeve.ca
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+ccs -f /etc/cluster/cluster.conf --createcluster an-anvil-04
+ccs -f /etc/cluster/cluster.conf --setcman two_node="1" expected_votes="1"
+ccs -f /etc/cluster/cluster.conf --addnode an-a04n01.alteeve.ca
+ccs -f /etc/cluster/cluster.conf --addnode an-a04n02.alteeve.ca
+ccs -f /etc/cluster/cluster.conf --addfencedev pcmk agent=fence_pcmk
+ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect an-a04n01.alteeve.ca
+ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect an-a04n02.alteeve.ca
+ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk an-a04n01.alteeve.ca pcmk-redirect port=an-a04n01.alteeve.ca
+ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk an-a04n02.alteeve.ca pcmk-redirect port=an-a04n02.alteeve.ca
+ccs -f /etc/cluster/cluster.conf --setfencedaemon post_join_delay="30"
+cat /etc/cluster/cluster.conf
 </syntaxhighlight>
-<syntaxhighlight lang="text">
+<syntaxhighlight lang="xml">
-an-a03n01.alteeve.ca: Succeeded
+<cluster config_version="10" name="an-anvil-04">
-an-a03n02.alteeve.ca: Succeeded
+  <fence_daemon post_join_delay="30"/>
+  <clusternodes>
+    <clusternode name="an-a04n01.alteeve.ca" nodeid="1">
+      <fence>
+        <method name="pcmk-redirect">
+          <device name="pcmk" port="an-a04n01.alteeve.ca"/>
+        </method>
+      </fence>
+    </clusternode>
+    <clusternode name="an-a04n02.alteeve.ca" nodeid="2">
+      <fence>
+        <method name="pcmk-redirect">
+          <device name="pcmk" port="an-a04n02.alteeve.ca"/>
+        </method>
+      </fence>
+    </clusternode>
+  </clusternodes>
+  <cman expected_votes="1" two_node="1"/>
+  <fencedevices>
+    <fencedevice agent="fence_pcmk" name="pcmk"/>
+  </fencedevices>
+  <rm>
+    <failoverdomains/>
+    <resources/>
+  </rm>
+</cluster>
 </syntaxhighlight>
+|}
-This will create the corosync configuration file <span class="code">/etc/corosync/corosync.conf</span>;
+Copy it to an-a04n02;
-<syntaxhighlight lang="bash">
+{|class="wikitable"
-cat /etc/corosync/corosync.conf
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+rsync -av /etc/cluster/cluster.conf root@an-a04n02:/etc/cluster/
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-totem {
+sending incremental file list
-version: 2
+cluster.conf
-secauth: off
-cluster_name: an-cluster-03
-transport: udpu
-}
-nodelist {
+sent 838 bytes  received 31 bytes  579.33 bytes/sec
-   node {
+total size is 758  speedup is 0.87
-        ring0_addr: an-a03n01.alteeve.ca
+</syntaxhighlight>
-        nodeid: 1
+|-
-       }
+!<span class="code">an-a04n02</span>
-  node {
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-        ring0_addr: an-a03n02.alteeve.ca
+cat /etc/cluster/cluster.conf
-         nodeid: 2
+</syntaxhighlight>
-       }
+<syntaxhighlight lang="xml">
-}
+<cluster config_version="10" name="an-anvil-04">
+  <fence_daemon post_join_delay="30"/>
-quorum {
+   <clusternodes>
-provider: corosync_votequorum
+    <clusternode name="an-a04n01.alteeve.ca" nodeid="1">
-two_node: 1
+      <fence>
-}
+        <method name="pcmk-redirect">
+          <device name="pcmk" port="an-a04n01.alteeve.ca"/>
-logging {
+         </method>
-to_syslog: yes
+      </fence>
-}
+    </clusternode>
+    <clusternode name="an-a04n02.alteeve.ca" nodeid="2">
+      <fence>
+        <method name="pcmk-redirect">
+          <device name="pcmk" port="an-a04n02.alteeve.ca"/>
+        </method>
+      </fence>
+    </clusternode>
+  </clusternodes>
+  <cman expected_votes="1" two_node="1"/>
+  <fencedevices>
+    <fencedevice agent="fence_pcmk" name="pcmk"/>
+  </fencedevices>
+  <rm>
+    <failoverdomains/>
+    <resources/>
+  </rm>
+</cluster>
 </syntaxhighlight>
+|}
-== Start the Cluster For the First Time ==
+== Starting Pacemaker ==
-This starts the cluster communication and membership layer for the first time.
+Now start pacemaker proper.
-'''On one node only''';
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
-<syntaxhighlight lang="bash">
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs cluster start --all
+/etc/init.d/pacemaker start
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+Starting cluster:
+   Checking if cluster has been disabled at boot...        [  OK  ]
+   Checking Network Manager...                             [  OK  ]
+   Global setup...                                         [  OK  ]
+   Loading kernel modules...                               [  OK  ]
+   Mounting configfs...                                    [  OK  ]
+   Starting cman...                                        [  OK  ]
+   Waiting for quorum...                                   [  OK  ]
+   Starting fenced...                                      [  OK  ]
+   Starting dlm_controld...                                [  OK  ]
+   Tuning DLM kernel config...                             [  OK  ]
+   Starting gfs_controld...                                [  OK  ]
+   Unfencing self...                                       [  OK  ]
+   Joining fence domain...                                 [  OK  ]
+Starting Pacemaker Cluster Manager                         [  OK  ]
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n02</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+/etc/init.d/pacemaker start
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-an-a03n01.alteeve.ca: Starting Cluster...
+Starting cluster:
-an-a03n02.alteeve.ca: Starting Cluster...
+   Checking if cluster has been disabled at boot...        [  OK  ]
+   Checking Network Manager...                             [  OK  ]
+   Global setup...                                         [  OK  ]
+   Loading kernel modules...                               [  OK  ]
+   Mounting configfs...                                    [  OK  ]
+   Starting cman...                                        [  OK  ]
+   Waiting for quorum...                                   [  OK  ]
+   Starting fenced...                                      [  OK  ]
+   Starting dlm_controld...                                [  OK  ]
+   Tuning DLM kernel config...                             [  OK  ]
+   Starting gfs_controld...                                [  OK  ]
+   Unfencing self...                                       [  OK  ]
+   Joining fence domain...                                 [  OK  ]
+Starting Pacemaker Cluster Manager                         [  OK  ]
 </syntaxhighlight>
+|}
-After a few moments, you should be able to check the status;
+Verify pacemaker proper started as expected.
-<syntaxhighlight lang="bash">
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs status
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Cluster name: an-cluster-03
+Cluster name: an-anvil-04
 WARNING: no stonith devices and stonith-enabled is not false
-Last updated: Mon Jun 24 23:28:29 2013
+Last updated: Wed May 28 20:59:33 2014
-Last change: Mon Jun 24 23:28:10 2013 via crmd on an-a03n01.alteeve.ca
+Last change: Wed May 28 20:59:18 2014 via crmd on an-a04n01.alteeve.ca
-Current DC: NONE
+Stack: cman
-Nodes configured, unknown expected votes
+Current DC: an-a04n01.alteeve.ca - partition with quorum
-Resources configured.
+Version: 1.1.10-14.el6_5.3-368c726
+Nodes configured
+Resources configured
-Node an-a03n01.alteeve.ca (1): UNCLEAN (offline)
+Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-Node an-a03n02.alteeve.ca (2): UNCLEAN (offline)
 Full list of resources:
 </syntaxhighlight>
+|-
+!<span class="code">an-a04n02</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+pcs status
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+WARNING: no stonith devices and stonith-enabled is not false
+Last updated: Wed May 28 20:59:29 2014
+Last change: Wed May 28 20:59:18 2014 via crmd on an-a04n01.alteeve.ca
+Stack: cman
+Current DC: an-a04n01.alteeve.ca - partition with quorum
+Version: 1.1.10-14.el6_5.3-368c726
+Nodes configured
+Resources configured
-The other node should show almost the identical output.
-== Disabling Quorum ==
+Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-{{note|1=Show the math.}}
+Full list of resources:
-With quorum enabled, a two node cluster will lose quorum once either node fails. So we have to disable quorum.
-By default, pacemaker uses quorum. You don't see this initially though;
-<syntaxhighlight lang="bash">
-pcs property
 </syntaxhighlight>
-<syntaxhighlight lang="text">
+|}
-Cluster Properties:
- dc-version: 1.1.9-0.1318.a7966fb.git.fc18-a7966fb
- cluster-infrastructure: corosync
-</syntaxhighlight>
-To disable it, we set <span class="code">no-quorum-policy=ignore</span>.
-<syntaxhighlight lang="bash">
+Note the error about stonith. We will address that momentarily.
-pcs property set no-quorum-policy=ignore
-pcs property
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-Cluster Properties:
- dc-version: 1.1.9-0.1318.a7966fb.git.fc18-a7966fb
- cluster-infrastructure: corosync
- no-quorum-policy: ignore
-</syntaxhighlight>
-== Enabling and Configuring Fencing ==
+== Configure and test stonith (aka fencing) ==
-We will use IPMI and PDU based fence devices for redundancy.
+We will use [[IPMI]] and [[PDU]] based fence devices with [http://clusterlabs.org/wiki/STONITH_Levels STONITH levels].
 You can see the list of available fence agents here. You will need to find the one for your hardware fence devices.
-<syntaxhighlight lang="bash">
+Note: [https://bugzilla.redhat.com/show_bug.cgi?id=1102444 Ignore the errors].
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs stonith list
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-fence_alom - Fence agent for Sun ALOM
 fence_apc - Fence agent for APC over telnet/ssh
 fence_apc_snmp - Fence agent for APC over SNMP
-fence_baytech - I/O Fencing agent for Baytech RPC switches in combination with a Cyclades Terminal
-                Server
 fence_bladecenter - Fence agent for IBM BladeCenter
+fence_bladecenter_snmp - Fence agent for IBM BladeCenter over SNMP
 fence_brocade - Fence agent for Brocade over telnet
-fence_bullpap - I/O Fencing agent for Bull FAME architecture controlled by a PAP management console.
+Error: no metadata for /usr/sbin/fence_check
 fence_cisco_mds - Fence agent for Cisco MDS
 fence_cisco_ucs - Fence agent for Cisco UCS
-fence_cpint - I/O Fencing agent for GFS on s390 and zSeries VM clusters
 fence_drac - fencing agent for Dell Remote Access Card
 fence_drac5 - Fence agent for Dell DRAC CMC/5
@@ Line 842: / Line 1,086: @@
 fence_ilo2 - Fence agent for HP iLO
 fence_ilo3 - Fence agent for IPMI over LAN
+fence_ilo4 - Fence agent for IPMI over LAN
 fence_ilo_mp - Fence agent for HP iLO MP
 fence_imm - Fence agent for IPMI over LAN
@@ Line 848: / Line 1,093: @@
 fence_ipmilan - Fence agent for IPMI over LAN
 fence_kdump - Fence agent for use with kdump
-fence_ldom - Fence agent for Sun LDOM
+Error: no metadata for /usr/sbin/fence_node
-fence_lpar - Fence agent for IBM LPAR
-fence_mcdata - I/O Fencing agent for McData FC switches
-fence_rackswitch - fence_rackswitch - I/O Fencing agent for RackSaver RackSwitch
 fence_rhevm - Fence agent for RHEV-M REST API
 fence_rsa - Fence agent for IBM RSA
@@ Line 857: / Line 1,099: @@
 fence_sanbox2 - Fence agent for QLogic SANBox2 FC switches
 fence_scsi - fence agent for SCSI-3 persistent reservations
+Error: no metadata for /usr/sbin/fence_tool
 fence_virsh - Fence agent for virsh
-fence_vixel - I/O Fencing agent for Vixel FC switches
+fence_virt - Fence agent for virtual machines
 fence_vmware - Fence agent for VMWare
 fence_vmware_soap - Fence agent for VMWare over SOAP API
 fence_wti - Fence agent for WTI
-fence_xcat - I/O Fencing agent for xcat environments
+fence_xvm - Fence agent for virtual machines
-fence_xenapi - XenAPI based fencing for the Citrix XenServer virtual machines.
-fence_zvm - I/O Fencing agent for GFS on s390 and zSeries VM clusters
 </syntaxhighlight>
+|}
 We will use <span class="code">fence_ipmilan</span> and <span class="code">fence_apc_snmp</span>.
 === Configuring IPMI Fencing ===
+Setup out IPMI BMCs (on LAN channel 2 and using user ID 2).
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+ipmitool lan set 2 ipsrc static
+ipmitool lan set 2 ipaddr 10.20.41.1
+ipmitool lan set 2 netmask 255.255.0.0
+ipmitool lan set 2 defgw ipaddr 10.20.255.254
+ipmitool user set password 2 Initial1
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n02</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+ipmitool lan set 2 ipsrc static
+ipmitool lan set 2 ipaddr 10.20.41.2
+ipmitool lan set 2 netmask 255.255.0.0
+ipmitool lan set 2 defgw ipaddr 10.20.255.254
+ipmitool user set password 2 Initial1
+</syntaxhighlight>
+|}
+Test the new settings (using the hostnames we set in /etc/hosts):
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+fence_ipmilan -a an-a04n02.ipmi -l admin -p Initial1 -o status
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+Getting status of IPMI:an-a04n02.ipmi...Chassis power = On
+Done
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n02</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+fence_ipmilan -a an-a04n01.ipmi -l admin -p Initial1 -o status
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+Getting status of IPMI:an-a04n01.ipmi...Chassis power = On
+Done
+</syntaxhighlight>
+|}
+Good, now we can configure IPMI fencing.
 Every fence agent has a possibly unique subset of options that can be used. You can see a brief description of these options with the <span class="code">pcs stonith describe fence_X</span> command. Let's look at the options available for <span class="code">fence_ipmilan</span>.
-<syntaxhighlight lang="bash">
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs stonith describe fence_ipmilan
 </syntaxhighlight>
@@ Line 882: / Line 1,172: @@
    passwd: Password (if required) to control power on IPMI device
    passwd_script: Script to retrieve password (if required)
-   lanplus: Use Lanplus
+   lanplus: Use Lanplus to improve security of connection
    login: Username/Login (if required) to control power on IPMI device
    action: Operation to perform. Valid operations: on, off, reboot, status, list, diag, monitor or metadata
@@ Line 892: / Line 1,182: @@
    privlvl: Privilege level on IPMI device
    verbose: Verbose mode
+  stonith-timeout: How long to wait for the STONITH action to complete per a stonith device.
+  priority: The priority of the stonith resource. Devices are tried in order of highest priority to lowest.
+  pcmk_host_map: A mapping of host names to ports numbers for devices that do not support host names.
+  pcmk_host_list: A list of machines controlled by this device (Optional unless pcmk_host_check=static-list).
+  pcmk_host_check: How to determin which machines are controlled by the device.
 </syntaxhighlight>
+|}
 One of the nice things about pcs is that it allows us to create a test file to prepare all our changes in. Then, when we're happy with the changes, merge them into the running cluster. So let's make a copy called <span class="code">stonith_cfg</span>
-<syntaxhighlight lang="bash">
-pcs cluster cib stonith_cfg
-</syntaxhighlight>
 Now add [[IPMI]] fencing.
-<syntaxhighlight lang="bash">
+{|class="wikitable"
-#                  unique name    fence agent   target node                           device addr             options
+!<span class="code">an-a04n01</span>
-pcs stonith create fence_n01_ipmi fence_ipmilan pcmk_host_list="an-a03n01.alteeve.ca" ipaddr="an-a03n01.ipmi" action="reboot" login="admin" passwd="secret" delay=15 op monitor interval=60s
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs stonith create fence_n02_ipmi fence_ipmilan pcmk_host_list="an-a03n02.alteeve.ca" ipaddr="an-a03n02.ipmi" action="reboot" login="admin" passwd="secret" op monitor interval=60s
+pcs cluster cib stonith_cfg
+#   work in our temp file         unique name    fence agent   target node                           device addr             options
+pcs -f stonith_cfg stonith create fence_n01_ipmi fence_ipmilan pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-a04n01.ipmi" action="reboot" login="admin" passwd="Initial1" delay=15 op monitor interval=10s
+pcs -f stonith_cfg stonith create fence_n02_ipmi fence_ipmilan pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-a04n02.ipmi" action="reboot" login="admin" passwd="Initial1" op monitor interval=10s
+pcs cluster cib-push stonith_cfg
 </syntaxhighlight>
+|}
-Note that <span class="code">fence_n01_ipmi</span> has a <span class="code">delay=15</span> set but <span class="code">fence_n02_ipmi</span> does not. If the network connection breaks between the two nodes, they will both try to fence each other at the same time. If <span class="code">acpid</span> is running, the slower node will not die right away. It will continue to run for up to four more seconds, ample time for it to also initiate a fence against the faster node. The end result is that both nodes get fenced. The ten-second delay protects against this by causing <span class="code">an-a03n02</span> to pause for <span class="code">10</span> seconds before initiating a fence against <span class="code">an-a03n01</span>. If both nodes are alive, <span class="code">an-a03n02</span> will power off before the 10 seconds pass, so it will never fence <span class="code">an-a03n01</span>. However, if <span class="code">an-a03n01</span> really is dead, after the ten seconds have elapsed, fencing will proceed as normal.
+Note that <span class="code">fence_n01_ipmi</span> has a <span class="code">delay=15</span> set but <span class="code">fence_n02_ipmi</span> does not. If the network connection breaks between the two nodes, they will both try to fence each other at the same time. If <span class="code">acpid</span> is running, the slower node will not die right away. It will continue to run for up to four more seconds, ample time for it to also initiate a fence against the faster node. The end result is that both nodes get fenced. The ten-second delay protects against this by causing <span class="code">an-a04n02</span> to pause for <span class="code">10</span> seconds before initiating a fence against <span class="code">an-a04n01</span>. If both nodes are alive, <span class="code">an-a04n02</span> will power off before the 10 seconds pass, so it will never fence <span class="code">an-a04n01</span>. However, if <span class="code">an-a04n01</span> really is dead, after the ten seconds have elapsed, fencing will proceed as normal.
-{{note|1=At the time of writing, <span class="code">pcmk_reboot_action</span> is needed to override pacemaker's global fence action and <span class="code">pcmk_reboot_action</span> is not recognized by pcs. Both of these issues will be resolved shortly; Pacemaker will honour <span class="code">action="..."</span> in v1.1.10 and pcs will recognize <span class="code">pcmk_*</span> special attributes "real soon now". Until then, the <span class="code">--force</span> switch is needed.}}
-Next, add the [[PDU]] fencing. This requires distinct "off" and "on" actions for each outlet on each PDU. With two nodes, each with two [[PSU]]s, this translates to eight commands. The "off" commands will be monitored to alert us if the PDU fails for some reason. There is no reason to monitor the "on" actions (it would be redundant). Note also that we don't bother using a "delay". The IPMI fence method will go first, before the PDU actions, so the PDU is already delayed.
-<syntaxhighlight lang="bash">
-# Node 1 - off
-pcs stonith create fence_n01_pdu1_off fence_apc_snmp pcmk_host_list="an-a03n01.alteeve.ca" ipaddr="an-p01" action="off" port="1" op monitor interval="60s"
-pcs stonith create fence_n01_pdu2_off fence_apc_snmp pcmk_host_list="an-a03n01.alteeve.ca" ipaddr="an-p02" action="off" port="1" power_wait="5" op monitor interval="60s"
-# Node 1 - on
-pcs stonith create fence_n01_pdu1_on fence_apc_snmp pcmk_host_list="an-a03n01.alteeve.ca" ipaddr="an-p01" action="on" port="1"
-pcs stonith create fence_n01_pdu2_on fence_apc_snmp pcmk_host_list="an-a03n01.alteeve.ca" ipaddr="an-p02" action="on" port="1"
-# Node 2 - off
-pcs stonith create fence_n02_pdu1_off fence_apc_snmp pcmk_host_list="an-a03n02.alteeve.ca" ipaddr="an-p01" action="off" port="2" op monitor interval="60s"
-pcs stonith create fence_n02_pdu2_off fence_apc_snmp pcmk_host_list="an-a03n02.alteeve.ca" ipaddr="an-p02" action="off" port="2" power_wait="5" op monitor interval="60s"
-# Node 2 - on
+NOTE: Get my PDUs back and use them here!
-pcs stonith create fence_n02_pdu1_on fence_apc_snmp pcmk_host_list="an-a03n02.alteeve.ca" ipaddr="an-p01" action="on" port="2"
-pcs stonith create fence_n02_pdu2_on fence_apc_snmp pcmk_host_list="an-a03n02.alteeve.ca" ipaddr="an-p02" action="on" port="2"
-</syntaxhighlight>
 We can check the new configuration now;
-<syntaxhighlight lang="bash">
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs status
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Cluster name: an-cluster-03
+Cluster name: an-anvil-04
-Last updated: Tue Jul  2 16:41:55 2013
+Last updated: Wed May 28 22:01:14 2014
-Last change: Tue Jul  2 16:41:44 2013 via cibadmin on an-a03n01.alteeve.ca
+Last change: Wed May 28 21:55:59 2014 via cibadmin on an-a04n01.alteeve.ca
-Stack: corosync
+Stack: cman
-Current DC: an-a03n01.alteeve.ca (1) - partition with quorum
+Current DC: an-a04n01.alteeve.ca - partition with quorum
-Version: 1.1.9-3.fc19-781a388
+Version: 1.1.10-14.el6_5.3-368c726
-Nodes configured, unknown expected votes
+Nodes configured
-Resources configured.
+Resources configured
-Online: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
+Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Full list of resources:
-  fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a03n01.alteeve.ca
+  fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca
-  fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a03n02.alteeve.ca
+  fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca
- fence_n01_pdu1_off	(stonith:fence_apc_snmp):	Started an-a03n01.alteeve.ca
- fence_n01_pdu2_off	(stonith:fence_apc_snmp):	Started an-a03n02.alteeve.ca
- fence_n02_pdu1_off	(stonith:fence_apc_snmp):	Started an-a03n01.alteeve.ca
- fence_n02_pdu2_off	(stonith:fence_apc_snmp):	Started an-a03n02.alteeve.ca
- fence_n01_pdu1_on	(stonith:fence_apc_snmp):	Started an-a03n01.alteeve.ca
- fence_n01_pdu2_on	(stonith:fence_apc_snmp):	Started an-a03n02.alteeve.ca
- fence_n02_pdu1_on	(stonith:fence_apc_snmp):	Started an-a03n01.alteeve.ca
- fence_n02_pdu2_on	(stonith:fence_apc_snmp):	Started an-a03n02.alteeve.ca
 </syntaxhighlight>
+|}
-Before we proceed, we need to tell pacemaker to use fencing;
+Tell pacemaker to use fencing;
-<syntaxhighlight lang="bash">
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs property set stonith-enabled=true
+pcs property set no-quorum-policy=ignore
 pcs property
 </syntaxhighlight>
 <syntaxhighlight lang="text">
 Cluster Properties:
-Cluster Properties:
+  cluster-infrastructure: cman
-  cluster-infrastructure: corosync
+  dc-version: 1.1.10-14.el6_5.3-368c726
-  dc-version: 1.1.9-3.fc19-781a388
   no-quorum-policy: ignore
   stonith-enabled: true
 </syntaxhighlight>
+|}
 Excellent!
@@ Line 983: / Line 1,258: @@
 == Configuring Fence Levels ==
-The goal of fence levels is to tell pacemaker that there are "fence methods" to try and to impose an order on those methods. Each method composes one or more fence primitives and, when 2 or more primitives are tied together, that all primitives must succeed for the overall method to succeed.
+TODO...
-So in our case; the order we want is;
-* IPMI -> PDUs
+=== Test Fencing ===
-The reason is that when IPMI fencing succeeds, we can be very certain the node is truly fenced. When PDU fencing succeeds, it only confirms that the power outlets were cycled. If someone moved a node's power cables to another outlet, we'll get a false positive. On that topic, tie-down the node's PSU cables to the PDU's cable tray when possible, clearly label the power cables and wrap the fingers of anyone who might move them around.
+ToDo: Kill each node with <span class="code">echo c > /proc/sysrq-trigger</span> and make sure the other node fences it.
-The PDU fencing needs to be implemented using four steps;
+= Shared Storage =
-* PDU 1, outlet X -> off
+DRBD -> Clustered LVM -> GFS2
-* PDU 2, outlet X -> off
-** The <span class="code">power_wait="5"</span> setting for the <span class="code">fence_n0X_pdu2_off</span> primitives will cause a 5 second delay here, giving ample time to ensure the nodes lose power
-* PDU 1, outlet X -> on
-* PDU 2, outlet X -> on
-This is to ensure that both outlets are off at the same time, ensuring that the node loses power. This works because <span class="code">fencing_topology</span> acts serially.
+== DRBD ==
-Putting all this together, we issue this command;
+We will use DRBD 8.4.
-<syntaxhighlight lang="bash">
-pcs stonith level add 1 an-a03n01.alteeve.ca fence_n01_ipmi
-pcs stonith level add 1 an-a03n02.alteeve.ca fence_n02_ipmi
-</syntaxhighlight>
-The <span class="code">1</span> tells pacemaker that this is our highest priority fence method. We can see that this was set using pcs;
-<syntaxhighlight lang="bash">
-pcs stonith level
-</syntaxhighlight>
-<syntaxhighlight lang="bash">
- Node: an-a03n01.alteeve.ca
-  Level 1 - fence_n01_ipmi
- Node: an-a03n02.alteeve.ca
-  Level 1 - fence_n02_ipmi
-</syntaxhighlight>
-Now we'll tell pacemaker to use the PDUs as the second fence method. Here we tie together the two <span class="code">off</span> calls and the two <span class="code">on</span> calls into a single method.
+=== Partition Storage ===
-<syntaxhighlight lang="bash">
+How you do this will depend a lot on your storage (local disks, md software RAID, hardware RAID, 1 or multiple arrays, etc). It will also depend on how you plan to divy up your servers; you need two partitions; One for servers that will run on node 1 and another for node 2. It also depends on how much space you want for the /shared partition.
-pcs stonith level add 2 an-a03n01.alteeve.ca fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on
-pcs stonith level add 2 an-a03n02.alteeve.ca fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on
-</syntaxhighlight>
-Check again and we'll see that the new methods were added.
+In our case, we're using a single hardware RAID array, we'll set aside 40 GB of space for /shared and we're going to divide the remaining free space evenly.
-<syntaxhighlight lang="bash">
-pcs stonith level
-</syntaxhighlight>
-<syntaxhighlight lang="bash">
- Node: an-a03n01.alteeve.ca
-  Level 1 - fence_n01_ipmi
-  Level 2 - fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on
- Node: an-a03n02.alteeve.ca
-  Level 1 - fence_n02_ipmi
-  Level 2 - fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on
-</syntaxhighlight>
-For those of us who are [[XML]] fans, this is what the [[cib]] looks like now:
-<syntaxhighlight lang="bash">
-cat /var/lib/pacemaker/cib/cib.xml
-</syntaxhighlight>
-<syntaxhighlight lang="xml">
-<cib epoch="18" num_updates="0" admin_epoch="0" validate-with="pacemaker-1.2" cib-last-written="Thu Jul 18 13:15:53 2013" update-origin="an-a03n01.alteeve.ca" update-client="cibadmin" crm_feature_set="3.0.7" have-quorum="1" dc-uuid="1">
-  <configuration>
-    <crm_config>
-      <cluster_property_set id="cib-bootstrap-options">
-        <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.9-dde1c52"/>
-        <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
-        <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/>
-      </cluster_property_set>
-    </crm_config>
-    <nodes>
-      <node id="1" uname="an-a03n01.alteeve.ca"/>
-      <node id="2" uname="an-a03n02.alteeve.ca"/>
-    </nodes>
-    <resources>
-      <primitive class="stonith" id="fence_n01_ipmi" type="fence_ipmilan">
-        <instance_attributes id="fence_n01_ipmi-instance_attributes">
-          <nvpair id="fence_n01_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a03n01.alteeve.ca"/>
-          <nvpair id="fence_n01_ipmi-instance_attributes-ipaddr" name="ipaddr" value="an-a03n01.ipmi"/>
-          <nvpair id="fence_n01_ipmi-instance_attributes-action" name="action" value="reboot"/>
-          <nvpair id="fence_n01_ipmi-instance_attributes-login" name="login" value="admin"/>
-          <nvpair id="fence_n01_ipmi-instance_attributes-passwd" name="passwd" value="secret"/>
-          <nvpair id="fence_n01_ipmi-instance_attributes-delay" name="delay" value="15"/>
-        </instance_attributes>
-        <operations>
-          <op id="fence_n01_ipmi-monitor-interval-60s" interval="60s" name="monitor"/>
-        </operations>
-      </primitive>
-      <primitive class="stonith" id="fence_n02_ipmi" type="fence_ipmilan">
-        <instance_attributes id="fence_n02_ipmi-instance_attributes">
-          <nvpair id="fence_n02_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a03n02.alteeve.ca"/>
-          <nvpair id="fence_n02_ipmi-instance_attributes-ipaddr" name="ipaddr" value="an-a03n02.ipmi"/>
-          <nvpair id="fence_n02_ipmi-instance_attributes-action" name="action" value="reboot"/>
-          <nvpair id="fence_n02_ipmi-instance_attributes-login" name="login" value="admin"/>
-          <nvpair id="fence_n02_ipmi-instance_attributes-passwd" name="passwd" value="secret"/>
-        </instance_attributes>
-        <operations>
-          <op id="fence_n02_ipmi-monitor-interval-60s" interval="60s" name="monitor"/>
-        </operations>
-      </primitive>
-      <primitive class="stonith" id="fence_n01_pdu1_off" type="fence_apc_snmp">
-        <instance_attributes id="fence_n01_pdu1_off-instance_attributes">
-          <nvpair id="fence_n01_pdu1_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a03n01.alteeve.ca"/>
-          <nvpair id="fence_n01_pdu1_off-instance_attributes-ipaddr" name="ipaddr" value="an-p01"/>
-          <nvpair id="fence_n01_pdu1_off-instance_attributes-action" name="action" value="off"/>
-          <nvpair id="fence_n01_pdu1_off-instance_attributes-port" name="port" value="1"/>
-        </instance_attributes>
-        <operations>
-          <op id="fence_n01_pdu1_off-monitor-interval-60s" interval="60s" name="monitor"/>
-        </operations>
-      </primitive>
-      <primitive class="stonith" id="fence_n01_pdu2_off" type="fence_apc_snmp">
-        <instance_attributes id="fence_n01_pdu2_off-instance_attributes">
-          <nvpair id="fence_n01_pdu2_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a03n01.alteeve.ca"/>
-          <nvpair id="fence_n01_pdu2_off-instance_attributes-ipaddr" name="ipaddr" value="an-p02"/>
-          <nvpair id="fence_n01_pdu2_off-instance_attributes-action" name="action" value="off"/>
-          <nvpair id="fence_n01_pdu2_off-instance_attributes-port" name="port" value="1"/>
-          <nvpair id="fence_n01_pdu2_off-instance_attributes-power_wait" name="power_wait" value="5"/>
-        </instance_attributes>
-        <operations>
-          <op id="fence_n01_pdu2_off-monitor-interval-60s" interval="60s" name="monitor"/>
-        </operations>
-      </primitive>
-      <primitive class="stonith" id="fence_n01_pdu1_on" type="fence_apc_snmp">
-        <instance_attributes id="fence_n01_pdu1_on-instance_attributes">
-          <nvpair id="fence_n01_pdu1_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a03n01.alteeve.ca"/>
-          <nvpair id="fence_n01_pdu1_on-instance_attributes-ipaddr" name="ipaddr" value="an-p01"/>
-          <nvpair id="fence_n01_pdu1_on-instance_attributes-action" name="action" value="on"/>
-          <nvpair id="fence_n01_pdu1_on-instance_attributes-port" name="port" value="1"/>
-        </instance_attributes>
-        <operations>
-          <op id="fence_n01_pdu1_on-monitor-interval-60s" interval="60s" name="monitor"/>
-        </operations>
-      </primitive>
-      <primitive class="stonith" id="fence_n01_pdu2_on" type="fence_apc_snmp">
-        <instance_attributes id="fence_n01_pdu2_on-instance_attributes">
-          <nvpair id="fence_n01_pdu2_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a03n01.alteeve.ca"/>
-          <nvpair id="fence_n01_pdu2_on-instance_attributes-ipaddr" name="ipaddr" value="an-p02"/>
-          <nvpair id="fence_n01_pdu2_on-instance_attributes-action" name="action" value="on"/>
-          <nvpair id="fence_n01_pdu2_on-instance_attributes-port" name="port" value="1"/>
-        </instance_attributes>
-        <operations>
-          <op id="fence_n01_pdu2_on-monitor-interval-60s" interval="60s" name="monitor"/>
-        </operations>
-      </primitive>
-      <primitive class="stonith" id="fence_n02_pdu1_off" type="fence_apc_snmp">
-        <instance_attributes id="fence_n02_pdu1_off-instance_attributes">
-          <nvpair id="fence_n02_pdu1_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a03n02.alteeve.ca"/>
-          <nvpair id="fence_n02_pdu1_off-instance_attributes-ipaddr" name="ipaddr" value="an-p01"/>
-          <nvpair id="fence_n02_pdu1_off-instance_attributes-action" name="action" value="off"/>
-          <nvpair id="fence_n02_pdu1_off-instance_attributes-port" name="port" value="2"/>
-        </instance_attributes>
-        <operations>
-          <op id="fence_n02_pdu1_off-monitor-interval-60s" interval="60s" name="monitor"/>
-        </operations>
-      </primitive>
-      <primitive class="stonith" id="fence_n02_pdu2_off" type="fence_apc_snmp">
-        <instance_attributes id="fence_n02_pdu2_off-instance_attributes">
-          <nvpair id="fence_n02_pdu2_off-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a03n02.alteeve.ca"/>
-          <nvpair id="fence_n02_pdu2_off-instance_attributes-ipaddr" name="ipaddr" value="an-p02"/>
-          <nvpair id="fence_n02_pdu2_off-instance_attributes-action" name="action" value="off"/>
-          <nvpair id="fence_n02_pdu2_off-instance_attributes-port" name="port" value="2"/>
-          <nvpair id="fence_n02_pdu2_off-instance_attributes-power_wait" name="power_wait" value="5"/>
-        </instance_attributes>
-        <operations>
-          <op id="fence_n02_pdu2_off-monitor-interval-60s" interval="60s" name="monitor"/>
-        </operations>
-      </primitive>
-      <primitive class="stonith" id="fence_n02_pdu1_on" type="fence_apc_snmp">
-        <instance_attributes id="fence_n02_pdu1_on-instance_attributes">
-          <nvpair id="fence_n02_pdu1_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a03n02.alteeve.ca"/>
-          <nvpair id="fence_n02_pdu1_on-instance_attributes-ipaddr" name="ipaddr" value="an-p01"/>
-          <nvpair id="fence_n02_pdu1_on-instance_attributes-action" name="action" value="on"/>
-          <nvpair id="fence_n02_pdu1_on-instance_attributes-port" name="port" value="2"/>
-        </instance_attributes>
-        <operations>
-          <op id="fence_n02_pdu1_on-monitor-interval-60s" interval="60s" name="monitor"/>
-        </operations>
-      </primitive>
-      <primitive class="stonith" id="fence_n02_pdu2_on" type="fence_apc_snmp">
-        <instance_attributes id="fence_n02_pdu2_on-instance_attributes">
-          <nvpair id="fence_n02_pdu2_on-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="an-a03n02.alteeve.ca"/>
-          <nvpair id="fence_n02_pdu2_on-instance_attributes-ipaddr" name="ipaddr" value="an-p02"/>
-          <nvpair id="fence_n02_pdu2_on-instance_attributes-action" name="action" value="on"/>
-          <nvpair id="fence_n02_pdu2_on-instance_attributes-port" name="port" value="2"/>
-        </instance_attributes>
-        <operations>
-          <op id="fence_n02_pdu2_on-monitor-interval-60s" interval="60s" name="monitor"/>
-        </operations>
-      </primitive>
-    </resources>
-    <constraints/>
-    <fencing-topology>
-      <fencing-level devices="fence_n01_ipmi" id="fl-an-a03n01.alteeve.ca-1" index="1" target="an-a03n01.alteeve.ca"/>
-      <fencing-level devices="fence_n02_ipmi" id="fl-an-a03n02.alteeve.ca-1" index="1" target="an-a03n02.alteeve.ca"/>
-      <fencing-level devices="fence_n01_pdu1_off,fence_n01_pdu2_off,fence_n01_pdu1_on,fence_n01_pdu2_on" id="fl-an-a03n01.alteeve.ca-2" index="2" target="an-a03n01.alteeve.ca"/>
-      <fencing-level devices="fence_n02_pdu1_off,fence_n02_pdu2_off,fence_n02_pdu1_on,fence_n02_pdu2_on" id="fl-an-a03n02.alteeve.ca-2" index="2" target="an-a03n02.alteeve.ca"/>
-    </fencing-topology>
-  </configuration>
-</cib>
-</syntaxhighlight>
-== Fencing using fence_virsh ==
-{{note|1=To write this section, I used two virtual machines called <span class="code">pcmk1</span> and <span class="code">pcmk2</span>.}}
-If you are trying to learn fencing using KVM or Xen virtual machines, you can use the <span class="code">fence_virsh</span>. You can also use <span class="code">[[Fencing KVM Virtual Servers|fence_virtd]]</span>, which is actually recommended by many, but I have found it to be rather unreliable.
-To use <span class="code">fence_virsh</span>, first install it.
-<syntaxhighlight lang="bash">
-yum -y install fence-agents-virsh
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-<lots of yum output>
-</syntaxhighlight>
-Now test it from the command line. To do this, we need to know a few things;
-* The VM host is at IP <span class="code">192.168.122.1</span>
-* The username and password (<span class="code">-l</span> and <span class="code">-p</span> respectively) are the credentials used to log into VM host over [[SSH]].
-** If you don't want your password to be shown, create a little shell script that simply prints your password and then use <span class="code">-S /path/to/script</span> instead of <span class="code">-p "secret"</span>.
-* The name of the target VM, as shown by <span class="code">virsh list --all</span> on the host, is the node (<span class="code">-n</span>) value. For me, the nodes are called <span class="code">an-a03n01</span> and <span class="code">an-a03n02</span>.
-=== Create the Password Script ===
-In my case, the host is called '<span class="code">lemass</span>', so I want to create a password script called '<span class="code">/root/lemass.pw</span>'. The name of the script is entirely up to you.
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-vim /root/lemass.pw
+parted -a opt /dev/sda "print free"
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-echo "my secret password"
-</syntaxhighlight>
-<syntaxhighlight lang="bash">
-chmod 755 /root/lemass.pw
-/root/lemass.pw
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-my secret password
+Model: LSI RAID 5/6 SAS 6G (scsi)
-</syntaxhighlight>
+Disk /dev/sda: 898GB
-<syntaxhighlight lang="bash">
+Sector size (logical/physical): 512B/512B
-rsync -av /root/lemass.pw root@an-a03n02:/root/
+Partition Table: msdos
-</syntaxhighlight>
-<syntaxhighlight lang="bash">
-sending incremental file list
-lemass.pw
-sent 102 bytes  received 31 bytes  266.00 bytes/sec
+Number  Start   End     Size    Type     File system     Flags
-total size is 25  speedup is 0.19
+.3kB  1049kB  1016kB           Free Space
+      1049kB  538MB   537MB   primary  ext4            boot
+      538MB   4833MB  4295MB  primary  linux-swap(v1)
+      4833MB  26.3GB  21.5GB  primary  ext4
+.3GB  898GB   872GB            Free Space
 </syntaxhighlight>
 |-
-!<span class="code">an-a03n02</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-/root/lemass.pw
+# same as an-a04n01
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-my secret password
 </syntaxhighlight>
 |}
-Done.
+So 872 GB of free space, less 40 for /shared leaves 832 GB for servers. Divided evenly in 2 gives us 416 GB per server pool. Our first partition will then be 446 GB (40 for /shared) and the second will be 416 GB.
+The free space starts at 26.3 GB, so our first partition will start at 26.3 GB and end at 492 GB (rounding off the .3). The second partition will then start at 492 GB and end at 898 GB, the end of the disk. Both of these new partitions will be contained in an extended partition.
-=== Test fence_virsh Status from the Command Line ===
+{{note|1=After each change, we will get an error saying "Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).  As
+a result, it may not reflect all of your changes until after reboot.". Will reboot once done to address this.}}
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-fence_virsh -a 192.168.122.1 -l root -S /root/lemass.pw -n an-a03n02 -o status
+parted -a opt /dev/sda "mkpart extended 26.3GB 898GB"
+parted -a opt /dev/sda "mkpart logical 26.3GB 492GB"
+parted -a opt /dev/sda "mkpart logical 492GB 898GB"
+parted -a opt /dev/sda "print free"
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Status: ON
+Model: LSI RAID 5/6 SAS 6G (scsi)
+Disk /dev/sda: 898GB
+Sector size (logical/physical): 512B/512B
+Partition Table: msdos
+Number  Start   End     Size    Type      File system     Flags
+.3kB  1049kB  1016kB            Free Space
+      1049kB  538MB   537MB   primary   ext4            boot
+      538MB   4833MB  4295MB  primary   linux-swap(v1)
+      4833MB  26.3GB  21.5GB  primary   ext4
+      26.3GB  898GB   872GB   extended                  lba
+      26.3GB  492GB   466GB   logical
+      492GB   898GB   406GB   logical
 </syntaxhighlight>
 |-
-!<span class="code">an-a03n02</span>
+!<span class="code">an-a04n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-fence_virsh -a 192.168.122.1 -l root -S /root/lemass.pw -n an-a03n01 -o status
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-Status: ON
-</syntaxhighlight>
-|}
-Excellent! Now to configure it in pacemaker;
-{|class="wikitable"
-!<span class="code">an-a03n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs stonith create fence_n01_virsh fence_virsh pcmk_host_list="an-a03n01.alteeve.ca" ipaddr="192.168.122.1" action="reboot" login="root" passwd_script="/root/lemass.pw" port="an-a03n01" delay=15 op monitor interval=60s
+# same as an-a04n01
-pcs stonith create fence_n02_virsh fence_virsh pcmk_host_list="an-a03n02.alteeve.ca" ipaddr="192.168.122.1" action="reboot" login="root" passwd_script="/root/lemass.pw" port="an-a03n02" op monitor interval=60s
-pcs cluster status
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-Cluster Status:
- Last updated: Sun Jan 26 15:45:31 2014
- Last change: Sun Jan 26 15:06:14 2014 via crmd on an-a03n01.alteeve.ca
- Stack: corosync
- Current DC: an-a03n02.alteeve.ca (2) - partition with quorum
- Version: 1.1.10-19.el7-368c726
-Nodes configured
-Resources configured
-PCSD Status:
-an-a03n01.alteeve.ca:
-  an-a03n01.alteeve.ca: Online
-an-a03n02.alteeve.ca:
-  an-a03n02.alteeve.ca: Online
 </syntaxhighlight>
 |}
-=== Test Fencing ===
+Reboot
-ToDo: Kill each node with <span class="code">echo c > /proc/sysrq-trigger</span> and make sure the other node fences it.
-= Shared Storage =
-== DRBD ==
-We will use DRBD 8.4.
-=== Install DRBD 8.4.4 from AN! ===
-{{warning|1=this doesn't work.}}
-ToDo: Make a proper repo
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-rpm -Uvh https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-8.4.4-4.el7.x86_64.rpm \
+reboot
-         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-bash-completion-8.4.4-4.el7.x86_64.rpm \
-         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-pacemaker-8.4.4-4.el7.x86_64.rpm \
-         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-udev-8.4.4-4.el7.x86_64.rpm \
-         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-utils-8.4.4-4.el7.x86_64.rpm \
-         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-heartbeat-8.4.4-4.el7.x86_64.rpm \
-         https://alteeve.ca/files/AN-Cluster_Tutorial_3/drbd84/drbd-xen-8.4.4-4.el7.x86_64.rpm
-</syntaxhighlight>
-<syntaxhighlight lang="text">
 </syntaxhighlight>
 |-
-!<span class="code">an-a03n02</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-</syntaxhighlight>
+reboot
-<syntaxhighlight lang="text">
 </syntaxhighlight>
 |}
-=== Install DRBD 8.4.4 From Source ===
-At this time, no EPEL repo exists for RHEL7, and the Fedora RPMs don't work, so we will install DRBD 8.4.4 from source.
-Install dependencies:
-<syntaxhighlight lang="bash">
-yum -y install gcc flex rpm-build wget kernel-devel
-wget -c http://oss.linbit.com/drbd/8.4/drbd-8.4.4.tar.gz
-tar -xvzf drbd-8.4.4.tar.gz
-cd drbd-8.4.4
-./configure \
-  --prefix=/usr \
-  --localstatedir=/var \
-  --sysconfdir=/etc \
-  --with-km \
-  --with-udev \
-  --with-pacemaker \
-  --with-bashcompletion \
-  --with-utils \
-  --without-xen \
-  --without-rgmanager \
-  --without-heartbeat
-make
-make install
-</syntaxhighlight>
-Don't let DRBD start on boot (pacemaker will handle it for us).
-<syntaxhighlight lang="bash">
-systemctl disable drbd.service
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-drbd.service is not a native service, redirecting to /sbin/chkconfig.
-Executing /sbin/chkconfig drbd off
-</syntaxhighlight>
-Done.
-=== Optional; Make RPMs ===
-{{warning|1=I've not been able to get the RPMs genreated here to install yet. I'd recommend skipping this, unless you want to help sort out the problems. :) }}
-After <span class="code">./configure</span> above, you can make RPMs instead of installing directly.
-Dependencies:
-<syntaxhighlight lang="bash">
-yum install rpmdevtools redhat-rpm-config kernel-devel
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-<install text>
-</syntaxhighlight>
-Setup RPM dev tree:
-<syntaxhighlight lang="bash">
-cd ~
-rpmdev-setuptree
-ls -lah ~/rpmbuild/
-wget -c http://oss.linbit.com/drbd/8.4/drbd-8.4.4.tar.gz
-tar -xvzf drbd-8.4.4.tar.gz
-cd drbd-8.4.4
-./configure \
-  --prefix=/usr \
-  --localstatedir=/var \
-  --sysconfdir=/etc \
-  --with-km \
-  --with-udev \
-  --with-pacemaker \
-  --with-bashcompletion \
-  --with-utils \
-  --without-xen \
-  --without-heartbeat
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-total 4.0K
-drwxr-xr-x. 7 root root   67 Dec 23 20:06 .
-dr-xr-x---. 6 root root 4.0K Dec 23 20:06 ..
-drwxr-xr-x. 2 root root    6 Dec 23 20:06 BUILD
-drwxr-xr-x. 2 root root    6 Dec 23 20:06 RPMS
-drwxr-xr-x. 2 root root    6 Dec 23 20:06 SOURCES
-drwxr-xr-x. 2 root root    6 Dec 23 20:06 SPECS
-drwxr-xr-x. 2 root root    6 Dec 23 20:06 SRPMS
-</syntaxhighlight>
-Userland tools:
-<syntaxhighlight lang="bash">
-make rpm
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-checking for presence of 8\.4\.4 in various changelog files
-<snip>
-+ exit 0
-You have now:
-/root/rpmbuild/RPMS/x86_64/drbd-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-utils-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-xen-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-udev-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-pacemaker-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-heartbeat-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-bash-completion-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-debuginfo-8.4.4-4.el7.x86_64.rpm
-</syntaxhighlight>
-Kernel module:
-<syntaxhighlight lang="bash">
-make kmp-rpm
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-checking for presence of 8\.4\.4 in various changelog files
-<snip>
-+ exit 0
-You have now:
-/root/rpmbuild/RPMS/x86_64/drbd-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-utils-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-xen-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-udev-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-pacemaker-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-heartbeat-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-bash-completion-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-debuginfo-8.4.4-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/kmod-drbd-8.4.4_3.10.0_54.0.1-4.el7.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/drbd-kernel-debuginfo-8.4.4-4.el7.x86_64.rpm
-</syntaxhighlight>
 === Configure DRBD ===
@@ Line 1,467: / Line 1,359: @@
 Configure <span class="code">global-common.conf</span>;
-<syntaxhighlight lang="bash">
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 vim /etc/drbd.d/global_common.conf
 </syntaxhighlight>
-<syntaxhighlight lang="bash">
+<syntaxhighlight lang="text">
 # These are options to set for the DRBD daemon sets the default values for
 # resources.
@@ Line 1,478: / Line 1,372: @@
 	# this to 'no'. The default is 'ask' which will prompt you each time
 	# DRBD is updated. Set to 'yes' to allow it without being prompted.
-	usage-count no;
+	usage-count yes;
 	# minor-count dialog-refresh disable-ip-verification
 }
 common {
 	handlers {
-		pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
+		# pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
-		pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
+		# pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
-		local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
+		# local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
 		# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
 		# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
 		# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
 		# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
 		# Hook into Pacemaker's fencing.
 		fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
+		before-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
 	}
 	startup {
 		# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
 	}
 	options {
 		# cpu-mask on-no-data-accessible
 	}
 	disk {
 		# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
@@ Line 1,512: / Line 1,407: @@
                  fencing resource-and-stonith;
 	}
 	net {
 		# protocol timeout max-epoch-size max-buffers unplug-watermark
@@ Line 1,521: / Line 1,416: @@
 		# congestion-fill congestion-extents csums-alg verify-alg
 		# use-rle
 		# Protocol "C" tells DRBD not to tell the operating system that
 		# the write is complete until the data has reach persistent
@@ Line 1,529: / Line 1,424: @@
 		# be using.
 		protocol C;
 		# Tell DRBD to allow dual-primary. This is needed to enable
 		# live-migration of our servers.
 		allow-two-primaries yes;
 		# This tells DRBD what to do in the case of a split-brain when
 		# neither node was primary, when one node was primary and when
@@ Line 1,546: / Line 1,441: @@
 }
 </syntaxhighlight>
+|}
 And now configure the first resource;
-<syntaxhighlight lang="bash">
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 vim /etc/drbd.d/r0.res
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-# This is the first DRBD resource. If will store the shared file systems and
+# This is the first DRBD resource. It will store the shared file systems and
 # the servers designed to run on node 01.
 resource r0 {
@@ Line 1,559: / Line 1,457: @@
 	# need to set unique values per node, you can move these to the
 	# 'on <name> { ... }' section.
 	# This sets the device name of this DRBD resouce.
 	device /dev/drbd0;
 	# This tells DRBD what the backing device is for this resource.
 	disk /dev/sda5;
 	# This controls the location of the metadata. When "internal" is used,
 	# as we use here, a little space at the end of the backing devices is
@@ Line 1,573: / Line 1,471: @@
 	# available for the metadata.
 	meta-disk internal;
 	# NOTE: this is not required or even recommended with pacemaker. remove
 	# 	this options as soon as pacemaker is setup.
@@ Line 1,583: / Line 1,481: @@
 		#become-primary-on both;
 	}
 	# NOTE: Later, make it an option in the dashboard to trigger a manual
 	# 	verify and/or schedule periodic automatic runs
@@ Line 1,593: / Line 1,491: @@
 		# out-of-sync.
 		verify-alg md5;
 		# TODO: Test the performance hit of this being enabled.
 		# This tells DRBD to generate a checksum for each transmitted
@@ Line 1,603: / Line 1,501: @@
 		data-integrity-alg md5;
 	}
 	# WARNING: Confirm that these are safe when the controller's BBU is
 	#          depleted/failed and the controller enters write-through
@@ Line 1,619: / Line 1,517: @@
 		md-flushes no;
 	}
 	# This sets up the resource on node 01. The name used below must be the
 	# named returned by "uname -n".
-	on an-a03n01.alteeve.ca {
+	on an-a04n01.alteeve.ca {
 		# This is the address and port to use for DRBD traffic on this
 		# node. Multiple resources can use the same IP but the ports
@@ Line 1,628: / Line 1,526: @@
 		# second uses 7789 and so on, incrementing by one for each
 		# additional resource.
-		address 10.10.30.1:7788;
+		address 10.10.40.1:7788;
 	}
-	on an-a03n02.alteeve.ca {
+	on an-a04n02.alteeve.ca {
-		address 10.10.30.2:7788;
+		address 10.10.40.2:7788;
 	}
 }
 </syntaxhighlight>
+|}
-Disable <span class="code">drbd</span> from starting on boot.
+And the second.
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+vim /etc/drbd.d/r1.res
+</syntaxhighlight>
 <syntaxhighlight lang="bash">
-systemctl disable drbd.service
+# This is the first DRBD resource. It will store the servers designed
+# to run on node 02.
+resource r1 {
+	device /dev/drbd1;
+	disk /dev/sda6;
+	meta-disk internal;
+	net {
+		verify-alg md5;
+		data-integrity-alg md5;
+	}
+	disk {
+		disk-flushes no;
+		md-flushes no;
+	}
+	on an-a04n01.alteeve.ca {
+		address 10.10.40.1:7789;
+	}
+	on an-a04n02.alteeve.ca {
+		address 10.10.40.2:7789;
+	}
+}
+</syntaxhighlight>
+|}
+Test the config;
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+drbdadm dump
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-drbd.service is not a native service, redirecting to /sbin/chkconfig.
+# /etc/drbd.conf
-Executing /sbin/chkconfig drbd off
+common {
+}
+# resource r0 on an-a04n01.alteeve.ca: not ignored, not stacked
+# defined at /etc/drbd.d/r0.res:3
+resource r0 {
+    on an-a04n01.alteeve.ca {
+        volume 0 {
+            device       /dev/drbd0 minor 0;
+            disk         /dev/sda5;
+            meta-disk    internal;
+        }
+        address          ipv4 10.10.40.1:7788;
+    }
+    on an-a04n02.alteeve.ca {
+        volume 0 {
+            device       /dev/drbd0 minor 0;
+            disk         /dev/sda5;
+            meta-disk    internal;
+        }
+        address          ipv4 10.10.40.2:7788;
+    }
+    net {
+        verify-alg       md5;
+        data-integrity-alg md5;
+    }
+    disk {
+        disk-flushes      no;
+        md-flushes        no;
+    }
+}
+# resource r1 on an-a04n01.alteeve.ca: not ignored, not stacked
+# defined at /etc/drbd.d/r1.res:3
+resource r1 {
+    on an-a04n01.alteeve.ca {
+        volume 0 {
+            device       /dev/drbd1 minor 1;
+            disk         /dev/sda6;
+            meta-disk    internal;
+        }
+        address          ipv4 10.10.40.1:7789;
+    }
+    on an-a04n02.alteeve.ca {
+        volume 0 {
+            device       /dev/drbd1 minor 1;
+            disk         /dev/sda6;
+            meta-disk    internal;
+        }
+        address          ipv4 10.10.40.2:7789;
+    }
+    net {
+        verify-alg       md5;
+        data-integrity-alg md5;
+    }
+    disk {
+        disk-flushes      no;
+        md-flushes        no;
+    }
+}
 </syntaxhighlight>
+|}
-Load the config;
+Good, copy it to the other node and test it there.
-<syntaxhighlight lang="bash">
+{|class="wikitable"
-modprobe drbd
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+rsync -av /etc/drbd.* root@an-a04n02:/etc/
 </syntaxhighlight>
+<syntaxhighlight lang="text">
+sending incremental file list
+drbd.d/
+drbd.d/global_common.conf
+drbd.d/r0.res
+drbd.d/r1.res
-Now check the config;
+sent 5738 bytes  received 73 bytes  11622.00 bytes/sec
+total size is 5618  speedup is 0.97
-<syntaxhighlight lang="bash">
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 drbdadm dump
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-  --==  Thank you for participating in the global usage survey  ==--
+# /etc/drbd.conf
-The server's response is:
+common {
+}
+# resource r0 on an-a04n02.alteeve.ca: not ignored, not stacked
+# defined at /etc/drbd.d/r0.res:3
+resource r0 {
+    on an-a04n01.alteeve.ca {
+        volume 0 {
+            device       /dev/drbd0 minor 0;
+            disk         /dev/sda5;
+            meta-disk    internal;
+        }
+        address          ipv4 10.10.40.1:7788;
+    }
+    on an-a04n02.alteeve.ca {
+        volume 0 {
+            device       /dev/drbd0 minor 0;
+            disk         /dev/sda5;
+            meta-disk    internal;
+        }
+        address          ipv4 10.10.40.2:7788;
+    }
+    net {
+        verify-alg       md5;
+        data-integrity-alg md5;
+    }
+    disk {
+        disk-flushes      no;
+        md-flushes        no;
+    }
+}
-you are the 69th user to install this version
+# resource r1 on an-a04n02.alteeve.ca: not ignored, not stacked
-/etc/drbd.d/r0.res:3: in resource r0:
+# defined at /etc/drbd.d/r1.res:3
-become-primary-on is set to both, but allow-two-primaries is not set.
+resource r1 {
+    on an-a04n01.alteeve.ca {
+        volume 0 {
+            device       /dev/drbd1 minor 1;
+            disk         /dev/sda6;
+            meta-disk    internal;
+        }
+        address          ipv4 10.10.40.1:7789;
+    }
+    on an-a04n02.alteeve.ca {
+        volume 0 {
+            device       /dev/drbd1 minor 1;
+            disk         /dev/sda6;
+            meta-disk    internal;
+        }
+        address          ipv4 10.10.40.2:7789;
+    }
+    net {
+        verify-alg       md5;
+        data-integrity-alg md5;
+    }
+    disk {
+        disk-flushes      no;
+        md-flushes        no;
+    }
+}
 </syntaxhighlight>
+|}
+This isn't a plain dump of your configs, you will notice things have been shifted around. The point is that it dumped the configuration without errors, so we're good to go.
+=== Start DRBD for the first time ===
-Ignore that error. It has been reported and does not effect operation.
+Load the config;
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+modprobe drbd
+lsmod | grep drbd
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+drbd                  333723  0
+libcrc32c               1246  1 drbd
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+modprobe drbd
+lsmod | grep drbd
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+drbd                  333723  0
+libcrc32c               1246  1 drbd
+</syntaxhighlight>
+|}
+{{note|1=If you have used these partitions before, drbd may see an FS and refuse to create the MD. If that happens, use 'dd' to zero out the partition.}}
 Create the metadisk;
-<syntaxhighlight lang="bash">
+{|class="wikitable"
-drbdadm create-md r0
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+drbdadm create-md r{0,1}
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+Writing meta data...
+initializing activity log
+NOT initializing bitmap
+New drbd meta data block successfully created.
+success
+Writing meta data...
+initializing activity log
+NOT initializing bitmap
+New drbd meta data block successfully created.
+success
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+drbdadm create-md r{0,1}
 </syntaxhighlight>
 <syntaxhighlight lang="text">
+Writing meta data...
+initializing activity log
+NOT initializing bitmap
+New drbd meta data block successfully created.
+success
 Writing meta data...
 initializing activity log
@@ Line 1,680: / Line 1,795: @@
 success
 </syntaxhighlight>
+|}
-Start the DRBD resource on both nodes;
+Bring up the new resources.
-<syntaxhighlight lang="bash">
+{|class="wikitable"
-drbdadm up r0
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+drbdadm up r{0,1}
+cat /proc/drbd
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+version: 8.4.4 (api:1/proto:86-101)
+GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
+: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/Outdated C r----s
+    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:454762916
+: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/Outdated C r----s
+    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396782732
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+drbdadm up r{0,1}
+cat /proc/drbd
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+version: 8.4.4 (api:1/proto:86-101)
+GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
+: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
+    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:454762916
+: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
+    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396782732
 </syntaxhighlight>
+|}
-Once <span class="code">/proc/drbd</span> shows both nodes connected, force one to primary and it will sync over the second.
+Neither node has data, so we'll arbitrarily force node 01 to become primary, then normally promote node 02 to primary.
-<syntaxhighlight lang="bash">
+{|class="wikitable"
-drbdadm primary --force r0
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+drbdadm primary --force r{0,1}
+cat /proc/drbd
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+version: 8.4.4 (api:1/proto:86-101)
+GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
+: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
+    ns:2136 nr:0 dw:0 dr:2800 al:0 bm:0 lo:0 pe:3 ua:0 ap:0 ep:1 wo:d oos:454760880
+        [>....................] sync'ed:  0.1% (444100/444104)M
+        finish: 421:04:29 speed: 252 (252) K/sec
+: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
+    ns:24696 nr:0 dw:0 dr:25360 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396758036
+        [>....................] sync'ed:  0.1% (387456/387480)M
+        finish: 35:33:06 speed: 3,084 (3,084) K/sec
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+drbdadm primary r{0,1}
+cat /proc/drbd
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+version: 8.4.4 (api:1/proto:86-101)
+GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
+: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
+    ns:0 nr:859488 dw:859432 dr:608 al:0 bm:52 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:453903484
+        [>....................] sync'ed:  0.2% (443264/444104)M
+        finish: 71:24:53 speed: 1,752 (4,428) want: 440 K/sec
+: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
+    ns:0 nr:1140588 dw:1140532 dr:608 al:0 bm:69 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:395642200
+        [>....................] sync'ed:  0.3% (386368/387480)M
+        finish: 70:30:41 speed: 1,548 (5,876) want: 4,400 K/sec
 </syntaxhighlight>
+|}
+The sync rate starts low, but it will continue to climb, you can keep an eye on it if you wish. DRBD 8.4 is smarter than 8.3 in that it will adjust the sync rate automatically based on load.
+We can proceed now, we do not have to wait for the sync to complete.
+== Clustered LVM and GFS2 ==
-You should see the resource syncing now. Push both nodes to primary;
+Clustered LVM provides the logical volumes that will back our /shared GFS2 partition and the storage for the HA servers.
-<syntaxhighlight lang="bash">
+=== Configure lvm.conf ===
-drbdadm primary r0
-</syntaxhighlight>
-== DLM, Clustered LVM and GFS2 ==
+Configure clustered LVM.
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 sed -i.anvil 's^filter = \[ "a/\.\*/" \]^filter = \[ "a|/dev/drbd*|", "r/.*/" \]^' /etc/lvm/lvm.conf
 sed -i 's/locking_type = 1$/locking_type = 3/' /etc/lvm/lvm.conf
 sed -i 's/fallback_to_local_locking = 1$/fallback_to_local_locking = 0/' /etc/lvm/lvm.conf
-sed -i 's/use_lvmetad = 1$/use_lvmetad = 0/' /etc/lvm/lvm.conf
+diff -u /etc/lvm/lvm.conf.anvil /etc/lvm/lvm.conf
 </syntaxhighlight>
 <syntaxhighlight lang="diff">
---- /etc/lvm/lvm.conf.anvil	2013-11-27 03:28:08.000000000 -0500
+--- /etc/lvm/lvm.conf.anvil	2013-10-30 04:10:42.000000000 -0400
-+++ /etc/lvm/lvm.conf	2014-01-26 18:57:41.026928464 -0500
++++ /etc/lvm/lvm.conf	2014-06-04 18:38:15.545166869 -0400
-@@ -84,7 +84,7 @@
+@@ -82,7 +82,7 @@
-     # lvmetad is used" comment that is attached to global/use_lvmetad setting.
       # By default we accept every block device:
@@ Line 1,721: / Line 1,901: @@
       # Exclude the cdrom drive
       # filter = [ "r|/dev/cdrom|" ]
-@@ -451,7 +451,7 @@
+@@ -459,7 +459,7 @@
-      # supported in clustered environment. If use_lvmetad=1 and locking_type=3
+      # Type 3 uses built-in clustered locking.
-      # is set at the same time, LVM always issues a warning message about this
+      # Type 4 uses read-only locking which forbids any operations that might
-      # and then it automatically disables lvmetad use.
+      # change metadata.
 -    locking_type = 1
 +    locking_type = 3
@@ Line 1,730: / Line 1,910: @@
       # Set to 0 to fail when a lock request cannot be satisfied immediately.
       wait_for_locks = 1
-@@ -467,7 +467,7 @@
+@@ -475,7 +475,7 @@
       # to 1 an attempt will be made to use local file-based locking (type 1).
       # If this succeeds, only commands against local volume groups will proceed.
@@ Line 1,739: / Line 1,919: @@
       # Local non-LV directory that holds file-based locks while commands are
       # in progress.  A directory like /tmp that may get wiped on reboot is OK.
-@@ -594,7 +594,7 @@
-     # supported in clustered environment. If use_lvmetad=1 and locking_type=3
-     # is set at the same time, LVM always issues a warning message about this
-     # and then it automatically disables lvmetad use.
--    use_lvmetad = 1
-+    use_lvmetad = 0
-     # Full path of the utility called to check that a thin metadata device
-     # is in a state that allows it to be used.
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-rsync -av /etc/lvm/lvm.conf* root@an-a03n02:/etc/lvm/
+rsync -av /etc/lvm/lvm.conf* root@an-a04n02:/etc/lvm/
 </syntaxhighlight>
 <syntaxhighlight lang="text">
@@ Line 1,757: / Line 1,928: @@
 lvm.conf.anvil
-sent 48536 bytes  received 440 bytes  97952.00 bytes/sec
+sent 47499 bytes  received 440 bytes  95878.00 bytes/sec
-total size is 90673  speedup is 1.85
+total size is 89999  speedup is 1.88
 </syntaxhighlight>
 |-
-!<span class="code">an-a03n02</span>
+!<span class="code">an-a04n02</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 diff -u /etc/lvm/lvm.conf.anvil /etc/lvm/lvm.conf
 </syntaxhighlight>
 <syntaxhighlight lang="diff">
---- /etc/lvm/lvm.conf.anvil	2013-11-27 03:28:08.000000000 -0500
+--- /etc/lvm/lvm.conf.anvil	2013-10-30 04:10:42.000000000 -0400
-+++ /etc/lvm/lvm.conf	2014-01-26 18:57:41.000000000 -0500
++++ /etc/lvm/lvm.conf	2014-06-04 18:38:15.000000000 -0400
-@@ -84,7 +84,7 @@
+@@ -82,7 +82,7 @@
-     # lvmetad is used" comment that is attached to global/use_lvmetad setting.
       # By default we accept every block device:
@@ Line 1,777: / Line 1,948: @@
       # Exclude the cdrom drive
       # filter = [ "r|/dev/cdrom|" ]
-@@ -451,7 +451,7 @@
+@@ -459,7 +459,7 @@
-      # supported in clustered environment. If use_lvmetad=1 and locking_type=3
+      # Type 3 uses built-in clustered locking.
-      # is set at the same time, LVM always issues a warning message about this
+      # Type 4 uses read-only locking which forbids any operations that might
-      # and then it automatically disables lvmetad use.
+      # change metadata.
 -    locking_type = 1
 +    locking_type = 3
@@ Line 1,786: / Line 1,957: @@
       # Set to 0 to fail when a lock request cannot be satisfied immediately.
       wait_for_locks = 1
-@@ -467,7 +467,7 @@
+@@ -475,7 +475,7 @@
       # to 1 an attempt will be made to use local file-based locking (type 1).
       # If this succeeds, only commands against local volume groups will proceed.
@@ Line 1,795: / Line 1,966: @@
       # Local non-LV directory that holds file-based locks while commands are
       # in progress.  A directory like /tmp that may get wiped on reboot is OK.
-@@ -594,7 +594,7 @@
-     # supported in clustered environment. If use_lvmetad=1 and locking_type=3
-     # is set at the same time, LVM always issues a warning message about this
-     # and then it automatically disables lvmetad use.
--    use_lvmetad = 1
-+    use_lvmetad = 0
-     # Full path of the utility called to check that a thin metadata device
-     # is in a state that allows it to be used.
 </syntaxhighlight>
 |}
-Disable <span class="code">lvmetad</span> as it's not cluster-aware.
+=== Start clvmd ===
+{{note|1=This will be moved to pacemaker shortly. We're enabling it here just long enough to configure pacemaker.}}
+Make sure the cluster is up (you could use 'pcs status', 'cman_tool status', etc):
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+dlm_tool dump | grep node
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+1401921044 cluster node 1 added seq 68
+1401921044 set_configfs_node 1 10.20.40.1 local 1
+1401921044 cluster node 2 added seq 68
+1401921044 set_configfs_node 2 10.20.40.2 local 0
+1401921044 run protocol from nodeid 1
+</syntaxhighlight>
+|}
+Make sure DRBD is up as primary on both nodes:
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+cat /proc/drbd
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+version: 8.3.16 (api:88/proto:86-97)
+GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
+: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r-----
+    ns:1519672 nr:0 dw:0 dr:1520336 al:0 bm:93 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:448214308
+	[>....................] sync'ed:  0.4% (437708/439192)M
+	finish: 6:20:02 speed: 19,652 (15,992) K/sec
+: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r-----
+    ns:1896504 nr:0 dw:0 dr:1897168 al:0 bm:115 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:390577164
+	[>....................] sync'ed:  0.5% (381420/383272)M
+	finish: 2:33:17 speed: 42,440 (19,960) K/sec
+</syntaxhighlight>
+|}
+Note that we don't have to wait for the sync to finish.
+Start clvmd;
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-systemctl disable lvm2-lvmetad.service
+/etc/init.d/clvmd start
-systemctl disable lvm2-lvmetad.socket
-systemctl stop lvm2-lvmetad.service
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-rm '/etc/systemd/system/sockets.target.wants/lvm2-lvmetad.socket'
+Starting clvmd:
+Activating VG(s):   No volume groups found
+                                                           [  OK  ]
 </syntaxhighlight>
 |-
-!<span class="code">an-a03n02</span>
+!<span class="code">an-a04n02</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-systemctl disable lvm2-lvmetad.service
+/etc/init.d/clvmd start
-systemctl disable lvm2-lvmetad.socket
-systemctl stop lvm2-lvmetad.service
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-rm '/etc/systemd/system/sockets.target.wants/lvm2-lvmetad.socket'
+Starting clvmd:
+Activating VG(s):   No volume groups found
+                                                           [  OK  ]
 </syntaxhighlight>
 |}
-{{note|1=This will be moved to pacemaker shortly. We're enabling it here just long enough to configure pacemaker.}}
+{{note|1=If this fails, showing a timeout or simply never returning, make sure that TCP port 21064 is opened in your firewall on both nodes.}}
-Start DLM and clvmd;
+From here on, pacemaker will start clvmd when pacemaker itself start, *if* clvmd is set to start on boot. So lets set that.
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-systemctl start dlm.service
+chkconfig clvmd on
-systemctl start clvmd.service
+chkconfig --list clvmd
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+clvmd          	0:off	1:off	2:on	3:on	4:on	5:on	6:off
 </syntaxhighlight>
 |-
-!<span class="code">an-a03n02</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-systemctl start dlm.service
+chkconfig clvmd on
-systemctl start clvmd.service
+chkconfig --list clvmd
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+clvmd          	0:off	1:off	2:on	3:on	4:on	5:on	6:off
 </syntaxhighlight>
 |}
+=== Create Initial PVs, VGs and the /shared LV ===
 Create the [[PV]], [[VG]] and the <span class="code">/shared</span> [[LV]];
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pvcreate /dev/drbd0
+pvcreate /dev/drbd{0,1}
 </syntaxhighlight>
 <syntaxhighlight lang="text">
    Physical volume "/dev/drbd0" successfully created
+  Physical volume "/dev/drbd1" successfully created
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+vgcreate an-a04n01_vg0 /dev/drbd0
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+  Clustered volume group "an-a04n01_vg0" successfully created
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-vgcreate an-a03n01_vg0 /dev/drbd0
+vgcreate an-a04n02_vg0 /dev/drbd1
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-  /proc/devices: No entry for device-mapper found
+   Clustered volume group "an-a04n02_vg0" successfully created
-   Clustered volume group "an-a03n01_vg0" successfully created
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-lvcreate -L 10G -n shared an-a03n01_vg0
+lvcreate -L 40GiB -n shared an-a04n01_vg0
 </syntaxhighlight>
 <syntaxhighlight lang="text">
@@ Line 1,873: / Line 2,092: @@
 </syntaxhighlight>
 |-
-!<span class="code">an-a03n02</span>
+!<span class="code">an-a04n02</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pvscan
+pvdisplay
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-   PV /dev/drbd0   VG an-a03n01_vg0   lvm2 [20.00 GiB / 20.00 GiB free]
+  --- Physical volume ---
-   Total: 1 [20.00 GiB] / in use: 1 [20.00 GiB] / in no VG: 0 [0   ]
+   PV Name               /dev/drbd1
+   VG Name               an-a04n02_vg0
+   PV Size               378.40 GiB / not usable 3.14 MiB
+  Allocatable           yes
+  PE Size               4.00 MiB
+   Total PE              96870
+  Free PE               96870
+  Allocated PE          0
+  PV UUID               TpEXBC-7822-UGz0-ICz1-AJdg-v5eS-lyB7C5
+  --- Physical volume ---
+  PV Name               /dev/drbd0
+  VG Name               an-a04n01_vg0
+  PV Size               433.70 GiB / not usable 4.41 MiB
+  Allocatable           yes
+  PE Size               4.00 MiB
+  Total PE              111025
+  Free PE               100785
+  Allocated PE          10240
+   PV UUID               RoHAJQ-qrsO-Ofwz-f8W7-jIXd-2cvG-oPgfFR
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-vgscan
+vgdisplay
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-   Reading all physical volumes.  This may take a while...
+   --- Volume group ---
-   Found volume group "an-a03n01_vg0" using metadata type lvm2
+  VG Name               an-a04n02_vg0
+  System ID
+  Format                lvm2
+  Metadata Areas        1
+  Metadata Sequence No  1
+  VG Access             read/write
+  VG Status             resizable
+  Clustered             yes
+  Shared                no
+  MAX LV                0
+  Cur LV                0
+  Open LV               0
+  Max PV                0
+  Cur PV                1
+  Act PV                1
+  VG Size               378.40 GiB
+  PE Size               4.00 MiB
+  Total PE              96870
+  Alloc PE / Size       0 / 0
+  Free  PE / Size       96870 / 378.40 GiB
+  VG UUID               9bTBDu-JSma-kwKR-4oBI-sxi1-YT6i-1uIM4C
+   --- Volume group ---
+  VG Name               an-a04n01_vg0
+  System ID
+  Format                lvm2
+  Metadata Areas        1
+  Metadata Sequence No  2
+  VG Access             read/write
+  VG Status             resizable
+  Clustered             yes
+  Shared                no
+  MAX LV                0
+  Cur LV                1
+  Open LV               0
+  Max PV                0
+  Cur PV                1
+  Act PV                1
+  VG Size               433.69 GiB
+  PE Size               4.00 MiB
+  Total PE              111025
+  Alloc PE / Size       10240 / 40.00 GiB
+  Free  PE / Size       100785 / 393.69 GiB
+  VG UUID               hLnvle-EScm-cP1t-xodO-cKyv-5EyC-TyIpj5
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-lvscan
+lvdisplay
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-   ACTIVE            '/dev/an-a03n01_vg0/shared' [10.00 GiB] inherit
+   --- Logical volume ---
+  LV Path                /dev/an-a04n01_vg0/shared
+  LV Name                shared
+  VG Name                an-a04n01_vg0
+  LV UUID                tvolRF-cb3L-29Dn-Vgqd-e4rf-Qq2e-JFIcbA
+  LV Write Access        read/write
+  LV Creation host, time an-a04n01.alteeve.ca, 2014-06-07 18:54:41 -0400
+  LV Status              available
+  # open                 0
+  LV Size                40.00 GiB
+  Current LE             10240
+  Segments               1
+  Allocation             inherit
+  Read ahead sectors     auto
+  - currently set to     256
+  Block device           253:0
 </syntaxhighlight>
 |}
-Format the <span class="code">/dev/an-a03n01_vg0/shared</span>;
+=== Create the /shared GFS2 filesystem ===
+Format the <span class="code">/dev/an-a04n01_vg0/shared</span> logical volume as a GFS2 filesystem;
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-mkfs.gfs2 -j 2 -p lock_dlm -t an-cluster-03:shared /dev/an-a03n01_vg0/shared
+mkfs.gfs2 -j 2 -p lock_dlm -t an-anvil-04:shared /dev/an-a04n01_vg0/shared
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-/dev/an-a03n01_vg0/shared is a symbolic link to /dev/dm-0
+This will destroy any data on /dev/an-a04n01_vg0/shared.
-This will destroy any data on /dev/dm-0
+It appears to contain: symbolic link to `../dm-0'
-</syntaxhighlight>
-<syntaxhighlight lang="text">
+Are you sure you want to proceed? [y/n] y
-Are you sure you want to proceed? [y/n]y
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Device:                    /dev/an-a03n01_vg0/shared
+Device:                    /dev/an-a04n01_vg0/shared
-Block size:                4096
+Blocksize:                 4096
-Device size:               10.00 GB (2621440 blocks)
+Device Size                40.00 GB (10485760 blocks)
-Filesystem size:           10.00 GB (2621438 blocks)
+Filesystem Size:           40.00 GB (10485758 blocks)
 Journals:                  2
-Resource groups:           40
+Resource Groups:           160
-Locking protocol:          "lock_dlm"
+Locking Protocol:          "lock_dlm"
-Lock table:                "an-cluster-03:shared"
+Lock Table:                "an-anvil-04:shared"
-UUID:                      20bafdb0-1f86-f424-405b-9bf608c0c486
+UUID:                      e07d35fe-6860-f790-38cd-af075366c27b
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
 mkdir /shared
-mount /dev/an-a03n01_vg0/shared /shared
+mount /dev/an-a04n01_vg0/shared /shared
-df -h
+df -hP
 </syntaxhighlight>
 <syntaxhighlight lang="text">
 Filesystem                         Size  Used Avail Use% Mounted on
-/dev/vda3                           18G  5.6G   12G  32% /
+/dev/sda3                           20G  1.5G   18G   8% /
-devtmpfs                           932M     0  932M   0% /dev
+tmpfs                               12G   67M   12G   1% /dev/shm
-tmpfs                              937M   61M  877M   7% /dev/shm
+/dev/sda1                          504M   72M  407M  16% /boot
-tmpfs                              937M  1.9M  935M   1% /run
+/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared
-tmpfs                              937M     0  937M   0% /sys/fs/cgroup
-/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
-/dev/vda1                          484M   83M  401M  18% /boot
-/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared
 </syntaxhighlight>
 |-
-!<span class="code">an-a03n02</span>
+!<span class="code">an-a04n02</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+mkdir /shared
+mount /dev/an-a04n01_vg0/shared /shared
+df -hP
 </syntaxhighlight>
 <syntaxhighlight lang="text">
 Filesystem                         Size  Used Avail Use% Mounted on
-/dev/vda3                           18G  5.6G   12G  32% /
+/dev/sda3                           20G  1.5G   18G   8% /
-devtmpfs                           932M     0  932M   0% /dev
+tmpfs                               12G   52M   12G   1% /dev/shm
-tmpfs                              937M   76M  862M   9% /dev/shm
+/dev/sda1                          504M   72M  407M  16% /boot
-tmpfs                              937M  2.0M  935M   1% /run
+/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared
-tmpfs                              937M     0  937M   0% /sys/fs/cgroup
-/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
-/dev/vda1                          484M   83M  401M  18% /boot
-/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared
 </syntaxhighlight>
 |}
-Shut down <span class="code">gfs2</span>, <span class="code">clvmd</span> and <span class="code">drbd</span> now.
-{|class="wikitable"
-!<span class="code">an-a03n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-umount /shared/
-systemctl stop clvmd.service
-drbdadm down r0
-</syntaxhighlight>
-|-
-!<span class="code">an-a03n02</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-umount /shared/
-systemctl stop clvmd.service
-drbdadm down r0
-</syntaxhighlight>
-|}
-Done.
 = Add Storage to Pacemaker =
@@ Line 1,979: / Line 2,251: @@
 Setup DRBD as a dual-primary resource.
+Notes:
+* Clones allow for a given service to run on multiple nodes.
+** master-max is how many copies of the resource can be promoted to master at the same time across the cluster.
+** master-node-max is how many copies of the resource can be promoted to master on a given node.
+** clone-max is how many copies can run in the cluster, default is to the number of nodes in the cluster.
+** clone-node-max is the number of instances of the resource that can run on each node.
+** notify controls whether other nodes are notified before and after a resource is started or stopped on a given node.
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs cluster cib drbd_cfg
-pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 op monitor interval=60s
+pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 op monitor interval=10s
+pcs -f drbd_cfg resource create drbd_r1 ocf:linbit:drbd drbd_resource=r1 op monitor interval=10s
+### Ignore this for now.
+#pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 \
+#                op monitor interval=29s role=Master \
+#                op monitor interval=31s role=Slave \
+#                op promote interval=0 timeout=90s start-delay=2s \
+#                op start interval=0 timeout=240s \
+#                op stop interval=0 timeout=120s
 pcs -f drbd_cfg resource master drbd_r0_Clone drbd_r0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
+pcs -f drbd_cfg resource master drbd_r1_Clone drbd_r1 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
 pcs cluster cib-push drbd_cfg
 </syntaxhighlight>
@@ Line 1,998: / Line 2,287: @@
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs status
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Cluster name: an-cluster-03
+Cluster name: an-anvil-04
-Last updated: Sun Jan 26 20:26:33 2014
+Last updated: Sat Jun  7 20:29:09 2014
-Last change: Sun Jan 26 20:23:23 2014 via cibadmin on an-a03n01.alteeve.ca
+Last change: Sat Jun  7 20:28:36 2014 via cibadmin on an-a04n01.alteeve.ca
-Stack: corosync
+Stack: cman
-Current DC: an-a03n02.alteeve.ca (2) - partition with quorum
+Current DC: an-a04n01.alteeve.ca - partition with quorum
-Version: 1.1.10-19.el7-368c726
+Version: 1.1.10-14.el6_5.3-368c726
 Nodes configured
 Resources configured
-Online: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
+Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Full list of resources:
-  fence_n01_virsh	(stonith:fence_virsh):	Started an-a03n01.alteeve.ca
+  fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca
-  fence_n02_virsh	(stonith:fence_virsh):	Started an-a03n02.alteeve.ca
+  fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca
   Master/Slave Set: drbd_r0_Clone [drbd_r0]
-      Masters: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
+      Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
+ Master/Slave Set: drbd_r1_Clone [drbd_r1]
-PCSD Status:
+     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-an-a03n01.alteeve.ca:
-  an-a03n01.alteeve.ca: Online
-an-a03n02.alteeve.ca:
-  an-a03n02.alteeve.ca: Online
-Daemon Status:
-  corosync: active/disabled
-  pacemaker: active/disabled
-  pcsd: active/enabled
 </syntaxhighlight>
 |-
-!<span class="code">an-a03n02</span>
+!<span class="code">an-a04n02</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs status
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Cluster name: an-cluster-03
+Cluster name: an-anvil-04
-Last updated: Sun Jan 26 20:26:58 2014
+Last updated: Sat Jun  7 20:29:36 2014
-Last change: Sun Jan 26 20:23:23 2014 via cibadmin on an-a03n01.alteeve.ca
+Last change: Sat Jun  7 20:28:36 2014 via cibadmin on an-a04n01.alteeve.ca
-Stack: corosync
+Stack: cman
-Current DC: an-a03n02.alteeve.ca (2) - partition with quorum
+Current DC: an-a04n01.alteeve.ca - partition with quorum
-Version: 1.1.10-19.el7-368c726
+Version: 1.1.10-14.el6_5.3-368c726
 Nodes configured
 Resources configured
-Online: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
+Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Full list of resources:
-  fence_n01_virsh	(stonith:fence_virsh):	Started an-a03n01.alteeve.ca
+  fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca
-  fence_n02_virsh	(stonith:fence_virsh):	Started an-a03n02.alteeve.ca
+  fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca
   Master/Slave Set: drbd_r0_Clone [drbd_r0]
-      Masters: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
+      Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
+ Master/Slave Set: drbd_r1_Clone [drbd_r1]
+     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
+</syntaxhighlight>
+|}
+=== Configure LVM ===
+We need to have pacemaker activate our clustered LVM LVs on start, and deactivate them when stopping. We don't start/stop clvmd directly because of stop timing issues that can lead to stray fencing.
-PCSD Status:
+{{note|1=This will throw errors if there are no LVs on a given VG... Do not add a volume group until at least one logical volume has been created.}}
-an-a03n01.alteeve.ca:
-  an-a03n01.alteeve.ca: Online
-an-a03n02.alteeve.ca:
-  an-a03n02.alteeve.ca: Online
-Daemon Status:
+{|class="wikitable"
-  corosync: active/disabled
+!<span class="code">an-a04n01</span>
-  pacemaker: active/disabled
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-  pcsd: active/enabled
+pcs cluster cib lvm_cfg
+pcs -f lvm_cfg resource create lvm_n01_vg0 ocf:heartbeat:lvm volgrpname=an-a04n01_vg0 op monitor interval=10s
+pcs -f lvm_cfg resource master lvm_n01_vg0_Clone lvm_n01_vg0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
+pcs cluster cib-push lvm_cfg
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+CIB updated
 </syntaxhighlight>
 |}
-== Configure DLM ==
+== Configure LVM to start after the DRBD PV is Primary ==
+It we stopped here, there is a good chance that on future starts of pacemaker, LVM and DRBD would start in parallel, DRBD would take too long, LVM would error out and stonith's would start to fly. To prevent this, we will tell Pacemaker not to start the LVM resource until after the DRBD resource that is behind the volume group has been promoted to primary.
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs cluster cib dlm_cfg
+pcs cluster cib cst_cfg
-pcs -f dlm_cfg resource create dlm ocf:pacemaker:controld op monitor interval=60s
+pcs -f cst_cfg constraint order promote drbd_r0_Clone then start lvm_n01_vg0_Clone
-pcs -f dlm_cfg resource clone dlm clone-max=2 clone-node-max=1
+</syntaxhighlight>
-pcs cluster cib-push dlm_cfg
+<syntaxhighlight lang="text">
+Adding drbd_r0_Clone lvm_n01_vg0_Clone (kind: Mandatory) (Options: first-action=promote then-action=start)
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+pcs cluster cib-push cst_cfg
 </syntaxhighlight>
 <syntaxhighlight lang="text">
 CIB updated
 </syntaxhighlight>
-|-
+<syntaxhighlight lang="bash">
-!<span class="code">an-a03n02</span>
+pcs constraint show
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs status
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Cluster name: an-cluster-03
+Location Constraints:
-Last updated: Sun Jan 26 20:34:36 2014
+Ordering Constraints:
-Last change: Sun Jan 26 20:33:31 2014 via cibadmin on an-a03n01.alteeve.ca
+  promote drbd_r0_Clone then start lvm_n01_vg0_Clone
-Stack: corosync
+Colocation Constraints:
-Current DC: an-a03n02.alteeve.ca (2) - partition with quorum
-Version: 1.1.10-19.el7-368c726
-Nodes configured
-Resources configured
-Online: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
-Full list of resources:
- fence_n01_virsh	(stonith:fence_virsh):	Started an-a03n01.alteeve.ca
- fence_n02_virsh	(stonith:fence_virsh):	Started an-a03n02.alteeve.ca
- Master/Slave Set: drbd_r0_Clone [drbd_r0]
-     Masters: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
- Clone Set: dlm-clone [dlm]
-     Started: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
-PCSD Status:
-an-a03n01.alteeve.ca:
-  an-a03n01.alteeve.ca: Online
-an-a03n02.alteeve.ca:
-  an-a03n02.alteeve.ca: Online
-Daemon Status:
-  corosync: active/disabled
-  pacemaker: active/disabled
-  pcsd: active/enabled
 </syntaxhighlight>
 |}
-== Configure Cluster LVM ==
+== Configure the /shared GFS2 Partition ==
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs cluster cib clvmd_cfg
+pcs cluster cib fs_cfg
-pcs -f clvmd_cfg resource create clvmd lsb:clvmd params daemon_timeout=30s op monitor interval=60s
+pcs -f fs_cfg resource create sharedFS Filesystem device="/dev/an-a04n01_vg0/shared" directory="/shared" fstype="gfs2"
-pcs -f clvmd_cfg resource clone clvmd clone-max=2 clone-node-max=1
+pcs -f fs_cfg resource clone sharedFS master-max=2 master-node-max=1 clone-max=2 clone-node-max=1
-pcs -f clvmd_cfg constraint colocation add dlm-clone clvmd-clone INFINITY
+pcs cluster cib-push fs_cfg
-pcs -f clvmd_cfg constraint order start dlm then start clvmd-clone
+</syntaxhighlight>
-pcs cluster cib-push clvmd_cfg</syntaxhighlight>
 <syntaxhighlight lang="text">
 CIB updated
 </syntaxhighlight>
-|-
+<syntaxhighlight lang="bash">
-!<span class="code">an-a03n02</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs status
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Cluster name: an-cluster-03
+Cluster name: an-anvil-04
-Last updated: Mon Jan 27 19:00:33 2014
+Last updated: Sat Jun  7 21:09:28 2014
-Last change: Mon Jan 27 19:00:19 2014 via crm_resource on an-a03n01.alteeve.ca
+Last change: Sat Jun  7 21:08:47 2014 via cibadmin on an-a04n01.alteeve.ca
-Stack: corosync
+Stack: cman
-Current DC: an-a03n01.alteeve.ca (1) - partition with quorum
+Current DC: an-a04n01.alteeve.ca - partition with quorum
-Version: 1.1.10-19.el7-368c726
+Version: 1.1.10-14.el6_5.3-368c726
 Nodes configured
 Resources configured
-Online: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
+Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Full list of resources:
-  fence_n01_virsh        (stonith:fence_virsh):  Started an-a03n01.alteeve.ca
+  fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca
-  fence_n02_virsh        (stonith:fence_virsh):  Started an-a03n02.alteeve.ca
+  fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca
   Master/Slave Set: drbd_r0_Clone [drbd_r0]
-      Masters: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
+      Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-  Clone Set: dlm-clone [dlm]
+  Master/Slave Set: drbd_r1_Clone [drbd_r1]
-      Started: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
+      Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-  Clone Set: clvmd-clone [clvmd]
+  Clone Set: sharedFS-clone [sharedFS]
-      Started: [ an-a03n01.alteeve.ca an-a03n02.alteeve.ca ]
+      Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-PCSD Status:
-an-a03n01.alteeve.ca:
-  an-a03n01.alteeve.ca: Online
-an-a03n02.alteeve.ca:
-  an-a03n02.alteeve.ca: Online
-Daemon Status:
-  corosync: active/disabled
-  pacemaker: active/disabled
-  pcsd: active/enabled
-</syntaxhighlight>
-|}
-== Configure the /shared GFS2 Partition ==
-{|class="wikitable"
-!<span class="code">an-a03n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs cluster cib fs_cfg
-pcs -f fs_cfg resource create sharedFS Filesystem device="/dev/an-a03n01_vg0/shared" directory="/shared" fstype="gfs2"
-pcs -f fs_cfg resource clone sharedFS
-pcs cluster cib-push fs_cfg
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-CIB updated
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-df -h
+df -hP
 </syntaxhighlight>
 <syntaxhighlight lang="text">
 Filesystem                         Size  Used Avail Use% Mounted on
-/dev/vda3                           18G  5.6G   12G  32% /
+/dev/sda3                           20G  1.5G   18G   8% /
-devtmpfs                           932M     0  932M   0% /dev
+tmpfs                               12G   67M   12G   1% /dev/shm
-tmpfs                              937M   61M  877M   7% /dev/shm
+/dev/sda1                          504M   72M  407M  16% /boot
-tmpfs                              937M  2.2M  935M   1% /run
+/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared
-tmpfs                              937M     0  937M   0% /sys/fs/cgroup
-/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
-/dev/vda1                          484M   83M  401M  18% /boot
-/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared
 </syntaxhighlight>
 |-
-!<span class="code">an-a03n02</span>
+!<span class="code">an-a04n02</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 df -h
@@ Line 2,214: / Line 2,448: @@
 <syntaxhighlight lang="text">
 Filesystem                         Size  Used Avail Use% Mounted on
-/dev/vda3                           18G  5.6G   12G  32% /
+/dev/sda3                           20G  1.5G   18G   8% /
-devtmpfs                           932M     0  932M   0% /dev
+tmpfs                               12G   52M   12G   1% /dev/shm
-tmpfs                              937M   76M  862M   9% /dev/shm
+/dev/sda1                          504M   72M  407M  16% /boot
-tmpfs                              937M  2.6M  935M   1% /run
+/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared
-tmpfs                              937M     0  937M   0% /sys/fs/cgroup
-/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
-/dev/vda1                          484M   83M  401M  18% /boot
-/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared
 </syntaxhighlight>
 |}
-== Configuring Constraints ==
+== Configur /shared to start after LVM ==
+As we did before in making sure LVM started after DRBD, this time we will make sure LVM starts before /shared is mounted.
 {|class="wikitable"
-!<span class="code">an-a03n01</span>
+!<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs cluster cib cst_cfg
-pcs -f cst_cfg constraint order start dlm then promote drbd_r0_Clone
+pcs -f cst_cfg constraint order start lvm_n01_vg0_Clone then start sharedFS
-pcs -f cst_cfg constraint order promote drbd_r0_Clone then start clvmd-clone
-pcs -f cst_cfg constraint order promote clvmd-clone then start sharedFS-clone
-pcs cluster cib-push cst_cfg
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-CIB updated
+Adding lvm_n01_vg0_Clone sharedFS (kind: Mandatory) (Options: first-action=start then-action=start)
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-pcs constraint show
+pcs cluster cib-push cst_cfg
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Location Constraints:
+CIB updated
-Ordering Constraints:
-  start dlm then promote drbd_r0_Clone
-  promote drbd_r0_Clone then start clvmd-clone
-  start clvmd-clone then start sharedFS-clone
-Colocation Constraints:
 </syntaxhighlight>
-|-
+<syntaxhighlight lang="bash">
-!<span class="code">an-a03n02</span>
+pcs constraint show --full
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs constraint show
 </syntaxhighlight>
 <syntaxhighlight lang="text">
 Location Constraints:
 Ordering Constraints:
-  start dlm then promote drbd_r0_Clone
+   promote drbd_r0_Clone then start lvm_n01_vg0_Clone (Mandatory) (id:order-drbd_r0_Clone-lvm_n01_vg0_Clone-mandatory)
-   promote drbd_r0_Clone then start clvmd-clone
+   start lvm_n01_vg0_Clone then start sharedFS-clone (Mandatory) (id:order-lvm_n01_vg0_Clone-sharedFS-clone-mandatory)
-   start clvmd-clone then start sharedFS-clone
 Colocation Constraints:
 </syntaxhighlight>
 |}
-= Odds and Sods =
+Note that this time we added '--full'. If you ever need to delete a constraint, you would use 'pcs constraint delete <id>'.
-This is a section for random notes. The stuff here will be integrated into the finished tutorial or removed.
-== Determine multicast Address ==
-Useful if you need to ensure that your switch has persistent multicast addresses set.
-<syntaxhighlight lang="bash">
-corosync-cmapctl | grep mcastaddr
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-totem.interface.0.mcastaddr (str) = 239.192.122.199
-</syntaxhighlight>
-<span class="code"></span>
-<syntaxhighlight lang="bash">
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-</syntaxhighlight>
-<syntaxhighlight lang="diff">
-</syntaxhighlight>
 = Notes =

Anvil! Tutorial 3 on EL6: Difference between revisions

Latest revision as of 12:23, 21 July 2014

Before We Begin

OS Setup

Post OS Install

Adding AN! Repo

Install

Setup Networking

Making ssh faster when the net is down

Setting the Hostname

Setup The hosts File

Setup SSH

Populating And Pushing ~/.ssh/known_hosts

Configuring the Firewall

Keeping Time in Sync

Configuring the Anvil!

Configuring cman

Starting Pacemaker

Configure and test stonith (aka fencing)

Configuring IPMI Fencing

Configuring Fence Levels

Test Fencing

Shared Storage

DRBD

Partition Storage

Configure DRBD

Start DRBD for the first time

Clustered LVM and GFS2

Configure lvm.conf

Start clvmd

Create Initial PVs, VGs and the /shared LV

Create the /shared GFS2 filesystem

Add Storage to Pacemaker

Configure Dual-Primary DRBD

Configure LVM

Configure LVM to start after the DRBD PV is Primary

Configure the /shared GFS2 Partition

Configur /shared to start after LVM

Notes

Thanks

Navigation menu