Latest revision as of 12:23, 21 July 2014

AN!Wiki :: How To :: Anvil! Tutorial 3 on EL6

Warning: This tutorial is incomplete, flawed and generally sucks at this time. Do not follow this and expect anything to work. In large part, it's a dumping ground for notes and little else. This warning will be removed when the tutorial is completed.

This is the third Anvil! tutorial built on Red Hat's Enterprise Linux 6.5 and newer. It is meant to be a stop-gap / learning cluster before RHEL 7 is released and stabilized.

Before We Begin

This tutorial does not require prior Anvil! experience (or any clustering experience), but it does expect a certain familiarity with Linux and a low-intermediate understanding of networking. Where possible, steps are explained in detail and rationale is provided for why certain decisions are made.

For those with Anvil! experience;

Please be careful not to skip too much. There are some major and some subtle changes from previous tutorials.

OS Setup

Warning: RHEL v6.5 or newer is required.

Post OS Install

Stuff.

If you're using RHEL proper, register your nodes with RHN.

Note: You need to replace $user and $pass with your RHN account details.

an-a04n01

rhnreg_ks --username "$user" --password "$pass" --force --profilename "an-a04n01.alteeve.ca"
rhn-channel --add --user "$user" --password "$pass" --channel=rhel-x86_64-server-rs-6

an-a04n02

rhnreg_ks --username "$user" --password "$pass" --force --profilename "an-a04n02.alteeve.ca"
rhn-channel --add --user "$user" --password "$pass" --channel=rhel-x86_64-server-rs-6

Adding AN! Repo

AN! offers a new repo with a few RPMs not in stock EL 6 distros.

an-a04n01

an-a04n02

cat <<-END>/etc/yum.repos.d/an.repo
[an-repo]
name=AN! Repo for Anvil! stuff
baseurl=https://alteeve.ca/repo/el6/
enabled=1
gpgcheck=0
protect=1
END

yum clean all

cat <<-END>/etc/yum.repos.d/an.repo
[an-repo]
name=AN! Repo for Anvil! stuff
baseurl=https://alteeve.ca/repo/el6/
enabled=1
gpgcheck=0
protect=1
END

yum clean all

Done.

Install

Not all of these are required, but most are used at one point or another in this tutorial.

Note: The fence-agents-virsh package is not available in RHEL 7 beta. Further, it's only needed if you're building your Anvil! using VMs.

an-a04n01

an-a04n02

yum -y update
yum -y install bridge-utils vim pacemaker corosync cman gfs2-utils \
               ccs pcs ipmitool OpenIPMI lvm2-cluster drbd84-utils \
               drbd84-kmod
chkconfig ipmi on
chkconfig acpid off
chkconfig kdump off
chkconfig drbd off
/etc/init.d/ipmi start
/etc/init.d/acpid stop
/etc/init.d/kdump stop
/etc/init.d/drbd stop

# same as an-a04n01

Setup Networking

TODO: Explain this.

Remap all NICs to have purpose-based names.

IFN Bridge

an-a04n01

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-ifn-bridge1

# Internet-Facing Network - Bridge
DEVICE="ifn-bridge1"
TYPE="Bridge"
BOOTPROTO="none"
IPADDR="10.255.40.1"
NETMASK="255.255.0.0"
GATEWAY="10.255.255.254"
DNS1="8.8.8.8"
DNS2="8.8.4.4"
DEFROUTE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-ifn-bridge1

# Internet-Facing Network - Bridge
DEVICE="ifn-bridge1"
TYPE="Bridge"
BOOTPROTO="none"
IPADDR="10.255.40.2"
NETMASK="255.255.0.0"
GATEWAY="10.255.255.254"
DNS1="8.8.8.8"
DNS2="8.8.4.4"
DEFROUTE="yes"

IFN Bond

an-a04n01

vim /etc/sysconfig/network-scripts/ifcfg-ifn-bond1

# Internet-Facing Network - Bond
DEVICE="ifn-bond1"
BRIDGE="ifn-bridge1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=ifn-link1"

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-ifn-bond1

# Internet-Facing Network - Bond
DEVICE="ifn-bond1"
BRIDGE="ifn-bridge1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=ifn-link1"

IFN Links

an-a04n01

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-ifn-link1

# Internet-Facing Network - Link 1
HWADDR="00:1B:21:81:C3:34"
DEVICE="ifn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="ifn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-ifn-link2

# Internet-Facing Network - Link 2
HWADDR="A0:36:9F:02:E0:05"
DEVICE="ifn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="ifn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-ifn-link1

# Internet-Facing Network - Link 1
HWADDR="00:1B:21:81:C2:EA"
DEVICE="ifn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="ifn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-ifn-link2

# Internet-Facing Network - Link 2
HWADDR="A0:36:9F:07:D6:2F"
DEVICE="ifn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="ifn-bond1"
SLAVE="yes"

SN Bond

an-a04n01

vim /etc/sysconfig/network-scripts/ifcfg-sn-bond1

# Storage Network - Bond
DEVICE="sn-bond1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=sn-link1"
IPADDR="10.10.40.1"
NETMASK="255.255.0.0"

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-sn-bond1

# Storage Network - Bond
DEVICE="sn-bond1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=sn-link1"
IPADDR="10.10.40.2"
NETMASK="255.255.0.0"

SN Links

an-a04n01

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-sn-link1

# Storage Network - Link 1
HWADDR="00:19:99:9C:9B:9F"
DEVICE="sn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="sn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-sn-link2

# Storage Network - Link 2
HWADDR="A0:36:9F:02:E0:04"
DEVICE="sn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="sn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-sn-link1

# Storage Network - Link 1
HWADDR="00:19:99:9C:A0:6D"
DEVICE="sn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="sn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-sn-link2

# Storage Network - Link 2
HWADDR="A0:36:9F:07:D6:2E"
DEVICE="sn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="sn-bond1"
SLAVE="yes"

BCN Bond

an-a04n01

vim /etc/sysconfig/network-scripts/ifcfg-bcn-bond1

# Back-Channel Network - Bond
DEVICE="bcn-bond1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=bcn-link1"
IPADDR="10.20.40.1"
NETMASK="255.255.0.0"

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-bcn-bond1

# Back-Channel Network - Bond
DEVICE="bcn-bond1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=bcn-link1"
IPADDR="10.20.40.2"
NETMASK="255.255.0.0"

BCN Links

an-a04n01

an-a04n02

vim /etc/sysconfig/network-scripts/ifcfg-bcn-link1

# Back-Channel Network - Link 1
HWADDR="00:19:99:9C:9B:9E"
DEVICE="bcn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="bcn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-bcn-link2

# Back-Channel Network - Link 2
HWADDR="00:1B:21:81:C3:35"
DEVICE="bcn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="bcn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-bcn-link1

# Back-Channel Network - Link 1
HWADDR="00:19:99:9C:A0:6C"
DEVICE="bcn-link1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="bcn-bond1"
SLAVE="yes"

vim /etc/sysconfig/network-scripts/ifcfg-bcn-link2

# Back-Channel Network - Link 2
HWADDR="00:1B:21:81:C2:EB"
DEVICE="bcn-link2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
MASTER="bcn-bond1"
SLAVE="yes"

Making ssh faster when the net is down

By default, the nodes will try to resolve the host name of an incoming ssh connection. When the internet connection is down, DNS lookups have to time out, which can make login times quite slow. When something goes wrong, seconds count and waiting for up to a minute for an SSH password prompt can be maddening.

For this reason, we will make two changes to /etc/ssh/sshd_config that disable this login delay.

Please be aware that this can reduce security. If this is a concern, skip this step.

an-a04n01

sed -i.anvil 's/#GSSAPIAuthentication no/GSSAPIAuthentication no/' /etc/ssh/sshd_config
sed -i 's/GSSAPIAuthentication yes/#GSSAPIAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
systemctl restart sshd.service
diff -u /etc/ssh/sshd_config.anvil /etc/ssh/sshd_config

--- /etc/ssh/sshd_config.anvil	2013-09-30 03:08:17.000000000 -0400
+++ /etc/ssh/sshd_config	2014-05-28 00:35:30.954000741 -0400
@@ -77,8 +77,8 @@
 #KerberosUseKuserok yes
 
 # GSSAPI options
-#GSSAPIAuthentication no
-GSSAPIAuthentication yes
+GSSAPIAuthentication no
+#GSSAPIAuthentication yes
 #GSSAPICleanupCredentials yes
 GSSAPICleanupCredentials yes
 #GSSAPIStrictAcceptorCheck yes
@@ -119,7 +119,7 @@
 #ClientAliveInterval 0
 #ClientAliveCountMax 3
 #ShowPatchLevel no
-#UseDNS yes
+UseDNS no
 #PidFile /var/run/sshd.pid
 #MaxStartups 10:30:100
 #PermitTunnel no

an-a04n02

sed -i.anvil 's/#GSSAPIAuthentication no/GSSAPIAuthentication no/' /etc/ssh/sshd_config
sed -i 's/GSSAPIAuthentication yes/#GSSAPIAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
systemctl restart sshd.service
diff -u /etc/ssh/sshd_config.anvil /etc/ssh/sshd_config

--- /etc/ssh/sshd_config.anvil	2013-09-30 03:08:17.000000000 -0400
+++ /etc/ssh/sshd_config	2014-05-28 00:35:33.016999110 -0400
@@ -77,8 +77,8 @@
 #KerberosUseKuserok yes
 
 # GSSAPI options
-#GSSAPIAuthentication no
-GSSAPIAuthentication yes
+GSSAPIAuthentication no
+#GSSAPIAuthentication yes
 #GSSAPICleanupCredentials yes
 GSSAPICleanupCredentials yes
 #GSSAPIStrictAcceptorCheck yes
@@ -119,7 +119,7 @@
 #ClientAliveInterval 0
 #ClientAliveCountMax 3
 #ShowPatchLevel no
-#UseDNS yes
+UseDNS no
 #PidFile /var/run/sshd.pid
 #MaxStartups 10:30:100
 #PermitTunnel no

Subsequent logins when the net is down should be quick.

Setting the Hostname

TODO

Setup The hosts File

You can use DNS if you prefer. For now, lets use /etc/hosts for node name resolution.

an-a04n01

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

### Anvil! systems
# Anvil! 03, Node 01
10.20.40.1	an-a04n01.bcn an-a04n01 an-a04n01.alteeve.ca
10.20.41.1	an-a04n01.ipmi
10.10.40.1	an-a04n01.sn
10.255.40.1	an-a04n01.ifn

# Anvil! 03, Node 02
10.20.40.2	an-a04n02.bcn an-a04n02 an-a04n02.alteeve.ca
10.20.41.2	an-a04n02.ipmi
10.10.40.2	an-a04n02.sn
10.255.40.2	an-a04n02.ifn

### Foundation Pack
# Network Switches
10.20.1.1	an-s01 an-s01.alteeve.ca
10.20.1.2	an-s02 an-s02.alteeve.ca	# Only accessible when out of the stack
 
# Switched PDUs
10.20.2.1	an-p01 an-p01.alteeve.ca
10.20.2.2	an-p02 an-p02.alteeve.ca
 
# Network-monitored UPSes
10.20.3.1	an-u01 an-u01.alteeve.ca
10.20.3.2	an-u02 an-u02.alteeve.ca
 
### Monitor Packs
10.20.4.1	an-m01 an-m01.alteeve.ca
10.255.4.1	an-m01.ifn
10.20.4.2	an-m02 an-m02.alteeve.ca
10.255.4.2	an-m02.ifn

an-a04n02

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

### Anvil! systems
# Anvil! 03, Node 01
10.20.40.1	an-a04n01.bcn an-a04n01 an-a04n01.alteeve.ca
10.20.41.1	an-a04n01.ipmi
10.10.40.1	an-a04n01.sn
10.255.40.1	an-a04n01.ifn

# Anvil! 03, Node 02
10.20.40.2	an-a04n02.bcn an-a04n02 an-a04n02.alteeve.ca
10.20.41.2	an-a04n02.ipmi
10.10.40.2	an-a04n02.sn
10.255.40.2	an-a04n02.ifn

### Foundation Pack
# Network Switches
10.20.1.1	an-s01 an-s01.alteeve.ca
10.20.1.2	an-s02 an-s02.alteeve.ca	# Only accessible when out of the stack
 
# Switched PDUs
10.20.2.1	an-p01 an-p01.alteeve.ca
10.20.2.2	an-p02 an-p02.alteeve.ca
 
# Network-monitored UPSes
10.20.3.1	an-u01 an-u01.alteeve.ca
10.20.3.2	an-u02 an-u02.alteeve.ca
 
### Monitor Packs
10.20.4.1	an-m01 an-m01.alteeve.ca
10.255.4.1	an-m01.ifn
10.20.4.2	an-m02 an-m02.alteeve.ca
10.255.4.2	an-m02.ifn

Setup SSH

Same as before.

Populating And Pushing ~/.ssh/known_hosts

an-a04n01

ssh-keygen -t rsa -N "" -b 8191 -f ~/.ssh/id_rsa

Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
f9:41:7e:aa:96:8e:fa:47:79:f5:3a:33:89:c3:9a:4b root@an-a04n01.alteeve.ca
The key's randomart image is:
+--[ RSA 8191]----+
|                 |
|                 |
|          .      |
|         +  .    |
|        S.o...   |
|        o..+  .  |
|       .E+o. o   |
|       o+o+ *    |
|    .oo+*o . +   |
+-----------------+

an-a04n01

ssh-keygen -t rsa -N "" -b 8191 -f ~/.ssh/id_rsa

Generating public/private rsa key pair.
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
3f:1a:02:17:44:10:5e:6f:2b:98:44:09:e5:e0:ea:4b root@an-a04n02.alteeve.ca
The key's randomart image is:
+--[ RSA 8191]----+
|  oo==+          |
| . =.o .         |
|  . + . o        |
| . . o o .       |
|.   + o S        |
|.    o . .       |
| E    . . o      |
|. .    . o .     |
| .      .        |
+-----------------+

Setup autorized_keys:

an-a04n01

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh root@an-a04n02 "cat /root/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys

The authenticity of host 'an-a04n02 (10.20.40.2)' can't be established.
RSA key fingerprint is 22:09:7b:0c:8b:d8:80:08:80:6d:0e:bc:fb:5a:e1:de.
Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'an-a04n02,10.20.40.2' (RSA) to the list of known hosts.

root@an-a04n02's password:

Populate ~/.ssh/known_hosts:

an-a04n01

ssh-keyscan an-a04n01.alteeve.ca >> ~/.ssh/known_hosts

# an-a04n01.alteeve.ca SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n01 >> ~/.ssh/known_hosts

# an-a04n01 SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n01.bcn >> ~/.ssh/known_hosts

# an-a04n01.bcn SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n01.sn >> ~/.ssh/known_hosts

# an-a04n01.sn SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n01.ifn >> ~/.ssh/known_hosts

# an-a04n01.ifn SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n02.alteeve.ca >> ~/.ssh/known_hosts

# an-a04n02.alteeve.ca SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n02 >> ~/.ssh/known_hosts

# an-a04n02 SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n02.bcn >> ~/.ssh/known_hosts

# an-a04n02.bcn SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n02.sn >> ~/.ssh/known_hosts

# an-a04n02.sn SSH-2.0-OpenSSH_5.3

ssh-keyscan an-a04n02.ifn >> ~/.ssh/known_hosts

# an-a04n02.ifn SSH-2.0-OpenSSH_5.3

Now copy the files to the second node:

an-a04n01

rsync -av ~/.ssh/authorized_keys root@an-a04n02:/root/.ssh/

root@an-a04n02's password:

sending incremental file list
authorized_keys

sent 2937 bytes  received 31 bytes  1187.20 bytes/sec
total size is 2854  speedup is 0.96

rsync -av ~/.ssh/known_hosts root@an-a04n02:/root/.ssh/

sending incremental file list
known_hosts

sent 4829 bytes  received 31 bytes  9720.00 bytes/sec
total size is 4750  speedup is 0.98

Note that there was no password prompt the second time. Hoozah!

Configuring the Firewall

an-a04n01

# cman (corosync's totem)
iptables -I INPUT -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 -d 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
iptables -I INPUT -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 --dports 5404,5405 -j ACCEPT

# dlm
iptables -I INPUT -m state --state NEW -p tcp -s 10.20.0.0/16 -d 10.20.0.0/16 --dport 21064 -j ACCEPT 

# DRBD resource 0 and 1 - on the SN
iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7788 -j ACCEPT
iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7789 -j ACCEPT

# Make the new rules persistent.
/etc/init.d/iptables save

iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]

an-a04n01

# cman (corosync's totem)
iptables -I INPUT -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 -d 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
iptables -I INPUT -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 --dports 5404,5405 -j ACCEPT

# dlm
iptables -I INPUT -m state --state NEW -p tcp -s 10.20.0.0/16 -d 10.20.0.0/16 --dport 21064 -j ACCEPT 

# DRBD resource 0 and 1 - on the SN
iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7788 -j ACCEPT
iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7789 -j ACCEPT

# Make the new rules persistent.
/etc/init.d/iptables save

iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]

Keeping Time in Sync

It's not as critical as it used to be to keep the clocks on the nodes in sync, but it's still a good idea.

an-a04n01	chkconfig ntpd on /etc/init.d/ntpd start Starting ntpd: [ OK ]
an-a04n01	chkconfig ntpd on /etc/init.d/ntpd start Starting ntpd: [ OK ]

Configuring the Anvil!

Now we're getting down to business!

For this section, we will be working on an-a04n01 and using ssh to perform tasks on an-a04n02.

Note: TODO: explain what this is and how it works.

Configuring cman

With RHEL 6, we do not need to configure corosync directly. We will create a "skeleton" cluster.conf file which will, in turn, handle corosync for us. Once configured and the configuration has been copied to the peer, we will start pacemaker and it will handle starting (and stopping) pacemaker and corosync for us.

We will use 'ccs' to configure the skeleton cluster.conf file.

an-a04n01

ccs -f /etc/cluster/cluster.conf --createcluster an-anvil-04
ccs -f /etc/cluster/cluster.conf --setcman two_node="1" expected_votes="1"
ccs -f /etc/cluster/cluster.conf --addnode an-a04n01.alteeve.ca
ccs -f /etc/cluster/cluster.conf --addnode an-a04n02.alteeve.ca
ccs -f /etc/cluster/cluster.conf --addfencedev pcmk agent=fence_pcmk 
ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect an-a04n01.alteeve.ca
ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect an-a04n02.alteeve.ca
ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk an-a04n01.alteeve.ca pcmk-redirect port=an-a04n01.alteeve.ca
ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk an-a04n02.alteeve.ca pcmk-redirect port=an-a04n02.alteeve.ca
ccs -f /etc/cluster/cluster.conf --setfencedaemon post_join_delay="30"
cat /etc/cluster/cluster.conf

<cluster config_version="10" name="an-anvil-04">
  <fence_daemon post_join_delay="30"/>
  <clusternodes>
    <clusternode name="an-a04n01.alteeve.ca" nodeid="1">
      <fence>
        <method name="pcmk-redirect">
          <device name="pcmk" port="an-a04n01.alteeve.ca"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="an-a04n02.alteeve.ca" nodeid="2">
      <fence>
        <method name="pcmk-redirect">
          <device name="pcmk" port="an-a04n02.alteeve.ca"/>
        </method>
      </fence>
    </clusternode>
  </clusternodes>
  <cman expected_votes="1" two_node="1"/>
  <fencedevices>
    <fencedevice agent="fence_pcmk" name="pcmk"/>
  </fencedevices>
  <rm>
    <failoverdomains/>
    <resources/>
  </rm>
</cluster>

Copy it to an-a04n02;

an-a04n01

rsync -av /etc/cluster/cluster.conf root@an-a04n02:/etc/cluster/

sending incremental file list
cluster.conf

sent 838 bytes  received 31 bytes  579.33 bytes/sec
total size is 758  speedup is 0.87

an-a04n02

cat /etc/cluster/cluster.conf

<cluster config_version="10" name="an-anvil-04">
  <fence_daemon post_join_delay="30"/>
  <clusternodes>
    <clusternode name="an-a04n01.alteeve.ca" nodeid="1">
      <fence>
        <method name="pcmk-redirect">
          <device name="pcmk" port="an-a04n01.alteeve.ca"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="an-a04n02.alteeve.ca" nodeid="2">
      <fence>
        <method name="pcmk-redirect">
          <device name="pcmk" port="an-a04n02.alteeve.ca"/>
        </method>
      </fence>
    </clusternode>
  </clusternodes>
  <cman expected_votes="1" two_node="1"/>
  <fencedevices>
    <fencedevice agent="fence_pcmk" name="pcmk"/>
  </fencedevices>
  <rm>
    <failoverdomains/>
    <resources/>
  </rm>
</cluster>

Starting Pacemaker

Now start pacemaker proper.

an-a04n01

/etc/init.d/pacemaker start

Starting cluster: 
   Checking if cluster has been disabled at boot...        [  OK  ]
   Checking Network Manager...                             [  OK  ]
   Global setup...                                         [  OK  ]
   Loading kernel modules...                               [  OK  ]
   Mounting configfs...                                    [  OK  ]
   Starting cman...                                        [  OK  ]
   Waiting for quorum...                                   [  OK  ]
   Starting fenced...                                      [  OK  ]
   Starting dlm_controld...                                [  OK  ]
   Tuning DLM kernel config...                             [  OK  ]
   Starting gfs_controld...                                [  OK  ]
   Unfencing self...                                       [  OK  ]
   Joining fence domain...                                 [  OK  ]
Starting Pacemaker Cluster Manager                         [  OK  ]

an-a04n02

/etc/init.d/pacemaker start

Starting cluster: 
   Checking if cluster has been disabled at boot...        [  OK  ]
   Checking Network Manager...                             [  OK  ]
   Global setup...                                         [  OK  ]
   Loading kernel modules...                               [  OK  ]
   Mounting configfs...                                    [  OK  ]
   Starting cman...                                        [  OK  ]
   Waiting for quorum...                                   [  OK  ]
   Starting fenced...                                      [  OK  ]
   Starting dlm_controld...                                [  OK  ]
   Tuning DLM kernel config...                             [  OK  ]
   Starting gfs_controld...                                [  OK  ]
   Unfencing self...                                       [  OK  ]
   Joining fence domain...                                 [  OK  ]
Starting Pacemaker Cluster Manager                         [  OK  ]

Verify pacemaker proper started as expected.

an-a04n01

pcs status

Cluster name: an-anvil-04
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Wed May 28 20:59:33 2014
Last change: Wed May 28 20:59:18 2014 via crmd on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
0 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

an-a04n02

pcs status

WARNING: no stonith devices and stonith-enabled is not false
Last updated: Wed May 28 20:59:29 2014
Last change: Wed May 28 20:59:18 2014 via crmd on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
0 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

Note the error about stonith. We will address that momentarily.

Configure and test stonith (aka fencing)

We will use IPMI and PDU based fence devices with STONITH levels.

You can see the list of available fence agents here. You will need to find the one for your hardware fence devices.

Note: Ignore the errors.

an-a04n01

pcs stonith list

fence_apc - Fence agent for APC over telnet/ssh
fence_apc_snmp - Fence agent for APC over SNMP
fence_bladecenter - Fence agent for IBM BladeCenter
fence_bladecenter_snmp - Fence agent for IBM BladeCenter over SNMP
fence_brocade - Fence agent for Brocade over telnet
Error: no metadata for /usr/sbin/fence_check
fence_cisco_mds - Fence agent for Cisco MDS
fence_cisco_ucs - Fence agent for Cisco UCS
fence_drac - fencing agent for Dell Remote Access Card
fence_drac5 - Fence agent for Dell DRAC CMC/5
fence_eaton_snmp - Fence agent for Eaton over SNMP
fence_egenera - I/O Fencing agent for the Egenera BladeFrame
fence_eps - Fence agent for ePowerSwitch
fence_hpblade - Fence agent for HP BladeSystem
fence_ibmblade - Fence agent for IBM BladeCenter over SNMP
fence_idrac - Fence agent for IPMI over LAN
fence_ifmib - Fence agent for IF MIB
fence_ilo - Fence agent for HP iLO
fence_ilo2 - Fence agent for HP iLO
fence_ilo3 - Fence agent for IPMI over LAN
fence_ilo4 - Fence agent for IPMI over LAN
fence_ilo_mp - Fence agent for HP iLO MP
fence_imm - Fence agent for IPMI over LAN
fence_intelmodular - Fence agent for Intel Modular
fence_ipdu - Fence agent for iPDU over SNMP
fence_ipmilan - Fence agent for IPMI over LAN
fence_kdump - Fence agent for use with kdump
Error: no metadata for /usr/sbin/fence_node
fence_rhevm - Fence agent for RHEV-M REST API
fence_rsa - Fence agent for IBM RSA
fence_rsb - I/O Fencing agent for Fujitsu-Siemens RSB
fence_sanbox2 - Fence agent for QLogic SANBox2 FC switches
fence_scsi - fence agent for SCSI-3 persistent reservations
Error: no metadata for /usr/sbin/fence_tool
fence_virsh - Fence agent for virsh
fence_virt - Fence agent for virtual machines
fence_vmware - Fence agent for VMWare
fence_vmware_soap - Fence agent for VMWare over SOAP API
fence_wti - Fence agent for WTI
fence_xvm - Fence agent for virtual machines

We will use fence_ipmilan and fence_apc_snmp.

Configuring IPMI Fencing

Setup out IPMI BMCs (on LAN channel 2 and using user ID 2).

an-a04n01

ipmitool lan set 2 ipsrc static
ipmitool lan set 2 ipaddr 10.20.41.1
ipmitool lan set 2 netmask 255.255.0.0
ipmitool lan set 2 defgw ipaddr 10.20.255.254
ipmitool user set password 2 Initial1

an-a04n02

ipmitool lan set 2 ipsrc static
ipmitool lan set 2 ipaddr 10.20.41.2
ipmitool lan set 2 netmask 255.255.0.0
ipmitool lan set 2 defgw ipaddr 10.20.255.254
ipmitool user set password 2 Initial1

Test the new settings (using the hostnames we set in /etc/hosts):

an-a04n01

fence_ipmilan -a an-a04n02.ipmi -l admin -p Initial1 -o status

Getting status of IPMI:an-a04n02.ipmi...Chassis power = On
Done

an-a04n02

fence_ipmilan -a an-a04n01.ipmi -l admin -p Initial1 -o status

Getting status of IPMI:an-a04n01.ipmi...Chassis power = On
Done

Good, now we can configure IPMI fencing.

Every fence agent has a possibly unique subset of options that can be used. You can see a brief description of these options with the pcs stonith describe fence_X command. Let's look at the options available for fence_ipmilan.

an-a04n01

pcs stonith describe fence_ipmilan

Stonith options for: fence_ipmilan
  auth: IPMI Lan Auth type (md5, password, or none)
  ipaddr: IPMI Lan IP to talk to
  passwd: Password (if required) to control power on IPMI device
  passwd_script: Script to retrieve password (if required)
  lanplus: Use Lanplus to improve security of connection
  login: Username/Login (if required) to control power on IPMI device
  action: Operation to perform. Valid operations: on, off, reboot, status, list, diag, monitor or metadata
  timeout: Timeout (sec) for IPMI operation
  cipher: Ciphersuite to use (same as ipmitool -C parameter)
  method: Method to fence (onoff or cycle)
  power_wait: Wait X seconds after on/off operation
  delay: Wait X seconds before fencing is started
  privlvl: Privilege level on IPMI device
  verbose: Verbose mode
  stonith-timeout: How long to wait for the STONITH action to complete per a stonith device.
  priority: The priority of the stonith resource. Devices are tried in order of highest priority to lowest.
  pcmk_host_map: A mapping of host names to ports numbers for devices that do not support host names.
  pcmk_host_list: A list of machines controlled by this device (Optional unless pcmk_host_check=static-list).
  pcmk_host_check: How to determin which machines are controlled by the device.

One of the nice things about pcs is that it allows us to create a test file to prepare all our changes in. Then, when we're happy with the changes, merge them into the running cluster. So let's make a copy called stonith_cfg

Now add IPMI fencing.

an-a04n01

pcs cluster cib stonith_cfg
#   work in our temp file         unique name    fence agent   target node                           device addr             options
pcs -f stonith_cfg stonith create fence_n01_ipmi fence_ipmilan pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-a04n01.ipmi" action="reboot" login="admin" passwd="Initial1" delay=15 op monitor interval=10s
pcs -f stonith_cfg stonith create fence_n02_ipmi fence_ipmilan pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-a04n02.ipmi" action="reboot" login="admin" passwd="Initial1" op monitor interval=10s
pcs cluster cib-push stonith_cfg

Note that fence_n01_ipmi has a delay=15 set but fence_n02_ipmi does not. If the network connection breaks between the two nodes, they will both try to fence each other at the same time. If acpid is running, the slower node will not die right away. It will continue to run for up to four more seconds, ample time for it to also initiate a fence against the faster node. The end result is that both nodes get fenced. The ten-second delay protects against this by causing an-a04n02 to pause for 10 seconds before initiating a fence against an-a04n01. If both nodes are alive, an-a04n02 will power off before the 10 seconds pass, so it will never fence an-a04n01. However, if an-a04n01 really is dead, after the ten seconds have elapsed, fencing will proceed as normal.

NOTE: Get my PDUs back and use them here!

We can check the new configuration now;

an-a04n01

pcs status

Cluster name: an-anvil-04
Last updated: Wed May 28 22:01:14 2014
Last change: Wed May 28 21:55:59 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
2 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca 
 fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca

Tell pacemaker to use fencing;

an-a04n01

pcs property set stonith-enabled=true
pcs property set no-quorum-policy=ignore
pcs property

Cluster Properties:
 cluster-infrastructure: cman
 dc-version: 1.1.10-14.el6_5.3-368c726
 no-quorum-policy: ignore
 stonith-enabled: true

Excellent!

Configuring Fence Levels

TODO...

Test Fencing

ToDo: Kill each node with echo c > /proc/sysrq-trigger and make sure the other node fences it.

Shared Storage

DRBD -> Clustered LVM -> GFS2

DRBD

We will use DRBD 8.4.

Partition Storage

How you do this will depend a lot on your storage (local disks, md software RAID, hardware RAID, 1 or multiple arrays, etc). It will also depend on how you plan to divy up your servers; you need two partitions; One for servers that will run on node 1 and another for node 2. It also depends on how much space you want for the /shared partition.

In our case, we're using a single hardware RAID array, we'll set aside 40 GB of space for /shared and we're going to divide the remaining free space evenly.

an-a04n01

parted -a opt /dev/sda "print free"

Model: LSI RAID 5/6 SAS 6G (scsi)
Disk /dev/sda: 898GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system     Flags
        32.3kB  1049kB  1016kB           Free Space
 1      1049kB  538MB   537MB   primary  ext4            boot
 2      538MB   4833MB  4295MB  primary  linux-swap(v1)
 3      4833MB  26.3GB  21.5GB  primary  ext4
        26.3GB  898GB   872GB            Free Space

an-a04n01

# same as an-a04n01

So 872 GB of free space, less 40 for /shared leaves 832 GB for servers. Divided evenly in 2 gives us 416 GB per server pool. Our first partition will then be 446 GB (40 for /shared) and the second will be 416 GB.

The free space starts at 26.3 GB, so our first partition will start at 26.3 GB and end at 492 GB (rounding off the .3). The second partition will then start at 492 GB and end at 898 GB, the end of the disk. Both of these new partitions will be contained in an extended partition.

Note: After each change, we will get an error saying "Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy). As a result, it may not reflect all of your changes until after reboot.". Will reboot once done to address this.

an-a04n01

parted -a opt /dev/sda "mkpart extended 26.3GB 898GB"
parted -a opt /dev/sda "mkpart logical 26.3GB 492GB"
parted -a opt /dev/sda "mkpart logical 492GB 898GB"
parted -a opt /dev/sda "print free"

Model: LSI RAID 5/6 SAS 6G (scsi)
Disk /dev/sda: 898GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type      File system     Flags
        32.3kB  1049kB  1016kB            Free Space
 1      1049kB  538MB   537MB   primary   ext4            boot
 2      538MB   4833MB  4295MB  primary   linux-swap(v1)
 3      4833MB  26.3GB  21.5GB  primary   ext4
 4      26.3GB  898GB   872GB   extended                  lba
 5      26.3GB  492GB   466GB   logical
 6      492GB   898GB   406GB   logical

an-a04n01

# same as an-a04n01

Reboot

an-a04n01	reboot
an-a04n01	reboot

Configure DRBD

Configure global-common.conf;

an-a04n01

vim /etc/drbd.d/global_common.conf

# These are options to set for the DRBD daemon sets the default values for
# resources.
global {
	# This tells DRBD that you allow it to report this installation to 
	# LINBIT for statistical purposes. If you have privacy concerns, set
	# this to 'no'. The default is 'ask' which will prompt you each time
	# DRBD is updated. Set to 'yes' to allow it without being prompted.
	usage-count yes;
 
	# minor-count dialog-refresh disable-ip-verification
}
 
common {
	handlers {
		# pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
		# pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
		# local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
		# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
		# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
		# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
		# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
 
		# Hook into Pacemaker's fencing.
		fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
		before-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
	}
 
	startup {
		# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
	}
 
	options {
		# cpu-mask on-no-data-accessible
	}
 
	disk {
		# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
		# disk-drain md-flushes resync-rate resync-after al-extents
                # c-plan-ahead c-delay-target c-fill-target c-max-rate
                # c-min-rate disk-timeout
                fencing resource-and-stonith;
	}
 
	net {
		# protocol timeout max-epoch-size max-buffers unplug-watermark
		# connect-int ping-int sndbuf-size rcvbuf-size ko-count
		# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
		# after-sb-1pri after-sb-2pri always-asbp rr-conflict
		# ping-timeout data-integrity-alg tcp-cork on-congestion
		# congestion-fill congestion-extents csums-alg verify-alg
		# use-rle
 
		# Protocol "C" tells DRBD not to tell the operating system that
		# the write is complete until the data has reach persistent
		# storage on both nodes. This is the slowest option, but it is
		# also the only one that guarantees consistency between the
		# nodes. It is also required for dual-primary, which we will 
		# be using.
		protocol C;
 
		# Tell DRBD to allow dual-primary. This is needed to enable 
		# live-migration of our servers.
		allow-two-primaries yes;
 
		# This tells DRBD what to do in the case of a split-brain when
		# neither node was primary, when one node was primary and when
		# both nodes are primary. In our case, we'll be running
		# dual-primary, so we can not safely recover automatically. The
		# only safe option is for the nodes to disconnect from one
		# another and let a human decide which node to invalidate. Of 
		after-sb-0pri discard-zero-changes;
		after-sb-1pri discard-secondary;
		after-sb-2pri disconnect;
	}
}

And now configure the first resource;

an-a04n01

vim /etc/drbd.d/r0.res

# This is the first DRBD resource. It will store the shared file systems and
# the servers designed to run on node 01.
resource r0 {
	# These options here are common to both nodes. If for some reason you
	# need to set unique values per node, you can move these to the
	# 'on <name> { ... }' section.
 
	# This sets the device name of this DRBD resouce.
	device /dev/drbd0;
 
	# This tells DRBD what the backing device is for this resource.
	disk /dev/sda5;
 
	# This controls the location of the metadata. When "internal" is used,
	# as we use here, a little space at the end of the backing devices is
	# set aside (roughly 32 MB per 1 TB of raw storage). External metadata
	# can be used to put the metadata on another partition when converting
	# existing file systems to be DRBD backed, when there is no extra space
	# available for the metadata.
	meta-disk internal;
 
	# NOTE: this is not required or even recommended with pacemaker. remove
	# 	this options as soon as pacemaker is setup.
	startup {
		# This tells DRBD to promote both nodes to 'primary' when this
		# resource starts. However, we will let pacemaker control this
		# so we comment it out, which tells DRBD to leave both nodes
		# as secondary when drbd starts.
		#become-primary-on both;
	}
 
	# NOTE: Later, make it an option in the dashboard to trigger a manual
	# 	verify and/or schedule periodic automatic runs
	net {
		# TODO: Test performance differences between sha1 and md5
		# This tells DRBD how to do a block-by-block verification of
		# the data stored on the backing devices. Any verification
		# failures will result in the effected block being marked
		# out-of-sync.
		verify-alg md5;
 
		# TODO: Test the performance hit of this being enabled.
		# This tells DRBD to generate a checksum for each transmitted
		# packet. If the data received data doesn't generate the same
		# sum, a retransmit request is generated. This protects against
		# otherwise-undetected errors in transmission, like 
		# bit-flipping. See:
		# http://www.drbd.org/users-guide/s-integrity-check.html
		data-integrity-alg md5;
	}
 
	# WARNING: Confirm that these are safe when the controller's BBU is
	#          depleted/failed and the controller enters write-through 
	#          mode.
	disk {
		# TODO: Test the real-world performance differences gained with
		#       these options.
		# This tells DRBD not to bypass the write-back caching on the
		# RAID controller. Normally, DRBD forces the data to be flushed
		# to disk, rather than allowing the write-back cachine to 
		# handle it. Normally this is dangerous, but with BBU-backed
		# caching, it is safe. The first option disables disk flushing
		# and the second disabled metadata flushes.
		disk-flushes no;
		md-flushes no;
	}
 
	# This sets up the resource on node 01. The name used below must be the
	# named returned by "uname -n".
	on an-a04n01.alteeve.ca {
		# This is the address and port to use for DRBD traffic on this
		# node. Multiple resources can use the same IP but the ports
		# must differ. By convention, the first resource uses 7788, the
		# second uses 7789 and so on, incrementing by one for each
		# additional resource. 
		address 10.10.40.1:7788;
	}
	on an-a04n02.alteeve.ca {
		address 10.10.40.2:7788;
	}
}

And the second.

an-a04n01

vim /etc/drbd.d/r1.res

# This is the first DRBD resource. It will store the servers designed 
# to run on node 02.
resource r1 {
	device /dev/drbd1;
	disk /dev/sda6;
	meta-disk internal;
 
	net {
		verify-alg md5;
		data-integrity-alg md5;
	}
 
	disk {
		disk-flushes no;
		md-flushes no;
	}
 
	on an-a04n01.alteeve.ca {
		address 10.10.40.1:7789;
	}
	on an-a04n02.alteeve.ca {
		address 10.10.40.2:7789;
	}
}

Test the config;

an-a04n01

drbdadm dump

# /etc/drbd.conf
common {
}

# resource r0 on an-a04n01.alteeve.ca: not ignored, not stacked
# defined at /etc/drbd.d/r0.res:3
resource r0 {
    on an-a04n01.alteeve.ca {
        volume 0 {
            device       /dev/drbd0 minor 0;
            disk         /dev/sda5;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.1:7788;
    }
    on an-a04n02.alteeve.ca {
        volume 0 {
            device       /dev/drbd0 minor 0;
            disk         /dev/sda5;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.2:7788;
    }
    net {
        verify-alg       md5;
        data-integrity-alg md5;
    }
    disk {
        disk-flushes      no;
        md-flushes        no;
    }
}

# resource r1 on an-a04n01.alteeve.ca: not ignored, not stacked
# defined at /etc/drbd.d/r1.res:3
resource r1 {
    on an-a04n01.alteeve.ca {
        volume 0 {
            device       /dev/drbd1 minor 1;
            disk         /dev/sda6;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.1:7789;
    }
    on an-a04n02.alteeve.ca {
        volume 0 {
            device       /dev/drbd1 minor 1;
            disk         /dev/sda6;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.2:7789;
    }
    net {
        verify-alg       md5;
        data-integrity-alg md5;
    }
    disk {
        disk-flushes      no;
        md-flushes        no;
    }
}

Good, copy it to the other node and test it there.

an-a04n01

rsync -av /etc/drbd.* root@an-a04n02:/etc/

sending incremental file list
drbd.d/
drbd.d/global_common.conf
drbd.d/r0.res
drbd.d/r1.res

sent 5738 bytes  received 73 bytes  11622.00 bytes/sec
total size is 5618  speedup is 0.97

an-a04n01

drbdadm dump

# /etc/drbd.conf
common {
}

# resource r0 on an-a04n02.alteeve.ca: not ignored, not stacked
# defined at /etc/drbd.d/r0.res:3
resource r0 {
    on an-a04n01.alteeve.ca {
        volume 0 {
            device       /dev/drbd0 minor 0;
            disk         /dev/sda5;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.1:7788;
    }
    on an-a04n02.alteeve.ca {
        volume 0 {
            device       /dev/drbd0 minor 0;
            disk         /dev/sda5;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.2:7788;
    }
    net {
        verify-alg       md5;
        data-integrity-alg md5;
    }
    disk {
        disk-flushes      no;
        md-flushes        no;
    }
}

# resource r1 on an-a04n02.alteeve.ca: not ignored, not stacked
# defined at /etc/drbd.d/r1.res:3
resource r1 {
    on an-a04n01.alteeve.ca {
        volume 0 {
            device       /dev/drbd1 minor 1;
            disk         /dev/sda6;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.1:7789;
    }
    on an-a04n02.alteeve.ca {
        volume 0 {
            device       /dev/drbd1 minor 1;
            disk         /dev/sda6;
            meta-disk    internal;
        }
        address          ipv4 10.10.40.2:7789;
    }
    net {
        verify-alg       md5;
        data-integrity-alg md5;
    }
    disk {
        disk-flushes      no;
        md-flushes        no;
    }
}

This isn't a plain dump of your configs, you will notice things have been shifted around. The point is that it dumped the configuration without errors, so we're good to go.

Start DRBD for the first time

Load the config;

an-a04n01	modprobe drbd lsmod \| grep drbd drbd 333723 0 libcrc32c 1246 1 drbd
an-a04n01	modprobe drbd lsmod \| grep drbd drbd 333723 0 libcrc32c 1246 1 drbd

Note: If you have used these partitions before, drbd may see an FS and refuse to create the MD. If that happens, use 'dd' to zero out the partition.

Create the metadisk;

an-a04n01

drbdadm create-md r{0,1}

Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success

an-a04n01

drbdadm create-md r{0,1}

Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success

Bring up the new resources.

an-a04n01

drbdadm up r{0,1}
cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
 0: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/Outdated C r----s
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:454762916
 1: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/Outdated C r----s
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396782732

an-a04n01

drbdadm up r{0,1}
cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:454762916
 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396782732

Neither node has data, so we'll arbitrarily force node 01 to become primary, then normally promote node 02 to primary.

an-a04n01

drbdadm primary --force r{0,1}
cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:2136 nr:0 dw:0 dr:2800 al:0 bm:0 lo:0 pe:3 ua:0 ap:0 ep:1 wo:d oos:454760880
        [>....................] sync'ed:  0.1% (444100/444104)M
        finish: 421:04:29 speed: 252 (252) K/sec
 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:24696 nr:0 dw:0 dr:25360 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396758036
        [>....................] sync'ed:  0.1% (387456/387480)M
        finish: 35:33:06 speed: 3,084 (3,084) K/sec

an-a04n01

drbdadm primary r{0,1}
cat /proc/drbd

version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
 0: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:859488 dw:859432 dr:608 al:0 bm:52 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:453903484
        [>....................] sync'ed:  0.2% (443264/444104)M
        finish: 71:24:53 speed: 1,752 (4,428) want: 440 K/sec
 1: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:1140588 dw:1140532 dr:608 al:0 bm:69 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:395642200
        [>....................] sync'ed:  0.3% (386368/387480)M
        finish: 70:30:41 speed: 1,548 (5,876) want: 4,400 K/sec

The sync rate starts low, but it will continue to climb, you can keep an eye on it if you wish. DRBD 8.4 is smarter than 8.3 in that it will adjust the sync rate automatically based on load.

We can proceed now, we do not have to wait for the sync to complete.

Clustered LVM and GFS2

Clustered LVM provides the logical volumes that will back our /shared GFS2 partition and the storage for the HA servers.

Configure lvm.conf

Configure clustered LVM.

an-a04n01

sed -i.anvil 's^filter = \[ "a/\.\*/" \]^filter = \[ "a|/dev/drbd*|", "r/.*/" \]^' /etc/lvm/lvm.conf
sed -i 's/locking_type = 1$/locking_type = 3/' /etc/lvm/lvm.conf
sed -i 's/fallback_to_local_locking = 1$/fallback_to_local_locking = 0/' /etc/lvm/lvm.conf 
diff -u /etc/lvm/lvm.conf.anvil /etc/lvm/lvm.conf

--- /etc/lvm/lvm.conf.anvil	2013-10-30 04:10:42.000000000 -0400
+++ /etc/lvm/lvm.conf	2014-06-04 18:38:15.545166869 -0400
@@ -82,7 +82,7 @@
 
 
     # By default we accept every block device:
-    filter = [ "a/.*/" ]
+    filter = [ "a|/dev/drbd*|", "r/.*/" ]
 
     # Exclude the cdrom drive
     # filter = [ "r|/dev/cdrom|" ]
@@ -459,7 +459,7 @@
     # Type 3 uses built-in clustered locking.
     # Type 4 uses read-only locking which forbids any operations that might 
     # change metadata.
-    locking_type = 1
+    locking_type = 3
 
     # Set to 0 to fail when a lock request cannot be satisfied immediately.
     wait_for_locks = 1
@@ -475,7 +475,7 @@
     # to 1 an attempt will be made to use local file-based locking (type 1).
     # If this succeeds, only commands against local volume groups will proceed.
     # Volume Groups marked as clustered will be ignored.
-    fallback_to_local_locking = 1
+    fallback_to_local_locking = 0
 
     # Local non-LV directory that holds file-based locks while commands are
     # in progress.  A directory like /tmp that may get wiped on reboot is OK.

rsync -av /etc/lvm/lvm.conf* root@an-a04n02:/etc/lvm/

sending incremental file list
lvm.conf
lvm.conf.anvil

sent 47499 bytes  received 440 bytes  95878.00 bytes/sec
total size is 89999  speedup is 1.88

an-a04n02

diff -u /etc/lvm/lvm.conf.anvil /etc/lvm/lvm.conf

--- /etc/lvm/lvm.conf.anvil	2013-10-30 04:10:42.000000000 -0400
+++ /etc/lvm/lvm.conf	2014-06-04 18:38:15.000000000 -0400
@@ -82,7 +82,7 @@
 
 
     # By default we accept every block device:
-    filter = [ "a/.*/" ]
+    filter = [ "a|/dev/drbd*|", "r/.*/" ]
 
     # Exclude the cdrom drive
     # filter = [ "r|/dev/cdrom|" ]
@@ -459,7 +459,7 @@
     # Type 3 uses built-in clustered locking.
     # Type 4 uses read-only locking which forbids any operations that might 
     # change metadata.
-    locking_type = 1
+    locking_type = 3
 
     # Set to 0 to fail when a lock request cannot be satisfied immediately.
     wait_for_locks = 1
@@ -475,7 +475,7 @@
     # to 1 an attempt will be made to use local file-based locking (type 1).
     # If this succeeds, only commands against local volume groups will proceed.
     # Volume Groups marked as clustered will be ignored.
-    fallback_to_local_locking = 1
+    fallback_to_local_locking = 0
 
     # Local non-LV directory that holds file-based locks while commands are
     # in progress.  A directory like /tmp that may get wiped on reboot is OK.

Start clvmd

Note: This will be moved to pacemaker shortly. We're enabling it here just long enough to configure pacemaker.

Make sure the cluster is up (you could use 'pcs status', 'cman_tool status', etc):

an-a04n01

dlm_tool dump | grep node

1401921044 cluster node 1 added seq 68
1401921044 set_configfs_node 1 10.20.40.1 local 1
1401921044 cluster node 2 added seq 68
1401921044 set_configfs_node 2 10.20.40.2 local 0
1401921044 run protocol from nodeid 1

Make sure DRBD is up as primary on both nodes:

an-a04n01

cat /proc/drbd

version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
 0: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r-----
    ns:1519672 nr:0 dw:0 dr:1520336 al:0 bm:93 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:448214308
	[>....................] sync'ed:  0.4% (437708/439192)M
	finish: 6:20:02 speed: 19,652 (15,992) K/sec
 1: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r-----
    ns:1896504 nr:0 dw:0 dr:1897168 al:0 bm:115 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:390577164
	[>....................] sync'ed:  0.5% (381420/383272)M
	finish: 2:33:17 speed: 42,440 (19,960) K/sec

Note that we don't have to wait for the sync to finish.

Start clvmd;

an-a04n01	/etc/init.d/clvmd start Starting clvmd: Activating VG(s): No volume groups found [ OK ]
an-a04n02	/etc/init.d/clvmd start Starting clvmd: Activating VG(s): No volume groups found [ OK ]

Note: If this fails, showing a timeout or simply never returning, make sure that TCP port 21064 is opened in your firewall on both nodes.

From here on, pacemaker will start clvmd when pacemaker itself start, *if* clvmd is set to start on boot. So lets set that.

an-a04n01	chkconfig clvmd on chkconfig --list clvmd clvmd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
an-a04n01	chkconfig clvmd on chkconfig --list clvmd clvmd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Create Initial PVs, VGs and the /shared LV

Create the PV, VG and the /shared LV;

an-a04n01

pvcreate /dev/drbd{0,1}

  Physical volume "/dev/drbd0" successfully created
  Physical volume "/dev/drbd1" successfully created

vgcreate an-a04n01_vg0 /dev/drbd0

  Clustered volume group "an-a04n01_vg0" successfully created

vgcreate an-a04n02_vg0 /dev/drbd1

  Clustered volume group "an-a04n02_vg0" successfully created

lvcreate -L 40GiB -n shared an-a04n01_vg0

  Logical volume "shared" created

an-a04n02

pvdisplay

  --- Physical volume ---
  PV Name               /dev/drbd1
  VG Name               an-a04n02_vg0
  PV Size               378.40 GiB / not usable 3.14 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              96870
  Free PE               96870
  Allocated PE          0
  PV UUID               TpEXBC-7822-UGz0-ICz1-AJdg-v5eS-lyB7C5
   
  --- Physical volume ---
  PV Name               /dev/drbd0
  VG Name               an-a04n01_vg0
  PV Size               433.70 GiB / not usable 4.41 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              111025
  Free PE               100785
  Allocated PE          10240
  PV UUID               RoHAJQ-qrsO-Ofwz-f8W7-jIXd-2cvG-oPgfFR

vgdisplay

  --- Volume group ---
  VG Name               an-a04n02_vg0
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  Clustered             yes
  Shared                no
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               378.40 GiB
  PE Size               4.00 MiB
  Total PE              96870
  Alloc PE / Size       0 / 0   
  Free  PE / Size       96870 / 378.40 GiB
  VG UUID               9bTBDu-JSma-kwKR-4oBI-sxi1-YT6i-1uIM4C
   
  --- Volume group ---
  VG Name               an-a04n01_vg0
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  Clustered             yes
  Shared                no
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               433.69 GiB
  PE Size               4.00 MiB
  Total PE              111025
  Alloc PE / Size       10240 / 40.00 GiB
  Free  PE / Size       100785 / 393.69 GiB
  VG UUID               hLnvle-EScm-cP1t-xodO-cKyv-5EyC-TyIpj5

lvdisplay

  --- Logical volume ---
  LV Path                /dev/an-a04n01_vg0/shared
  LV Name                shared
  VG Name                an-a04n01_vg0
  LV UUID                tvolRF-cb3L-29Dn-Vgqd-e4rf-Qq2e-JFIcbA
  LV Write Access        read/write
  LV Creation host, time an-a04n01.alteeve.ca, 2014-06-07 18:54:41 -0400
  LV Status              available
  # open                 0
  LV Size                40.00 GiB
  Current LE             10240
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

Create the /shared GFS2 filesystem

Format the /dev/an-a04n01_vg0/shared logical volume as a GFS2 filesystem;

an-a04n01

mkfs.gfs2 -j 2 -p lock_dlm -t an-anvil-04:shared /dev/an-a04n01_vg0/shared

This will destroy any data on /dev/an-a04n01_vg0/shared.
It appears to contain: symbolic link to `../dm-0'

Are you sure you want to proceed? [y/n] y

Device:                    /dev/an-a04n01_vg0/shared
Blocksize:                 4096
Device Size                40.00 GB (10485760 blocks)
Filesystem Size:           40.00 GB (10485758 blocks)
Journals:                  2
Resource Groups:           160
Locking Protocol:          "lock_dlm"
Lock Table:                "an-anvil-04:shared"
UUID:                      e07d35fe-6860-f790-38cd-af075366c27b

mkdir /shared
mount /dev/an-a04n01_vg0/shared /shared
df -hP

Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           20G  1.5G   18G   8% /
tmpfs                               12G   67M   12G   1% /dev/shm
/dev/sda1                          504M   72M  407M  16% /boot
/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared

an-a04n02

mkdir /shared
mount /dev/an-a04n01_vg0/shared /shared
df -hP

Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           20G  1.5G   18G   8% /
tmpfs                               12G   52M   12G   1% /dev/shm
/dev/sda1                          504M   72M  407M  16% /boot
/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared

Add Storage to Pacemaker

Configure Dual-Primary DRBD

Setup DRBD as a dual-primary resource.

Notes:

Clones allow for a given service to run on multiple nodes.
- master-max is how many copies of the resource can be promoted to master at the same time across the cluster.
- master-node-max is how many copies of the resource can be promoted to master on a given node.
- clone-max is how many copies can run in the cluster, default is to the number of nodes in the cluster.
- clone-node-max is the number of instances of the resource that can run on each node.
- notify controls whether other nodes are notified before and after a resource is started or stopped on a given node.

an-a04n01

pcs cluster cib drbd_cfg
pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 op monitor interval=10s
pcs -f drbd_cfg resource create drbd_r1 ocf:linbit:drbd drbd_resource=r1 op monitor interval=10s
### Ignore this for now.
#pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 \
#                op monitor interval=29s role=Master \
#                op monitor interval=31s role=Slave \
#                op promote interval=0 timeout=90s start-delay=2s \
#                op start interval=0 timeout=240s \
#                op stop interval=0 timeout=120s
pcs -f drbd_cfg resource master drbd_r0_Clone drbd_r0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs -f drbd_cfg resource master drbd_r1_Clone drbd_r1 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs cluster cib-push drbd_cfg

CIB updated

Give it a couple minutes to promote both nodes to Master on both nodes. Initially, it will appear as Master on one node only.

Once updated, you should see this:

an-a04n01

pcs status

Cluster name: an-anvil-04
Last updated: Sat Jun  7 20:29:09 2014
Last change: Sat Jun  7 20:28:36 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
6 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca 
 fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca 
 Master/Slave Set: drbd_r0_Clone [drbd_r0]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Master/Slave Set: drbd_r1_Clone [drbd_r1]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

an-a04n02

pcs status

Cluster name: an-anvil-04
Last updated: Sat Jun  7 20:29:36 2014
Last change: Sat Jun  7 20:28:36 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
6 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca 
 fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca 
 Master/Slave Set: drbd_r0_Clone [drbd_r0]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Master/Slave Set: drbd_r1_Clone [drbd_r1]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Configure LVM

We need to have pacemaker activate our clustered LVM LVs on start, and deactivate them when stopping. We don't start/stop clvmd directly because of stop timing issues that can lead to stray fencing.

Note: This will throw errors if there are no LVs on a given VG... Do not add a volume group until at least one logical volume has been created.

an-a04n01

pcs cluster cib lvm_cfg
pcs -f lvm_cfg resource create lvm_n01_vg0 ocf:heartbeat:lvm volgrpname=an-a04n01_vg0 op monitor interval=10s
pcs -f lvm_cfg resource master lvm_n01_vg0_Clone lvm_n01_vg0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs cluster cib-push lvm_cfg

CIB updated

Configure LVM to start after the DRBD PV is Primary

It we stopped here, there is a good chance that on future starts of pacemaker, LVM and DRBD would start in parallel, DRBD would take too long, LVM would error out and stonith's would start to fly. To prevent this, we will tell Pacemaker not to start the LVM resource until after the DRBD resource that is behind the volume group has been promoted to primary.

an-a04n01

pcs cluster cib cst_cfg 
pcs -f cst_cfg constraint order promote drbd_r0_Clone then start lvm_n01_vg0_Clone

Adding drbd_r0_Clone lvm_n01_vg0_Clone (kind: Mandatory) (Options: first-action=promote then-action=start)

pcs cluster cib-push cst_cfg

CIB updated

pcs constraint show

Location Constraints:
Ordering Constraints:
  promote drbd_r0_Clone then start lvm_n01_vg0_Clone
Colocation Constraints:

Configure the /shared GFS2 Partition

an-a04n01

pcs cluster cib fs_cfg
pcs -f fs_cfg resource create sharedFS Filesystem device="/dev/an-a04n01_vg0/shared" directory="/shared" fstype="gfs2"
pcs -f fs_cfg resource clone sharedFS master-max=2 master-node-max=1 clone-max=2 clone-node-max=1
pcs cluster cib-push fs_cfg

CIB updated

pcs status

Cluster name: an-anvil-04
Last updated: Sat Jun  7 21:09:28 2014
Last change: Sat Jun  7 21:08:47 2014 via cibadmin on an-a04n01.alteeve.ca
Stack: cman
Current DC: an-a04n01.alteeve.ca - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
8 Resources configured


Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

Full list of resources:

 fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca 
 fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca 
 Master/Slave Set: drbd_r0_Clone [drbd_r0]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Master/Slave Set: drbd_r1_Clone [drbd_r1]
     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 Clone Set: sharedFS-clone [sharedFS]
     Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]

df -hP

Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           20G  1.5G   18G   8% /
tmpfs                               12G   67M   12G   1% /dev/shm
/dev/sda1                          504M   72M  407M  16% /boot
/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared

an-a04n02

df -h

Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           20G  1.5G   18G   8% /
tmpfs                               12G   52M   12G   1% /dev/shm
/dev/sda1                          504M   72M  407M  16% /boot
/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared

Configur /shared to start after LVM

As we did before in making sure LVM started after DRBD, this time we will make sure LVM starts before /shared is mounted.

an-a04n01

pcs cluster cib cst_cfg
pcs -f cst_cfg constraint order start lvm_n01_vg0_Clone then start sharedFS

Adding lvm_n01_vg0_Clone sharedFS (kind: Mandatory) (Options: first-action=start then-action=start)

pcs cluster cib-push cst_cfg

CIB updated

pcs constraint show --full

Location Constraints:
Ordering Constraints:
  promote drbd_r0_Clone then start lvm_n01_vg0_Clone (Mandatory) (id:order-drbd_r0_Clone-lvm_n01_vg0_Clone-mandatory)
  start lvm_n01_vg0_Clone then start sharedFS-clone (Mandatory) (id:order-lvm_n01_vg0_Clone-sharedFS-clone-mandatory)
Colocation Constraints:

Note that this time we added '--full'. If you ever need to delete a constraint, you would use 'pcs constraint delete <id>'.

Notes

Pacemaker Logging

Thanks

This list will certainly grow as this tutorial progresses;

Olivier Allart, RCHE for doing a lot of the heavy lifting on the fencing_topology configuration.

Any questions, feedback, advice, complaints or meanderings are welcome.
`Alteeve's Niche!`	`Enterprise Support: Alteeve Support`	`Community Support`
© Alteeve's Niche! Inc. 1997-2024		Anvil! "Intelligent Availability®" Platform
`legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.`

@@ Line 48: / Line 48: @@
 |-
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-vi /etc/yum.repos.d/an.repo
+cat <<-END>/etc/yum.repos.d/an.repo
-</syntaxhighlight>
-<syntaxhighlight lang="bash">
 [an-repo]
 name=AN! Repo for Anvil! stuff
@@ Line 57: / Line 55: @@
 gpgcheck=0
 protect=1
+END
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
@@ Line 62: / Line 61: @@
 </syntaxhighlight>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-vi /etc/yum.repos.d/an.repo
+cat <<-END>/etc/yum.repos.d/an.repo
-</syntaxhighlight>
-<syntaxhighlight lang="bash">
 [an-repo]
 name=AN! Repo for Anvil! stuff
@@ Line 71: / Line 68: @@
 gpgcheck=0
 protect=1
+END
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
@@ Line 92: / Line 90: @@
 yum -y update
 yum -y install bridge-utils vim pacemaker corosync cman gfs2-utils \
-                ccs pcs ipmitool OpenIPMI drbd83-utils kmod-drbd83
+                ccs pcs ipmitool OpenIPMI lvm2-cluster drbd84-utils \
+               drbd84-kmod
 chkconfig ipmi on
 chkconfig acpid off
@@ Line 783: / Line 782: @@
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+# cman (corosync's totem)
 iptables -I INPUT -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 -d 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
 iptables -I INPUT -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
+# dlm
+iptables -I INPUT -m state --state NEW -p tcp -s 10.20.0.0/16 -d 10.20.0.0/16 --dport 21064 -j ACCEPT
+# DRBD resource 0 and 1 - on the SN
 iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7788 -j ACCEPT
 iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7789 -j ACCEPT
+# Make the new rules persistent.
 /etc/init.d/iptables save
 </syntaxhighlight>
@@ Line 795: / Line 802: @@
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+# cman (corosync's totem)
 iptables -I INPUT -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 -d 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
 iptables -I INPUT -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport -p udp -s 10.20.0.0/16 --dports 5404,5405 -j ACCEPT
+# dlm
+iptables -I INPUT -m state --state NEW -p tcp -s 10.20.0.0/16 -d 10.20.0.0/16 --dport 21064 -j ACCEPT
+# DRBD resource 0 and 1 - on the SN
 iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7788 -j ACCEPT
 iptables -I INPUT -m state --state NEW -p tcp -s 10.10.0.0/16 -d 10.10.0.0/16 --dport 7789 -j ACCEPT
+# Make the new rules persistent.
+/etc/init.d/iptables save
 </syntaxhighlight>
 <syntaxhighlight lang="text">
@@ Line 855: / Line 871: @@
 ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk an-a04n01.alteeve.ca pcmk-redirect port=an-a04n01.alteeve.ca
 ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk an-a04n02.alteeve.ca pcmk-redirect port=an-a04n02.alteeve.ca
+ccs -f /etc/cluster/cluster.conf --setfencedaemon post_join_delay="30"
 cat /etc/cluster/cluster.conf
 </syntaxhighlight>
 <syntaxhighlight lang="xml">
-<cluster config_version="9" name="an-anvil-04">
+<cluster config_version="10" name="an-anvil-04">
-   <fence_daemon/>
+   <fence_daemon post_join_delay="30"/>
    <clusternodes>
      <clusternode name="an-a04n01.alteeve.ca" nodeid="1">
        <fence>
-	<method name="pcmk-redirect">
+        <method name="pcmk-redirect">
            <device name="pcmk" port="an-a04n01.alteeve.ca"/>
          </method>
@@ Line 870: / Line 887: @@
      <clusternode name="an-a04n02.alteeve.ca" nodeid="2">
        <fence>
-	<method name="pcmk-redirect">
+        <method name="pcmk-redirect">
            <device name="pcmk" port="an-a04n02.alteeve.ca"/>
          </method>
@@ Line 899: / Line 916: @@
 cluster.conf
-sent 816 bytes  received 43 bytes  1718.00 bytes/sec
+sent 838 bytes  received 31 bytes  579.33 bytes/sec
-total size is 736  speedup is 0.86
+total size is 758  speedup is 0.87
 </syntaxhighlight>
 |-
@@ Line 908: / Line 925: @@
 </syntaxhighlight>
 <syntaxhighlight lang="xml">
-<cluster config_version="9" name="an-anvil-04">
+<cluster config_version="10" name="an-anvil-04">
-   <fence_daemon/>
+   <fence_daemon post_join_delay="30"/>
    <clusternodes>
      <clusternode name="an-a04n01.alteeve.ca" nodeid="1">
        <fence>
-	<method name="pcmk-redirect">
+        <method name="pcmk-redirect">
            <device name="pcmk" port="an-a04n01.alteeve.ca"/>
          </method>
@@ Line 920: / Line 937: @@
      <clusternode name="an-a04n02.alteeve.ca" nodeid="2">
        <fence>
-	<method name="pcmk-redirect">
+        <method name="pcmk-redirect">
            <device name="pcmk" port="an-a04n02.alteeve.ca"/>
          </method>
@@ Line 1,175: / Line 1,192: @@
 One of the nice things about pcs is that it allows us to create a test file to prepare all our changes in. Then, when we're happy with the changes, merge them into the running cluster. So let's make a copy called <span class="code">stonith_cfg</span>
-{{note|1=Use this properly later...}}
+Now add [[IPMI]] fencing.
 {|class="wikitable"
@@ Line 1,181: / Line 1,198: @@
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs cluster cib stonith_cfg
-</syntaxhighlight>
+#   work in our temp file         unique name    fence agent   target node                           device addr             options
-|}
+pcs -f stonith_cfg stonith create fence_n01_ipmi fence_ipmilan pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-a04n01.ipmi" action="reboot" login="admin" passwd="Initial1" delay=15 op monitor interval=10s
+pcs -f stonith_cfg stonith create fence_n02_ipmi fence_ipmilan pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-a04n02.ipmi" action="reboot" login="admin" passwd="Initial1" op monitor interval=10s
-Now add [[IPMI]] fencing.
+pcs cluster cib-push stonith_cfg
-{|class="wikitable"
-!<span class="code">an-a04n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-#                  unique name    fence agent   target node                           device addr             options
-pcs stonith create fence_n01_ipmi fence_ipmilan pcmk_host_list="an-a04n01.alteeve.ca" ipaddr="an-a04n01.ipmi" action="reboot" login="admin" passwd="Initial1" delay=15 op monitor interval=60s
-pcs stonith create fence_n02_ipmi fence_ipmilan pcmk_host_list="an-a04n02.alteeve.ca" ipaddr="an-a04n02.ipmi" action="reboot" login="admin" passwd="Initial1" op monitor interval=60s
 </syntaxhighlight>
 |}
@@ Line 1,232: / Line 1,242: @@
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 pcs property set stonith-enabled=true
+pcs property set no-quorum-policy=ignore
 pcs property
 </syntaxhighlight>
@@ Line 1,238: / Line 1,249: @@
   cluster-infrastructure: cman
   dc-version: 1.1.10-14.el6_5.3-368c726
-  expected-quorum-votes: 2
+  no-quorum-policy: ignore
   stonith-enabled: true
 </syntaxhighlight>
@@ Line 1,260: / Line 1,271: @@
 == DRBD ==
-We will use DRBD 8.3.
+We will use DRBD 8.4.
 === Partition Storage ===
@@ Line 1,354: / Line 1,365: @@
 </syntaxhighlight>
 <syntaxhighlight lang="text">
+# These are options to set for the DRBD daemon sets the default values for
+# resources.
 global {
-        usage-count yes;
+	# This tells DRBD that you allow it to report this installation to
+	# LINBIT for statistical purposes. If you have privacy concerns, set
+	# this to 'no'. The default is 'ask' which will prompt you each time
+	# DRBD is updated. Set to 'yes' to allow it without being prompted.
+	usage-count yes;
+	# minor-count dialog-refresh disable-ip-verification
 }
 common {
-        protocol C;
+	handlers {
+		# pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
-        handlers {
+		# pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
-                fence-peer "/usr/lib/drbd/stonith_admin-fence-peer.sh";
+		# local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
-        }
+		# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
+		# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
-        startup {
+		# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
-                # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
+		# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
-        }
+		# Hook into Pacemaker's fencing.
-        disk {
+		fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
+		before-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
+	}
+	startup {
+		# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
+	}
+	options {
+		# cpu-mask on-no-data-accessible
+	}
+	disk {
+		# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
+		# disk-drain md-flushes resync-rate resync-after al-extents
+                # c-plan-ahead c-delay-target c-fill-target c-max-rate
+                # c-min-rate disk-timeout
                  fencing resource-and-stonith;
-        }
+	}
-        net {
+	net {
-                allow-two-primaries;
+		# protocol timeout max-epoch-size max-buffers unplug-watermark
-                after-sb-0pri discard-zero-changes;
+		# connect-int ping-int sndbuf-size rcvbuf-size ko-count
-                after-sb-1pri discard-secondary;
+		# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
-                after-sb-2pri disconnect;
+		# after-sb-1pri after-sb-2pri always-asbp rr-conflict
-        }
+		# ping-timeout data-integrity-alg tcp-cork on-congestion
+		# congestion-fill congestion-extents csums-alg verify-alg
-        syncer {
+		# use-rle
-                rate 40M;
-        }
+		# Protocol "C" tells DRBD not to tell the operating system that
+		# the write is complete until the data has reach persistent
+		# storage on both nodes. This is the slowest option, but it is
+		# also the only one that guarantees consistency between the
+		# nodes. It is also required for dual-primary, which we will
+		# be using.
+		protocol C;
+		# Tell DRBD to allow dual-primary. This is needed to enable
+		# live-migration of our servers.
+		allow-two-primaries yes;
+		# This tells DRBD what to do in the case of a split-brain when
+		# neither node was primary, when one node was primary and when
+		# both nodes are primary. In our case, we'll be running
+		# dual-primary, so we can not safely recover automatically. The
+		# only safe option is for the nodes to disconnect from one
+		# another and let a human decide which node to invalidate. Of
+		after-sb-0pri discard-zero-changes;
+		after-sb-1pri discard-secondary;
+		after-sb-2pri disconnect;
+	}
 }
 </syntaxhighlight>
@@ Line 1,395: / Line 1,451: @@
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
+# This is the first DRBD resource. It will store the shared file systems and
+# the servers designed to run on node 01.
 resource r0 {
+	# These options here are common to both nodes. If for some reason you
+	# need to set unique values per node, you can move these to the
+	# 'on <name> { ... }' section.
+	# This sets the device name of this DRBD resouce.
 	device /dev/drbd0;
+	# This tells DRBD what the backing device is for this resource.
 	disk /dev/sda5;
+	# This controls the location of the metadata. When "internal" is used,
+	# as we use here, a little space at the end of the backing devices is
+	# set aside (roughly 32 MB per 1 TB of raw storage). External metadata
+	# can be used to put the metadata on another partition when converting
+	# existing file systems to be DRBD backed, when there is no extra space
+	# available for the metadata.
 	meta-disk internal;
+	# NOTE: this is not required or even recommended with pacemaker. remove
+	# 	this options as soon as pacemaker is setup.
+	startup {
+		# This tells DRBD to promote both nodes to 'primary' when this
+		# resource starts. However, we will let pacemaker control this
+		# so we comment it out, which tells DRBD to leave both nodes
+		# as secondary when drbd starts.
+		#become-primary-on both;
+	}
+	# NOTE: Later, make it an option in the dashboard to trigger a manual
+	# 	verify and/or schedule periodic automatic runs
+	net {
+		# TODO: Test performance differences between sha1 and md5
+		# This tells DRBD how to do a block-by-block verification of
+		# the data stored on the backing devices. Any verification
+		# failures will result in the effected block being marked
+		# out-of-sync.
+		verify-alg md5;
+		# TODO: Test the performance hit of this being enabled.
+		# This tells DRBD to generate a checksum for each transmitted
+		# packet. If the data received data doesn't generate the same
+		# sum, a retransmit request is generated. This protects against
+		# otherwise-undetected errors in transmission, like
+		# bit-flipping. See:
+		# http://www.drbd.org/users-guide/s-integrity-check.html
+		data-integrity-alg md5;
+	}
+	# WARNING: Confirm that these are safe when the controller's BBU is
+	#          depleted/failed and the controller enters write-through
+	#          mode.
+	disk {
+		# TODO: Test the real-world performance differences gained with
+		#       these options.
+		# This tells DRBD not to bypass the write-back caching on the
+		# RAID controller. Normally, DRBD forces the data to be flushed
+		# to disk, rather than allowing the write-back cachine to
+		# handle it. Normally this is dangerous, but with BBU-backed
+		# caching, it is safe. The first option disables disk flushing
+		# and the second disabled metadata flushes.
+		disk-flushes no;
+		md-flushes no;
+	}
 	# This sets up the resource on node 01. The name used below must be the
 	# named returned by "uname -n".
@@ Line 1,422: / Line 1,540: @@
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-vim /etc/drbd.d/r0.res
+vim /etc/drbd.d/r1.res
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
+# This is the first DRBD resource. It will store the servers designed
+# to run on node 02.
 resource r1 {
 	device /dev/drbd1;
 	disk /dev/sda6;
 	meta-disk internal;
+	net {
+		verify-alg md5;
+		data-integrity-alg md5;
+	}
+	disk {
+		disk-flushes no;
+		md-flushes no;
+	}
 	on an-a04n01.alteeve.ca {
 		address 10.10.40.1:7789;
@@ Line 1,450: / Line 1,580: @@
 # /etc/drbd.conf
 common {
-    protocol               C;
-    net {
-        allow-two-primaries;
-        after-sb-0pri    discard-zero-changes;
-        after-sb-1pri    discard-secondary;
-        after-sb-2pri    disconnect;
-    }
-    disk {
-        fencing          resource-and-stonith;
-    }
-    syncer {
-        rate             40M;
-    }
-    handlers {
-        fence-peer       /usr/lib/drbd/stonith_admin-fence-peer.sh;
-    }
 }
 # resource r0 on an-a04n01.alteeve.ca: not ignored, not stacked
+# defined at /etc/drbd.d/r0.res:3
 resource r0 {
      on an-a04n01.alteeve.ca {
-         device           /dev/drbd0 minor 0;
+         volume 0 {
-         disk             /dev/sda5;
+            device       /dev/drbd0 minor 0;
+            disk         /dev/sda5;
+            meta-disk    internal;
+        }
          address          ipv4 10.10.40.1:7788;
-        meta-disk        internal;
      }
      on an-a04n02.alteeve.ca {
-         device           /dev/drbd0 minor 0;
+         volume 0 {
-         disk             /dev/sda5;
+            device       /dev/drbd0 minor 0;
+            disk         /dev/sda5;
+            meta-disk    internal;
+        }
          address          ipv4 10.10.40.2:7788;
-         meta-disk        internal;
+    }
+    net {
+         verify-alg       md5;
+        data-integrity-alg md5;
+    }
+    disk {
+        disk-flushes      no;
+        md-flushes        no;
      }
 }
 # resource r1 on an-a04n01.alteeve.ca: not ignored, not stacked
+# defined at /etc/drbd.d/r1.res:3
 resource r1 {
      on an-a04n01.alteeve.ca {
-         device           /dev/drbd1 minor 1;
+         volume 0 {
-         disk             /dev/sda6;
+            device       /dev/drbd1 minor 1;
+            disk         /dev/sda6;
+            meta-disk    internal;
+        }
          address          ipv4 10.10.40.1:7789;
-        meta-disk        internal;
      }
      on an-a04n02.alteeve.ca {
-         device           /dev/drbd1 minor 1;
+         volume 0 {
-         disk             /dev/sda6;
+            device       /dev/drbd1 minor 1;
+            disk         /dev/sda6;
+            meta-disk    internal;
+        }
          address          ipv4 10.10.40.2:7789;
-         meta-disk        internal;
+    }
+    net {
+         verify-alg       md5;
+        data-integrity-alg md5;
+    }
+    disk {
+        disk-flushes      no;
+        md-flushes        no;
      }
 }
@@ Line 1,516: / Line 1,656: @@
 drbd.d/r1.res
-sent 1436 bytes  received 91 bytes  3054.00 bytes/sec
+sent 5738 bytes  received 73 bytes  11622.00 bytes/sec
-total size is 1316  speedup is 0.86
+total size is 5618  speedup is 0.97
 </syntaxhighlight>
 |-
@@ Line 1,527: / Line 1,667: @@
 # /etc/drbd.conf
 common {
-    protocol               C;
-    net {
-        allow-two-primaries;
-        after-sb-0pri    discard-zero-changes;
-        after-sb-1pri    discard-secondary;
-        after-sb-2pri    disconnect;
-    }
-    disk {
-        fencing          resource-and-stonith;
-    }
-    syncer {
-        rate             40M;
-    }
-    handlers {
-        fence-peer       /usr/lib/drbd/stonith_admin-fence-peer.sh;
-    }
 }
 # resource r0 on an-a04n02.alteeve.ca: not ignored, not stacked
+# defined at /etc/drbd.d/r0.res:3
 resource r0 {
      on an-a04n01.alteeve.ca {
-         device           /dev/drbd0 minor 0;
+         volume 0 {
-         disk             /dev/sda5;
+            device       /dev/drbd0 minor 0;
+            disk         /dev/sda5;
+            meta-disk    internal;
+        }
          address          ipv4 10.10.40.1:7788;
-        meta-disk        internal;
      }
      on an-a04n02.alteeve.ca {
-         device           /dev/drbd0 minor 0;
+         volume 0 {
-         disk             /dev/sda5;
+            device       /dev/drbd0 minor 0;
+            disk         /dev/sda5;
+            meta-disk    internal;
+        }
          address          ipv4 10.10.40.2:7788;
-         meta-disk        internal;
+    }
+    net {
+         verify-alg       md5;
+        data-integrity-alg md5;
+    }
+    disk {
+        disk-flushes      no;
+        md-flushes        no;
      }
 }
 # resource r1 on an-a04n02.alteeve.ca: not ignored, not stacked
+# defined at /etc/drbd.d/r1.res:3
 resource r1 {
      on an-a04n01.alteeve.ca {
-         device           /dev/drbd1 minor 1;
+         volume 0 {
-         disk             /dev/sda6;
+            device       /dev/drbd1 minor 1;
+            disk         /dev/sda6;
+            meta-disk    internal;
+        }
          address          ipv4 10.10.40.1:7789;
-        meta-disk        internal;
      }
      on an-a04n02.alteeve.ca {
-         device           /dev/drbd1 minor 1;
+         volume 0 {
-         disk             /dev/sda6;
+            device       /dev/drbd1 minor 1;
+            disk         /dev/sda6;
+            meta-disk    internal;
+        }
          address          ipv4 10.10.40.2:7789;
-         meta-disk        internal;
+    }
+    net {
+         verify-alg       md5;
+        data-integrity-alg md5;
+    }
+    disk {
+        disk-flushes      no;
+        md-flushes        no;
      }
 }
@@ Line 1,592: / Line 1,742: @@
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-drbd                  351432  0
+drbd                  333723  0
+libcrc32c               1246  1 drbd
 </syntaxhighlight>
 |-
@@ Line 1,601: / Line 1,752: @@
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-drbd                  351432  0
+drbd                  333723  0
+libcrc32c               1246  1 drbd
 </syntaxhighlight>
 |}
@@ Line 1,617: / Line 1,769: @@
 Writing meta data...
 initializing activity log
-NOT initialized bitmap
+NOT initializing bitmap
 New drbd meta data block successfully created.
 success
 Writing meta data...
 initializing activity log
-NOT initialized bitmap
+NOT initializing bitmap
 New drbd meta data block successfully created.
 success
@@ Line 1,634: / Line 1,786: @@
 Writing meta data...
 initializing activity log
-NOT initialized bitmap
+NOT initializing bitmap
 New drbd meta data block successfully created.
 success
 Writing meta data...
 initializing activity log
-NOT initialized bitmap
+NOT initializing bitmap
 New drbd meta data block successfully created.
 success
@@ Line 1,645: / Line 1,797: @@
 |}
-Attach the backing devices.
+Bring up the new resources.
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-drbdadm attach r{0,1}
+drbdadm up r{0,1}
 cat /proc/drbd
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-version: 8.3.16 (api:88/proto:86-97)
+version: 8.4.4 (api:1/proto:86-101)
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
+GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
-: cs:StandAlone ro:Secondary/Unknown ds:Inconsistent/DUnknown   r----s
+: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/Outdated C r----s
-     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:454762916
+     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:454762916
-: cs:StandAlone ro:Secondary/Unknown ds:Inconsistent/DUnknown   r----s
+: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/Outdated C r----s
-     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:396782732
+     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396782732
 </syntaxhighlight>
 |-
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-drbdadm attach r{0,1}
+drbdadm up r{0,1}
 cat /proc/drbd
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-version: 8.3.16 (api:88/proto:86-97)
+version: 8.4.4 (api:1/proto:86-101)
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
+GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
-: cs:StandAlone ro:Secondary/Unknown ds:Inconsistent/DUnknown   r----s
+: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
-     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:454762916
+     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:454762916
-: cs:StandAlone ro:Secondary/Unknown ds:Inconsistent/DUnknown   r----s
+: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
-     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:396782732
+     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396782732
 </syntaxhighlight>
 |}
-Connect the resources
+Neither node has data, so we'll arbitrarily force node 01 to become primary, then normally promote node 02 to primary.
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-drbdadm connect r{0,1}
+drbdadm primary --force r{0,1}
 cat /proc/drbd
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-version: 8.3.16 (api:88/proto:86-97)
+version: 8.4.4 (api:1/proto:86-101)
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
+GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
-: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----s
+: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
-     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:454762916
+     ns:2136 nr:0 dw:0 dr:2800 al:0 bm:0 lo:0 pe:3 ua:0 ap:0 ep:1 wo:d oos:454760880
-: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----s
+        [>....................] sync'ed:  0.1% (444100/444104)M
-     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:396782732
+        finish: 421:04:29 speed: 252 (252) K/sec
+: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
+     ns:24696 nr:0 dw:0 dr:25360 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:396758036
+        [>....................] sync'ed:  0.1% (387456/387480)M
+        finish: 35:33:06 speed: 3,084 (3,084) K/sec
 </syntaxhighlight>
 |-
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-drbdadm connect r{0,1}
+drbdadm primary r{0,1}
 cat /proc/drbd
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-version: 8.3.16 (api:88/proto:86-97)
+version: 8.4.4 (api:1/proto:86-101)
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
+GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by root@rhel6-builder.alteeve.ca, 2014-07-20 21:29:34
-: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----s
+: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
-     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:454762916
+     ns:0 nr:859488 dw:859432 dr:608 al:0 bm:52 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:453903484
-: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----s
+        [>....................] sync'ed:  0.2% (443264/444104)M
-     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:396782732
+        finish: 71:24:53 speed: 1,752 (4,428) want: 440 K/sec
+: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
+     ns:0 nr:1140588 dw:1140532 dr:608 al:0 bm:69 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:395642200
+        [>....................] sync'ed:  0.3% (386368/387480)M
+        finish: 70:30:41 speed: 1,548 (5,876) want: 4,400 K/sec
 </syntaxhighlight>
 |}
-After a couple seconds, we'll see that they've connected.
+The sync rate starts low, but it will continue to climb, you can keep an eye on it if you wish. DRBD 8.4 is smarter than 8.3 in that it will adjust the sync rate automatically based on load.
-{|class="wikitable"
+We can proceed now, we do not have to wait for the sync to complete.
-!<span class="code">an-a04n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+== Clustered LVM and GFS2 ==
-cat /proc/drbd
-</syntaxhighlight>
+Clustered LVM provides the logical volumes that will back our /shared GFS2 partition and the storage for the HA servers.
-<syntaxhighlight lang="text">
-version: 8.3.16 (api:88/proto:86-97)
+=== Configure lvm.conf ===
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
-: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
-    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:454762916
-: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
-    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:396782732
-</syntaxhighlight>
-|-
-!<span class="code">an-a04n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-cat /proc/drbd
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
-: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
-    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:454762916
-: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
-    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:396782732
-</syntaxhighlight>
-|}
-Now, on node 1 only, begin initial sync.
+Configure clustered LVM.
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-drbdadm -- --overwrite-data-of-peer primary r{0,1}
+sed -i.anvil 's^filter = \[ "a/\.\*/" \]^filter = \[ "a|/dev/drbd*|", "r/.*/" \]^' /etc/lvm/lvm.conf
-cat /proc/drbd
+sed -i 's/locking_type = 1$/locking_type = 3/' /etc/lvm/lvm.conf
+sed -i 's/fallback_to_local_locking = 1$/fallback_to_local_locking = 0/' /etc/lvm/lvm.conf
+diff -u /etc/lvm/lvm.conf.anvil /etc/lvm/lvm.conf
 </syntaxhighlight>
-<syntaxhighlight lang="text">
+<syntaxhighlight lang="diff">
-version: 8.3.16 (api:88/proto:86-97)
+--- /etc/lvm/lvm.conf.anvil	2013-10-30 04:10:42.000000000 -0400
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
++++ /etc/lvm/lvm.conf	2014-06-04 18:38:15.545166869 -0400
-: cs:Connected ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
+@@ -82,7 +82,7 @@
-    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:1 pe:0 ua:0 ap:1 ep:1 wo:f oos:454762916
-: cs:Connected ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
-    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:396782732
+     # By default we accept every block device:
+-    filter = [ "a/.*/" ]
++    filter = [ "a|/dev/drbd*|", "r/.*/" ]
+     # Exclude the cdrom drive
+     # filter = [ "r|/dev/cdrom|" ]
+@@ -459,7 +459,7 @@
+     # Type 3 uses built-in clustered locking.
+     # Type 4 uses read-only locking which forbids any operations that might
+     # change metadata.
+-    locking_type = 1
++    locking_type = 3
+     # Set to 0 to fail when a lock request cannot be satisfied immediately.
+     wait_for_locks = 1
+@@ -475,7 +475,7 @@
+     # to 1 an attempt will be made to use local file-based locking (type 1).
+     # If this succeeds, only commands against local volume groups will proceed.
+     # Volume Groups marked as clustered will be ignored.
+-    fallback_to_local_locking = 1
++    fallback_to_local_locking = 0
+     # Local non-LV directory that holds file-based locks while commands are
+     # in progress.  A directory like /tmp that may get wiped on reboot is OK.
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+rsync -av /etc/lvm/lvm.conf* root@an-a04n02:/etc/lvm/
 </syntaxhighlight>
-|}
+<syntaxhighlight lang="text">
+sending incremental file list
+lvm.conf
+lvm.conf.anvil
-After a second or two, we'll see the sync start.
+sent 47499 bytes  received 440 bytes  95878.00 bytes/sec
+total size is 89999  speedup is 1.88
-{|class="wikitable"
-!<span class="code">an-a04n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-cat /proc/drbd
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-version: 8.3.16 (api:88/proto:86-97)
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
-: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
-    ns:988 nr:0 dw:0 dr:1652 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:454761928
-	[>....................] sync'ed:  0.1% (444100/444104)M
-	finish: 315:48:24 speed: 328 (328) K/sec
-: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
-    ns:988 nr:0 dw:0 dr:1652 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:396781744
-	[>....................] sync'ed:  0.1% (387480/387480)M
-	finish: 275:32:34 speed: 328 (328) K/sec
 </syntaxhighlight>
 |-
-!<span class="code">an-a04n01</span>
+!<span class="code">an-a04n02</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-cat /proc/drbd
+diff -u /etc/lvm/lvm.conf.anvil /etc/lvm/lvm.conf
 </syntaxhighlight>
-<syntaxhighlight lang="text">
+<syntaxhighlight lang="diff">
-version: 8.3.16 (api:88/proto:86-97)
+--- /etc/lvm/lvm.conf.anvil	2013-10-30 04:10:42.000000000 -0400
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
++++ /etc/lvm/lvm.conf	2014-06-04 18:38:15.000000000 -0400
-: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
+@@ -82,7 +82,7 @@
-    ns:0 nr:8640 dw:8640 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:454754276
-	[>....................] sync'ed:  0.1% (444092/444104)M
-	finish: 428:35:16 speed: 276 (276) want: 250 K/sec
+     # By default we accept every block device:
-: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
+-    filter = [ "a/.*/" ]
-    ns:0 nr:8640 dw:8640 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:396774092
++    filter = [ "a|/dev/drbd*|", "r/.*/" ]
-	[>....................] sync'ed:  0.1% (387472/387480)M
-	finish: 373:56:37 speed: 276 (276) want: 250 K/sec
+     # Exclude the cdrom drive
-</syntaxhighlight>
+     # filter = [ "r|/dev/cdrom|" ]
-|}
+@@ -459,7 +459,7 @@
+     # Type 3 uses built-in clustered locking.
-Note how the sync rate is rather low? We can kick DRBD to get it to go up to our defined 40M.
+     # Type 4 uses read-only locking which forbids any operations that might
+     # change metadata.
+-    locking_type = 1
++    locking_type = 3
+     # Set to 0 to fail when a lock request cannot be satisfied immediately.
+     wait_for_locks = 1
+@@ -475,7 +475,7 @@
+     # to 1 an attempt will be made to use local file-based locking (type 1).
+     # If this succeeds, only commands against local volume groups will proceed.
+     # Volume Groups marked as clustered will be ignored.
+-    fallback_to_local_locking = 1
++    fallback_to_local_locking = 0
+     # Local non-LV directory that holds file-based locks while commands are
+     # in progress.  A directory like /tmp that may get wiped on reboot is OK.
+</syntaxhighlight>
+|}
+=== Start clvmd ===
+{{note|1=This will be moved to pacemaker shortly. We're enabling it here just long enough to configure pacemaker.}}
+Make sure the cluster is up (you could use 'pcs status', 'cman_tool status', etc):
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-drbdadm adjust all
+dlm_tool dump | grep node
-cat /proc/drbd
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-version: 8.3.16 (api:88/proto:86-97)
+1401921044 cluster node 1 added seq 68
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
+1401921044 set_configfs_node 1 10.20.40.1 local 1
-: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
+1401921044 cluster node 2 added seq 68
-    ns:57472 nr:0 dw:0 dr:58468 al:0 bm:3 lo:1 pe:0 ua:0 ap:0 ep:1 wo:f oos:454705444
+1401921044 set_configfs_node 2 10.20.40.2 local 0
-	[>....................] sync'ed:  0.1% (444048/444104)M
+1401921044 run protocol from nodeid 1
-	finish: 77:06:21 speed: 1,624 (540) K/sec
-: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
-    ns:57472 nr:0 dw:0 dr:58468 al:0 bm:3 lo:1 pe:0 ua:0 ap:0 ep:1 wo:f oos:396725260
-	[>....................] sync'ed:  0.1% (387424/387480)M
-	finish: 67:16:26 speed: 1,624 (540) K/sec
 </syntaxhighlight>
 |}
-?! No worries, it takes a little time to spool up. Check again after a minute or so.
+Make sure DRBD is up as primary on both nodes:
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
 cat /proc/drbd
 </syntaxhighlight>
 <syntaxhighlight lang="text">
 version: 8.3.16 (api:88/proto:86-97)
 GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
-: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
+: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r-----
-     ns:991360 nr:0 dw:0 dr:992356 al:0 bm:60 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:453771556
+     ns:1519672 nr:0 dw:0 dr:1520336 al:0 bm:93 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:448214308
-	[>....................] sync'ed:  0.3% (443136/444104)M
+	[>....................] sync'ed:  0.4% (437708/439192)M
-	finish: 6:18:08 speed: 19,996 (6,044) K/sec
+	finish: 6:20:02 speed: 19,652 (15,992) K/sec
-: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
+: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r-----
-     ns:770176 nr:0 dw:0 dr:771172 al:0 bm:47 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:396012556
+     ns:1896504 nr:0 dw:0 dr:1897168 al:0 bm:115 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:390577164
-	[>....................] sync'ed:  0.2% (386728/387480)M
+	[>....................] sync'ed:  0.5% (381420/383272)M
-	finish: 9:53:20 speed: 11,108 (4,696) K/sec
+	finish: 2:33:17 speed: 42,440 (19,960) K/sec
 </syntaxhighlight>
 |}
-It will continue to climb, you can keep an eye on it if you wish.
+Note that we don't have to wait for the sync to finish.
-We can proceed now, we do not have to wait for the sync to complete.
+Start clvmd;
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+/etc/init.d/clvmd start
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+Starting clvmd:
+Activating VG(s):   No volume groups found
+                                                           [  OK  ]
+</syntaxhighlight>
+|-
+!<span class="code">an-a04n02</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+/etc/init.d/clvmd start
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+Starting clvmd:
+Activating VG(s):   No volume groups found
+                                                           [  OK  ]
+</syntaxhighlight>
+|}
-=== Promote DRBD to Primary on both nodes ===
+{{note|1=If this fails, showing a timeout or simply never returning, make sure that TCP port 21064 is opened in your firewall on both nodes.}}
-We're only going to do this during setup. Once we're done, pacemaker will handle starting and stopping drbd for us.
+From here on, pacemaker will start clvmd when pacemaker itself start, *if* clvmd is set to start on boot. So lets set that.
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-drbdadm primary r{0,1}
+chkconfig clvmd on
-cat /proc/drbd
+chkconfig --list clvmd
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-version: 8.3.16 (api:88/proto:86-97)
+clvmd          	0:off	1:off	2:on	3:on	4:on	5:on	6:off
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
-: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
-    ns:3740032 nr:0 dw:0 dr:3741028 al:0 bm:228 lo:30 pe:2 ua:30 ap:0 ep:1 wo:f oos:451023140
-	[>....................] sync'ed:  0.9% (440452/444104)M
-	finish: 9:04:17 speed: 13,788 (12,260) K/sec
-: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---n-
-    ns:2299748 nr:0 dw:0 dr:2301924 al:0 bm:140 lo:9 pe:10 ua:19 ap:0 ep:1 wo:f oos:394484236
-	[>....................] sync'ed:  0.6% (385236/387480)M
-	finish: 7:55:20 speed: 13,816 (7,536) K/sec
 </syntaxhighlight>
 |-
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-drbdadm primary r{0,1}
+chkconfig clvmd on
-cat /proc/drbd
+chkconfig --list clvmd
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-version: 8.3.16 (api:88/proto:86-97)
+clvmd          	0:off	1:off	2:on	3:on	4:on	5:on	6:off
-GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by root@rhel6-builder.alteeve.ca, 2014-04-20 12:16:31
-: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
-    ns:0 nr:3748480 dw:3748480 dr:0 al:0 bm:228 lo:1 pe:28 ua:0 ap:0 ep:1 wo:f oos:451014436
-	[>....................] sync'ed:  0.9% (440440/444104)M
-	finish: 8:47:24 speed: 14,248 (12,248) want: 40,960 K/sec
-: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
-    ns:0 nr:2309888 dw:2309632 dr:0 al:0 bm:140 lo:3 pe:3 ua:2 ap:0 ep:1 wo:f oos:394473100
-	[>....................] sync'ed:  0.6% (385224/387480)M
-	finish: 7:35:53 speed: 14,400 (7,544) want: 40,960 K/sec
 </syntaxhighlight>
 |}
-There, both resources are dual primary.
+=== Create Initial PVs, VGs and the /shared LV ===
-TODO: Test DRBD-triggered fencing via an errant iptables rule.
+Create the [[PV]], [[VG]] and the <span class="code">/shared</span> [[LV]];
-== DLM, Clustered LVM and GFS2 ==
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-sed -i.anvil 's^filter = \[ "a/\.\*/" \]^filter = \[ "a|/dev/drbd*|", "r/.*/" \]^' /etc/lvm/lvm.conf
+pvcreate /dev/drbd{0,1}
-sed -i 's/locking_type = 1$/locking_type = 3/' /etc/lvm/lvm.conf
+</syntaxhighlight>
-sed -i 's/fallback_to_local_locking = 1$/fallback_to_local_locking = 0/' /etc/lvm/lvm.conf
+<syntaxhighlight lang="text">
-sed -i 's/use_lvmetad = 1$/use_lvmetad = 0/' /etc/lvm/lvm.conf
+  Physical volume "/dev/drbd0" successfully created
-</syntaxhighlight>
+  Physical volume "/dev/drbd1" successfully created
-<syntaxhighlight lang="diff">
---- /etc/lvm/lvm.conf.anvil	2013-11-27 03:28:08.000000000 -0500
-+++ /etc/lvm/lvm.conf	2014-01-26 18:57:41.026928464 -0500
-@@ -84,7 +84,7 @@
-     # lvmetad is used" comment that is attached to global/use_lvmetad setting.
-     # By default we accept every block device:
--    filter = [ "a/.*/" ]
-+    filter = [ "a|/dev/drbd*|", "r/.*/" ]
-     # Exclude the cdrom drive
-     # filter = [ "r|/dev/cdrom|" ]
-@@ -451,7 +451,7 @@
-     # supported in clustered environment. If use_lvmetad=1 and locking_type=3
-     # is set at the same time, LVM always issues a warning message about this
-     # and then it automatically disables lvmetad use.
--    locking_type = 1
-+    locking_type = 3
-     # Set to 0 to fail when a lock request cannot be satisfied immediately.
-     wait_for_locks = 1
-@@ -467,7 +467,7 @@
-     # to 1 an attempt will be made to use local file-based locking (type 1).
-     # If this succeeds, only commands against local volume groups will proceed.
-     # Volume Groups marked as clustered will be ignored.
--    fallback_to_local_locking = 1
-+    fallback_to_local_locking = 0
-     # Local non-LV directory that holds file-based locks while commands are
-     # in progress.  A directory like /tmp that may get wiped on reboot is OK.
-@@ -594,7 +594,7 @@
-     # supported in clustered environment. If use_lvmetad=1 and locking_type=3
-     # is set at the same time, LVM always issues a warning message about this
-     # and then it automatically disables lvmetad use.
--    use_lvmetad = 1
-+    use_lvmetad = 0
-     # Full path of the utility called to check that a thin metadata device
-     # is in a state that allows it to be used.
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-rsync -av /etc/lvm/lvm.conf* root@an-a04n02:/etc/lvm/
+vgcreate an-a04n01_vg0 /dev/drbd0
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-sending incremental file list
+  Clustered volume group "an-a04n01_vg0" successfully created
-lvm.conf
+</syntaxhighlight>
-lvm.conf.anvil
+<syntaxhighlight lang="bash">
+vgcreate an-a04n02_vg0 /dev/drbd1
-sent 48536 bytes  received 440 bytes  97952.00 bytes/sec
+</syntaxhighlight>
-total size is 90673  speedup is 1.85
+<syntaxhighlight lang="text">
+  Clustered volume group "an-a04n02_vg0" successfully created
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+lvcreate -L 40GiB -n shared an-a04n01_vg0
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+  Logical volume "shared" created
 </syntaxhighlight>
 |-
 !<span class="code">an-a04n02</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-diff -u /etc/lvm/lvm.conf.anvil /etc/lvm/lvm.conf
+pvdisplay
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+  --- Physical volume ---
+  PV Name               /dev/drbd1
+  VG Name               an-a04n02_vg0
+  PV Size               378.40 GiB / not usable 3.14 MiB
+  Allocatable           yes
+  PE Size               4.00 MiB
+  Total PE              96870
+  Free PE               96870
+  Allocated PE          0
+  PV UUID               TpEXBC-7822-UGz0-ICz1-AJdg-v5eS-lyB7C5
+  --- Physical volume ---
+  PV Name               /dev/drbd0
+  VG Name               an-a04n01_vg0
+  PV Size               433.70 GiB / not usable 4.41 MiB
+  Allocatable           yes
+  PE Size               4.00 MiB
+  Total PE              111025
+  Free PE               100785
+  Allocated PE          10240
+  PV UUID               RoHAJQ-qrsO-Ofwz-f8W7-jIXd-2cvG-oPgfFR
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+vgdisplay
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+  --- Volume group ---
+  VG Name               an-a04n02_vg0
+  System ID
+  Format                lvm2
+  Metadata Areas        1
+  Metadata Sequence No  1
+  VG Access             read/write
+  VG Status             resizable
+  Clustered             yes
+  Shared                no
+  MAX LV                0
+  Cur LV                0
+  Open LV               0
+  Max PV                0
+  Cur PV                1
+  Act PV                1
+  VG Size               378.40 GiB
+  PE Size               4.00 MiB
+  Total PE              96870
+  Alloc PE / Size       0 / 0
+  Free  PE / Size       96870 / 378.40 GiB
+  VG UUID               9bTBDu-JSma-kwKR-4oBI-sxi1-YT6i-1uIM4C
+  --- Volume group ---
+  VG Name               an-a04n01_vg0
+  System ID
+  Format                lvm2
+  Metadata Areas        1
+  Metadata Sequence No  2
+  VG Access             read/write
+  VG Status             resizable
+  Clustered             yes
+  Shared                no
+  MAX LV                0
+  Cur LV                1
+  Open LV               0
+  Max PV                0
+  Cur PV                1
+  Act PV                1
+  VG Size               433.69 GiB
+  PE Size               4.00 MiB
+  Total PE              111025
+  Alloc PE / Size       10240 / 40.00 GiB
+  Free  PE / Size       100785 / 393.69 GiB
+  VG UUID               hLnvle-EScm-cP1t-xodO-cKyv-5EyC-TyIpj5
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+lvdisplay
 </syntaxhighlight>
-<syntaxhighlight lang="diff">
+<syntaxhighlight lang="text">
---- /etc/lvm/lvm.conf.anvil	2013-11-27 03:28:08.000000000 -0500
+  --- Logical volume ---
-+++ /etc/lvm/lvm.conf	2014-01-26 18:57:41.000000000 -0500
+  LV Path                /dev/an-a04n01_vg0/shared
-@@ -84,7 +84,7 @@
+  LV Name                shared
-     # lvmetad is used" comment that is attached to global/use_lvmetad setting.
+  VG Name                an-a04n01_vg0
+  LV UUID                tvolRF-cb3L-29Dn-Vgqd-e4rf-Qq2e-JFIcbA
-     # By default we accept every block device:
+  LV Write Access        read/write
--    filter = [ "a/.*/" ]
+  LV Creation host, time an-a04n01.alteeve.ca, 2014-06-07 18:54:41 -0400
-+    filter = [ "a|/dev/drbd*|", "r/.*/" ]
+  LV Status              available
+  # open                 0
-     # Exclude the cdrom drive
+  LV Size                40.00 GiB
-     # filter = [ "r|/dev/cdrom|" ]
+  Current LE             10240
-@@ -451,7 +451,7 @@
+  Segments               1
-     # supported in clustered environment. If use_lvmetad=1 and locking_type=3
+  Allocation             inherit
-     # is set at the same time, LVM always issues a warning message about this
+  Read ahead sectors     auto
-     # and then it automatically disables lvmetad use.
+  - currently set to     256
--    locking_type = 1
+  Block device           253:0
-+    locking_type = 3
-     # Set to 0 to fail when a lock request cannot be satisfied immediately.
-     wait_for_locks = 1
-@@ -467,7 +467,7 @@
-     # to 1 an attempt will be made to use local file-based locking (type 1).
-     # If this succeeds, only commands against local volume groups will proceed.
-     # Volume Groups marked as clustered will be ignored.
--    fallback_to_local_locking = 1
-+    fallback_to_local_locking = 0
-     # Local non-LV directory that holds file-based locks while commands are
-     # in progress.  A directory like /tmp that may get wiped on reboot is OK.
-@@ -594,7 +594,7 @@
-     # supported in clustered environment. If use_lvmetad=1 and locking_type=3
-     # is set at the same time, LVM always issues a warning message about this
-     # and then it automatically disables lvmetad use.
--    use_lvmetad = 1
-+    use_lvmetad = 0
-     # Full path of the utility called to check that a thin metadata device
-     # is in a state that allows it to be used.
 </syntaxhighlight>
 |}
-Disable <span class="code">lvmetad</span> as it's not cluster-aware.
+=== Create the /shared GFS2 filesystem ===
+Format the <span class="code">/dev/an-a04n01_vg0/shared</span> logical volume as a GFS2 filesystem;
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-systemctl disable lvm2-lvmetad.service
+mkfs.gfs2 -j 2 -p lock_dlm -t an-anvil-04:shared /dev/an-a04n01_vg0/shared
-systemctl disable lvm2-lvmetad.socket
+</syntaxhighlight>
-systemctl stop lvm2-lvmetad.service
+<syntaxhighlight lang="text">
+This will destroy any data on /dev/an-a04n01_vg0/shared.
+It appears to contain: symbolic link to `../dm-0'
+Are you sure you want to proceed? [y/n] y
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-rm '/etc/systemd/system/sockets.target.wants/lvm2-lvmetad.socket'
+Device:                    /dev/an-a04n01_vg0/shared
+Blocksize:                 4096
+Device Size                40.00 GB (10485760 blocks)
+Filesystem Size:           40.00 GB (10485758 blocks)
+Journals:                  2
+Resource Groups:           160
+Locking Protocol:          "lock_dlm"
+Lock Table:                "an-anvil-04:shared"
+UUID:                      e07d35fe-6860-f790-38cd-af075366c27b
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+mkdir /shared
+mount /dev/an-a04n01_vg0/shared /shared
+df -hP
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+Filesystem                         Size  Used Avail Use% Mounted on
+/dev/sda3                           20G  1.5G   18G   8% /
+tmpfs                               12G   67M   12G   1% /dev/shm
+/dev/sda1                          504M   72M  407M  16% /boot
+/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared
 </syntaxhighlight>
 |-
 !<span class="code">an-a04n02</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-systemctl disable lvm2-lvmetad.service
+mkdir /shared
-systemctl disable lvm2-lvmetad.socket
+mount /dev/an-a04n01_vg0/shared /shared
-systemctl stop lvm2-lvmetad.service
+df -hP
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-rm '/etc/systemd/system/sockets.target.wants/lvm2-lvmetad.socket'
+Filesystem                         Size  Used Avail Use% Mounted on
+/dev/sda3                           20G  1.5G   18G   8% /
+tmpfs                               12G   52M   12G   1% /dev/shm
+/dev/sda1                          504M   72M  407M  16% /boot
+/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared
 </syntaxhighlight>
 |}
-{{note|1=This will be moved to pacemaker shortly. We're enabling it here just long enough to configure pacemaker.}}
+= Add Storage to Pacemaker =
+== Configure Dual-Primary DRBD ==
+Setup DRBD as a dual-primary resource.
-Start DLM and clvmd;
+Notes:
+* Clones allow for a given service to run on multiple nodes.
+** master-max is how many copies of the resource can be promoted to master at the same time across the cluster.
+** master-node-max is how many copies of the resource can be promoted to master on a given node.
+** clone-max is how many copies can run in the cluster, default is to the number of nodes in the cluster.
+** clone-node-max is the number of instances of the resource that can run on each node.
+** notify controls whether other nodes are notified before and after a resource is started or stopped on a given node.
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-systemctl start dlm.service
+pcs cluster cib drbd_cfg
-systemctl start clvmd.service
+pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 op monitor interval=10s
+pcs -f drbd_cfg resource create drbd_r1 ocf:linbit:drbd drbd_resource=r1 op monitor interval=10s
+### Ignore this for now.
+#pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 \
+#                op monitor interval=29s role=Master \
+#                op monitor interval=31s role=Slave \
+#                op promote interval=0 timeout=90s start-delay=2s \
+#                op start interval=0 timeout=240s \
+#                op stop interval=0 timeout=120s
+pcs -f drbd_cfg resource master drbd_r0_Clone drbd_r0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
+pcs -f drbd_cfg resource master drbd_r1_Clone drbd_r1 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
+pcs cluster cib-push drbd_cfg
 </syntaxhighlight>
-|-
+<syntaxhighlight lang="text">
-!<span class="code">an-a04n02</span>
+CIB updated
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-systemctl start dlm.service
-systemctl start clvmd.service
 </syntaxhighlight>
 |}
-Create the [[PV]], [[VG]] and the <span class="code">/shared</span> [[LV]];
+Give it a couple minutes to promote both nodes to <span class="code">Master</span> on both nodes. Initially, it will appear as <span class="code">Master</span> on one node only.
+Once updated, you should see this:
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pvcreate /dev/drbd0
+pcs status
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-  Physical volume "/dev/drbd0" successfully created
+Cluster name: an-anvil-04
-</syntaxhighlight>
+Last updated: Sat Jun  7 20:29:09 2014
-<syntaxhighlight lang="bash">
+Last change: Sat Jun  7 20:28:36 2014 via cibadmin on an-a04n01.alteeve.ca
-vgcreate an-a04n01_vg0 /dev/drbd0
+Stack: cman
-</syntaxhighlight>
+Current DC: an-a04n01.alteeve.ca - partition with quorum
-<syntaxhighlight lang="text">
+Version: 1.1.10-14.el6_5.3-368c726
-  /proc/devices: No entry for device-mapper found
+Nodes configured
-  Clustered volume group "an-a04n01_vg0" successfully created
+Resources configured
-</syntaxhighlight>
-<syntaxhighlight lang="bash">
-lvcreate -L 10G -n shared an-a04n01_vg0
+Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-</syntaxhighlight>
-<syntaxhighlight lang="text">
+Full list of resources:
-  Logical volume "shared" created
+ fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca
+ fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca
+ Master/Slave Set: drbd_r0_Clone [drbd_r0]
+     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
+ Master/Slave Set: drbd_r1_Clone [drbd_r1]
+     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
 </syntaxhighlight>
 |-
 !<span class="code">an-a04n02</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pvscan
+pcs status
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-  PV /dev/drbd0   VG an-a04n01_vg0   lvm2 [20.00 GiB / 20.00 GiB free]
+Cluster name: an-anvil-04
-  Total: 1 [20.00 GiB] / in use: 1 [20.00 GiB] / in no VG: 0 [0   ]
+Last updated: Sat Jun  7 20:29:36 2014
-</syntaxhighlight>
+Last change: Sat Jun  7 20:28:36 2014 via cibadmin on an-a04n01.alteeve.ca
-<syntaxhighlight lang="bash">
+Stack: cman
-vgscan
+Current DC: an-a04n01.alteeve.ca - partition with quorum
-</syntaxhighlight>
+Version: 1.1.10-14.el6_5.3-368c726
-<syntaxhighlight lang="text">
+Nodes configured
-  Reading all physical volumes.  This may take a while...
+Resources configured
-  Found volume group "an-a04n01_vg0" using metadata type lvm2
-</syntaxhighlight>
-<syntaxhighlight lang="bash">
+Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-lvscan
+Full list of resources:
+ fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca
+ fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca
+ Master/Slave Set: drbd_r0_Clone [drbd_r0]
+     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
+ Master/Slave Set: drbd_r1_Clone [drbd_r1]
+     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
+</syntaxhighlight>
+|}
+=== Configure LVM ===
+We need to have pacemaker activate our clustered LVM LVs on start, and deactivate them when stopping. We don't start/stop clvmd directly because of stop timing issues that can lead to stray fencing.
+{{note|1=This will throw errors if there are no LVs on a given VG... Do not add a volume group until at least one logical volume has been created.}}
+{|class="wikitable"
+!<span class="code">an-a04n01</span>
+|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+pcs cluster cib lvm_cfg
+pcs -f lvm_cfg resource create lvm_n01_vg0 ocf:heartbeat:lvm volgrpname=an-a04n01_vg0 op monitor interval=10s
+pcs -f lvm_cfg resource master lvm_n01_vg0_Clone lvm_n01_vg0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
+pcs cluster cib-push lvm_cfg
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-  ACTIVE            '/dev/an-a04n01_vg0/shared' [10.00 GiB] inherit
+CIB updated
 </syntaxhighlight>
 |}
-Format the <span class="code">/dev/an-a04n01_vg0/shared</span>;
+== Configure LVM to start after the DRBD PV is Primary ==
+It we stopped here, there is a good chance that on future starts of pacemaker, LVM and DRBD would start in parallel, DRBD would take too long, LVM would error out and stonith's would start to fly. To prevent this, we will tell Pacemaker not to start the LVM resource until after the DRBD resource that is behind the volume group has been promoted to primary.
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-mkfs.gfs2 -j 2 -p lock_dlm -t an-cluster-03:shared /dev/an-a04n01_vg0/shared
+pcs cluster cib cst_cfg
+pcs -f cst_cfg constraint order promote drbd_r0_Clone then start lvm_n01_vg0_Clone
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-/dev/an-a04n01_vg0/shared is a symbolic link to /dev/dm-0
+Adding drbd_r0_Clone lvm_n01_vg0_Clone (kind: Mandatory) (Options: first-action=promote then-action=start)
-This will destroy any data on /dev/dm-0
 </syntaxhighlight>
-<syntaxhighlight lang="text">
+<syntaxhighlight lang="bash">
-Are you sure you want to proceed? [y/n]y
+pcs cluster cib-push cst_cfg
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Device:                    /dev/an-a04n01_vg0/shared
+CIB updated
-Block size:                4096
-Device size:               10.00 GB (2621440 blocks)
-Filesystem size:           10.00 GB (2621438 blocks)
-Journals:                  2
-Resource groups:           40
-Locking protocol:          "lock_dlm"
-Lock table:                "an-cluster-03:shared"
-UUID:                      20bafdb0-1f86-f424-405b-9bf608c0c486
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-mkdir /shared
+pcs constraint show
-mount /dev/an-a04n01_vg0/shared /shared
-df -h
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-Filesystem                         Size  Used Avail Use% Mounted on
+Location Constraints:
-/dev/vda3                           18G  5.6G   12G  32% /
+Ordering Constraints:
-devtmpfs                           932M     0  932M   0% /dev
+   promote drbd_r0_Clone then start lvm_n01_vg0_Clone
-tmpfs                              937M   61M  877M   7% /dev/shm
+Colocation Constraints:
-tmpfs                              937M  1.9M  935M   1% /run
-tmpfs                              937M     0  937M   0% /sys/fs/cgroup
-/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
-/dev/vda1                          484M   83M  401M  18% /boot
-/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared
-</syntaxhighlight>
-|-
-!<span class="code">an-a04n02</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-Filesystem                         Size  Used Avail Use% Mounted on
-/dev/vda3                           18G  5.6G   12G  32% /
-devtmpfs                           932M     0  932M   0% /dev
-tmpfs                              937M   76M  862M   9% /dev/shm
-tmpfs                              937M  2.0M  935M   1% /run
-tmpfs                              937M     0  937M   0% /sys/fs/cgroup
-/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
-/dev/vda1                          484M   83M  401M  18% /boot
-/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared
 </syntaxhighlight>
 |}
-Shut down <span class="code">gfs2</span>, <span class="code">clvmd</span> and <span class="code">drbd</span> now.
+== Configure the /shared GFS2 Partition ==
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-umount /shared/
+pcs cluster cib fs_cfg
-systemctl stop clvmd.service
+pcs -f fs_cfg resource create sharedFS Filesystem device="/dev/an-a04n01_vg0/shared" directory="/shared" fstype="gfs2"
-drbdadm down r0
+pcs -f fs_cfg resource clone sharedFS master-max=2 master-node-max=1 clone-max=2 clone-node-max=1
+pcs cluster cib-push fs_cfg
+</syntaxhighlight>
+<syntaxhighlight lang="text">
+CIB updated
 </syntaxhighlight>
-|-
+<syntaxhighlight lang="bash">
-!<span class="code">an-a04n02</span>
+pcs status
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-umount /shared/
-systemctl stop clvmd.service
-drbdadm down r0
 </syntaxhighlight>
-|}
+<syntaxhighlight lang="text">
+Cluster name: an-anvil-04
-Done.
+Last updated: Sat Jun  7 21:09:28 2014
+Last change: Sat Jun  7 21:08:47 2014 via cibadmin on an-a04n01.alteeve.ca
+Stack: cman
+Current DC: an-a04n01.alteeve.ca - partition with quorum
+Version: 1.1.10-14.el6_5.3-368c726
+Nodes configured
+Resources configured
+Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-{|class="wikitable"
+Full list of resources:
-!<span class="code">an-a04n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+ fence_n01_ipmi	(stonith:fence_ipmilan):	Started an-a04n01.alteeve.ca
+ fence_n02_ipmi	(stonith:fence_ipmilan):	Started an-a04n02.alteeve.ca
+ Master/Slave Set: drbd_r0_Clone [drbd_r0]
+     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
+ Master/Slave Set: drbd_r1_Clone [drbd_r1]
+     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
+ Clone Set: sharedFS-clone [sharedFS]
+     Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
+</syntaxhighlight>
+<syntaxhighlight lang="bash">
+df -hP
 </syntaxhighlight>
 <syntaxhighlight lang="text">
+Filesystem                         Size  Used Avail Use% Mounted on
+/dev/sda3                           20G  1.5G   18G   8% /
+tmpfs                               12G   67M   12G   1% /dev/shm
+/dev/sda1                          504M   72M  407M  16% /boot
+/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared
 </syntaxhighlight>
 |-
 !<span class="code">an-a04n02</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
+df -h
 </syntaxhighlight>
 <syntaxhighlight lang="text">
+Filesystem                         Size  Used Avail Use% Mounted on
+/dev/sda3                           20G  1.5G   18G   8% /
+tmpfs                               12G   52M   12G   1% /dev/shm
+/dev/sda1                          504M   72M  407M  16% /boot
+/dev/mapper/an--a04n01_vg0-shared   40G  259M   40G   1% /shared
 </syntaxhighlight>
 |}
-= Add Storage to Pacemaker =
+== Configur /shared to start after LVM ==
-== Configure Dual-Primary DRBD ==
+As we did before in making sure LVM started after DRBD, this time we will make sure LVM starts before /shared is mounted.
-Setup DRBD as a dual-primary resource.
 {|class="wikitable"
 !<span class="code">an-a04n01</span>
 |style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs cluster cib drbd_cfg
+pcs cluster cib cst_cfg
-pcs -f drbd_cfg resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 op monitor interval=60s
+pcs -f cst_cfg constraint order start lvm_n01_vg0_Clone then start sharedFS
-pcs -f drbd_cfg resource master drbd_r0_Clone drbd_r0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
-pcs cluster cib-push drbd_cfg
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-CIB updated
-</syntaxhighlight>
-|}
-Give it a couple minutes to promote both nodes to <span class="code">Master</span> on both nodes. Initially, it will appear as <span class="code">Master</span> on one node only.
-Once updated, you should see this:
-{|class="wikitable"
-!<span class="code">an-a04n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs status
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-Cluster name: an-cluster-03
-Last updated: Sun Jan 26 20:26:33 2014
-Last change: Sun Jan 26 20:23:23 2014 via cibadmin on an-a04n01.alteeve.ca
-Stack: corosync
-Current DC: an-a04n02.alteeve.ca (2) - partition with quorum
-Version: 1.1.10-19.el7-368c726
-Nodes configured
-Resources configured
-Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-Full list of resources:
- fence_n01_virsh	(stonith:fence_virsh):	Started an-a04n01.alteeve.ca
- fence_n02_virsh	(stonith:fence_virsh):	Started an-a04n02.alteeve.ca
- Master/Slave Set: drbd_r0_Clone [drbd_r0]
-     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-PCSD Status:
-an-a04n01.alteeve.ca:
-  an-a04n01.alteeve.ca: Online
-an-a04n02.alteeve.ca:
-  an-a04n02.alteeve.ca: Online
-Daemon Status:
-  corosync: active/disabled
-  pacemaker: active/disabled
-  pcsd: active/enabled
-</syntaxhighlight>
-|-
-!<span class="code">an-a04n02</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs status
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-Cluster name: an-cluster-03
-Last updated: Sun Jan 26 20:26:58 2014
-Last change: Sun Jan 26 20:23:23 2014 via cibadmin on an-a04n01.alteeve.ca
-Stack: corosync
-Current DC: an-a04n02.alteeve.ca (2) - partition with quorum
-Version: 1.1.10-19.el7-368c726
-Nodes configured
-Resources configured
-Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-Full list of resources:
- fence_n01_virsh	(stonith:fence_virsh):	Started an-a04n01.alteeve.ca
- fence_n02_virsh	(stonith:fence_virsh):	Started an-a04n02.alteeve.ca
- Master/Slave Set: drbd_r0_Clone [drbd_r0]
-     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-PCSD Status:
-an-a04n01.alteeve.ca:
-  an-a04n01.alteeve.ca: Online
-an-a04n02.alteeve.ca:
-  an-a04n02.alteeve.ca: Online
-Daemon Status:
-  corosync: active/disabled
-  pacemaker: active/disabled
-  pcsd: active/enabled
-</syntaxhighlight>
-|}
-== Configure DLM ==
-{|class="wikitable"
-!<span class="code">an-a04n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs cluster cib dlm_cfg
-pcs -f dlm_cfg resource create dlm ocf:pacemaker:controld op monitor interval=60s
-pcs -f dlm_cfg resource clone dlm clone-max=2 clone-node-max=1
-pcs cluster cib-push dlm_cfg
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-CIB updated
-</syntaxhighlight>
-|-
-!<span class="code">an-a04n02</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs status
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-Cluster name: an-cluster-03
-Last updated: Sun Jan 26 20:34:36 2014
-Last change: Sun Jan 26 20:33:31 2014 via cibadmin on an-a04n01.alteeve.ca
-Stack: corosync
-Current DC: an-a04n02.alteeve.ca (2) - partition with quorum
-Version: 1.1.10-19.el7-368c726
-Nodes configured
-Resources configured
-Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-Full list of resources:
- fence_n01_virsh	(stonith:fence_virsh):	Started an-a04n01.alteeve.ca
- fence_n02_virsh	(stonith:fence_virsh):	Started an-a04n02.alteeve.ca
- Master/Slave Set: drbd_r0_Clone [drbd_r0]
-     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
- Clone Set: dlm-clone [dlm]
-     Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-PCSD Status:
-an-a04n01.alteeve.ca:
-  an-a04n01.alteeve.ca: Online
-an-a04n02.alteeve.ca:
-  an-a04n02.alteeve.ca: Online
-Daemon Status:
-  corosync: active/disabled
-  pacemaker: active/disabled
-  pcsd: active/enabled
-</syntaxhighlight>
-|}
-== Configure Cluster LVM ==
-{|class="wikitable"
-!<span class="code">an-a04n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs cluster cib clvmd_cfg
-pcs -f clvmd_cfg resource create clvmd lsb:clvmd params daemon_timeout=30s op monitor interval=60s
-pcs -f clvmd_cfg resource clone clvmd clone-max=2 clone-node-max=1
-pcs -f clvmd_cfg constraint colocation add dlm-clone clvmd-clone INFINITY
-pcs -f clvmd_cfg constraint order start dlm then start clvmd-clone
-pcs cluster cib-push clvmd_cfg</syntaxhighlight>
-<syntaxhighlight lang="text">
-CIB updated
-</syntaxhighlight>
-|-
-!<span class="code">an-a04n02</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs status
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-Cluster name: an-cluster-03
-Last updated: Mon Jan 27 19:00:33 2014
-Last change: Mon Jan 27 19:00:19 2014 via crm_resource on an-a04n01.alteeve.ca
-Stack: corosync
-Current DC: an-a04n01.alteeve.ca (1) - partition with quorum
-Version: 1.1.10-19.el7-368c726
-Nodes configured
-Resources configured
-Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-Full list of resources:
- fence_n01_virsh        (stonith:fence_virsh):  Started an-a04n01.alteeve.ca
- fence_n02_virsh        (stonith:fence_virsh):  Started an-a04n02.alteeve.ca
- Master/Slave Set: drbd_r0_Clone [drbd_r0]
-     Masters: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
- Clone Set: dlm-clone [dlm]
-     Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
- Clone Set: clvmd-clone [clvmd]
-     Started: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
-PCSD Status:
-an-a04n01.alteeve.ca:
-  an-a04n01.alteeve.ca: Online
-an-a04n02.alteeve.ca:
-  an-a04n02.alteeve.ca: Online
-Daemon Status:
-  corosync: active/disabled
-  pacemaker: active/disabled
-  pcsd: active/enabled
-</syntaxhighlight>
-|}
-== Configure the /shared GFS2 Partition ==
-{|class="wikitable"
-!<span class="code">an-a04n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs cluster cib fs_cfg
-pcs -f fs_cfg resource create sharedFS Filesystem device="/dev/an-a04n01_vg0/shared" directory="/shared" fstype="gfs2"
-pcs -f fs_cfg resource clone sharedFS
-pcs cluster cib-push fs_cfg
 </syntaxhighlight>
 <syntaxhighlight lang="text">
-CIB updated
+Adding lvm_n01_vg0_Clone sharedFS (kind: Mandatory) (Options: first-action=start then-action=start)
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-df -h
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-Filesystem                         Size  Used Avail Use% Mounted on
-/dev/vda3                           18G  5.6G   12G  32% /
-devtmpfs                           932M     0  932M   0% /dev
-tmpfs                              937M   61M  877M   7% /dev/shm
-tmpfs                              937M  2.2M  935M   1% /run
-tmpfs                              937M     0  937M   0% /sys/fs/cgroup
-/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
-/dev/vda1                          484M   83M  401M  18% /boot
-/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared
-</syntaxhighlight>
-|-
-!<span class="code">an-a04n02</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-df -h
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-Filesystem                         Size  Used Avail Use% Mounted on
-/dev/vda3                           18G  5.6G   12G  32% /
-devtmpfs                           932M     0  932M   0% /dev
-tmpfs                              937M   76M  862M   9% /dev/shm
-tmpfs                              937M  2.6M  935M   1% /run
-tmpfs                              937M     0  937M   0% /sys/fs/cgroup
-/dev/loop0                         4.4G  4.4G     0 100% /mnt/dvd
-/dev/vda1                          484M   83M  401M  18% /boot
-/dev/mapper/an--a03n01_vg0-shared   10G  259M  9.8G   3% /shared
-</syntaxhighlight>
-|}
-== Configuring Constraints ==
-{|class="wikitable"
-!<span class="code">an-a04n01</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs cluster cib cst_cfg
-pcs -f cst_cfg constraint order start dlm then promote drbd_r0_Clone
-pcs -f cst_cfg constraint order promote drbd_r0_Clone then start clvmd-clone
-pcs -f cst_cfg constraint order promote clvmd-clone then start sharedFS-clone
 pcs cluster cib-push cst_cfg
 </syntaxhighlight>
@@ Line 2,447: / Line 2,475: @@
 </syntaxhighlight>
 <syntaxhighlight lang="bash">
-pcs constraint show
+pcs constraint show --full
 </syntaxhighlight>
 <syntaxhighlight lang="text">
 Location Constraints:
 Ordering Constraints:
-  start dlm then promote drbd_r0_Clone
+   promote drbd_r0_Clone then start lvm_n01_vg0_Clone (Mandatory) (id:order-drbd_r0_Clone-lvm_n01_vg0_Clone-mandatory)
-   promote drbd_r0_Clone then start clvmd-clone
+   start lvm_n01_vg0_Clone then start sharedFS-clone (Mandatory) (id:order-lvm_n01_vg0_Clone-sharedFS-clone-mandatory)
-   start clvmd-clone then start sharedFS-clone
-Colocation Constraints:
-</syntaxhighlight>
-|-
-!<span class="code">an-a04n02</span>
-|style="white-space: nowrap;"|<syntaxhighlight lang="bash">
-pcs constraint show
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-Location Constraints:
-Ordering Constraints:
-  start dlm then promote drbd_r0_Clone
-  promote drbd_r0_Clone then start clvmd-clone
-  start clvmd-clone then start sharedFS-clone
 Colocation Constraints:
 </syntaxhighlight>
 |}
-= Odds and Sods =
+Note that this time we added '--full'. If you ever need to delete a constraint, you would use 'pcs constraint delete <id>'.
-This is a section for random notes. The stuff here will be integrated into the finished tutorial or removed.
-== Determine multicast Address ==
-Useful if you need to ensure that your switch has persistent multicast addresses set.
-<syntaxhighlight lang="bash">
-corosync-cmapctl | grep mcastaddr
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-totem.interface.0.mcastaddr (str) = 239.192.122.199
-</syntaxhighlight>
-<span class="code"></span>
-<syntaxhighlight lang="bash">
-</syntaxhighlight>
-<syntaxhighlight lang="text">
-</syntaxhighlight>
-<syntaxhighlight lang="diff">
-</syntaxhighlight>
 = Notes =

Anvil! Tutorial 3 on EL6: Difference between revisions

Latest revision as of 12:23, 21 July 2014

Before We Begin

OS Setup

Post OS Install

Adding AN! Repo

Install

Setup Networking

Making ssh faster when the net is down

Setting the Hostname

Setup The hosts File

Setup SSH

Populating And Pushing ~/.ssh/known_hosts

Configuring the Firewall

Keeping Time in Sync

Configuring the Anvil!

Configuring cman

Starting Pacemaker

Configure and test stonith (aka fencing)

Configuring IPMI Fencing

Configuring Fence Levels

Test Fencing

Shared Storage

DRBD

Partition Storage

Configure DRBD

Start DRBD for the first time

Clustered LVM and GFS2

Configure lvm.conf

Start clvmd

Create Initial PVs, VGs and the /shared LV

Create the /shared GFS2 filesystem

Add Storage to Pacemaker

Configure Dual-Primary DRBD

Configure LVM

Configure LVM to start after the DRBD PV is Primary

Configure the /shared GFS2 Partition

Configur /shared to start after LVM

Notes

Thanks

Navigation menu