Anvil! Tutorial 3
 |
Warning: This tutorial is incomplete, flawed and generally sucks at this time. Do not follow this and expect anything to work. In large part, it's a dumping ground for notes and little else. This warning will be removed when the tutorial is completed. |
|
Warning: This tutorial is built on a guess of what Red Hat's Enterprise Linux 7 will offer, based on what the author sees happening in Fedora upstream. Red Hat never confirms what a future release will contain until it is actually released. As such, this tutorial may turn out to be inappropriate for the final release of RHEL 7. In such a case, the warning above will remain in place until the tutorial is updated to reflect the final release. |
This is the third AN!Cluster tutorial built on Red Hat's Enterprise Linux 7. It improves on the RHEL 5, RHCS stable 2 and RHEL 6, RHCS stable3 tutorials.
As with the previous tutorials, the end goal of this tutorial is a 2-node cluster providing a platform for high-availability virtual servers. It's design attempts to remove all single points of failure from the system. Power and networking are made fully redundant in this version, along with minimizing the node failures which would lead to service interruption. This tutorial also covers the AN!Utilities; AN!Cluster Dashboard, AN!Cluster Monitor and AN!Safe Cluster Shutdown.
As it the previous tutorial, KVM will be the hypervisor used for facilitating virtual machines. The old cman and rgmanager tools are replaced in favour of pacemaker for resource management.
Before We Begin
This tutorial does not require prior cluster experience, but it does expect familiarity with Linux and a low-intermediate understanding of networking. Where possible, steps are explained in detail and rationale is provided for why certain decisions are made.
For those with cluster experience;
Please be careful not to skip too much. There are some major and some subtle changes from previous tutorials.
OS Setup
|
Warning: I used Fedora 17 at this point, obviously things will change, possibly a lot, once RHEL 7 is released. |
Network
We want static, named network devices. Follow this;
Then, use these configuration files;
Build the bridge;
vim /etc/sysconfig/network-scripts/ifcfg-ifn-vbr1
# Internet-Facing Network - Bridge
DEVICE="ifn-vbr1"
TYPE="Bridge"
BOOTPROTO="none"
IPADDR="10.255.10.1"
NETMASK="255.255.0.0"
GATEWAY="10.255.255.254"
DNS1="8.8.8.8"
DNS2="8.8.4.4"
DEFROUTE="yes"
Now build the bonds;
vim /etc/sysconfig/network-scripts/ifcfg-ifn-bond1
# Internet-Facing Network - Bond
DEVICE="ifn-bond1"
BRIDGE="ifn-vbr1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=ifn1"
vim /etc/sysconfig/network-scripts/ifcfg-sn-bond1
# Storage Network - Bond
DEVICE="sn-bond1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=sn1"
IPADDR="10.10.10.1"
NETMASK="255.255.0.0"
vim /etc/sysconfig/network-scripts/ifcfg-bcn-bond1
# Back-Channel Network - Bond
DEVICE="bcn-bond1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=bcn1"
IPADDR="10.20.10.1"
NETMASK="255.255.0.0"
Now tell the interfaces to be slaves to their bonds;
Internet-Facing Network;
vim /etc/sysconfig/network-scripts/ifcfg-ifn1
# Internet-Facing Network - Link 1
DEVICE="ifn1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="ifn-bond1"
vim /etc/sysconfig/network-scripts/ifcfg-ifn2
# Back-Channel Network - Link 2
DEVICE="ifn2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="ifn-bond1"
Storage Network;
vim /etc/sysconfig/network-scripts/ifcfg-sn1
# Storage Network - Link 1
DEVICE="sn1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="sn-bond1"
vim /etc/sysconfig/network-scripts/ifcfg-sn2
# Storage Network - Link 1
DEVICE="sn2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="sn-bond1"
Back-Channel Network
vim /etc/sysconfig/network-scripts/ifcfg-bcn1
# Back-Channel Network - Link 1
DEVICE="bcn1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="bcn-bond1"
vim /etc/sysconfig/network-scripts/ifcfg-bcn2
# Storage Network - Link 1
DEVICE="bcn2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="bcn-bond1"
Now restart the network, confirm that the bonds and bridge are up and you are ready to proceed.
Setup The hosts File
You can use DNS if you prefer. For now, lets use /etc/hosts for node name resolution.
vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# AN!Cluster 01, Node 01
10.255.10.1 an-c01n01.ifn
10.10.10.1 an-c01n01.sn
10.20.10.1 an-c01n01.bcn an-c01n01 an-c01n01.alteeve.ca
10.20.11.1 an-c01n01.ipmi
# AN!Cluster 01, Node 02
10.255.10.2 an-c01n02.ifn
10.10.10.2 an-c01n02.sn
10.20.10.2 an-c01n02.bcn an-c01n02 an-c01n02.alteeve.ca
10.20.11.2 an-c01n02.ipmi
# Foundation Pack
10.20.2.7 an-p03 an-p03.alteeve.ca
Setup SSH
Same as before.
Installing Programs
From this point forward, this tutorial borrows heavily from Pacemaker 1.1, Clusters from Scratch.
yum install pacemaker corosync
Loaded plugins: langpacks, presto, refresh-packagekit
updates/metalink | 15 kB 00:00
updates | 4.7 kB 00:00
updates/primary_db | 6.2 MB 00:14
updates/group_gz | 434 kB 00:00
Resolving Dependencies
--> Running transaction check
---> Package corosync.x86_64 0:2.0.1-1.fc17 will be installed
--> Processing Dependency: corosynclib = 2.0.1-1.fc17 for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libxslt for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libvotequorum.so.5(COROSYNC_VOTEQUORUM_1.0)(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libquorum.so.5(COROSYNC_QUORUM_1.0)(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libcpg.so.4(COROSYNC_CPG_1.0)(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libcmap.so.4(COROSYNC_CMAP_1.0)(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libcfg.so.6(COROSYNC_CFG_0.82)(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libvotequorum.so.5()(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libtotem_pg.so.5()(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libquorum.so.5()(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libqb.so.0()(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libnetsnmp.so.30()(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libcpg.so.4()(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libcorosync_common.so.4()(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libcmap.so.4()(64bit) for package: corosync-2.0.1-1.fc17.x86_64
--> Processing Dependency: libcfg.so.6()(64bit) for package: corosync-2.0.1-1.fc17.x86_64
---> Package pacemaker.x86_64 0:1.1.7-2.fc17 will be installed
--> Processing Dependency: pacemaker-libs = 1.1.7-2.fc17 for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: pacemaker-cluster-libs = 1.1.7-2.fc17 for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: pacemaker-cli = 1.1.7-2.fc17 for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: resource-agents for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: cluster-glue for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libtransitioner.so.1()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libstonithd.so.1()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libstonith.so.1()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libplumb.so.2()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libpils.so.2()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libpengine.so.3()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libpe_status.so.3()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libpe_rules.so.2()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libltdl.so.7()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: liblrm.so.2()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libcrmcommon.so.2()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libcrmcluster.so.1()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Processing Dependency: libcib.so.1()(64bit) for package: pacemaker-1.1.7-2.fc17.x86_64
--> Running transaction check
---> Package cluster-glue.x86_64 0:1.0.6-9.fc17.1 will be installed
--> Processing Dependency: perl-TimeDate for package: cluster-glue-1.0.6-9.fc17.1.x86_64
--> Processing Dependency: libOpenIPMIutils.so.0()(64bit) for package: cluster-glue-1.0.6-9.fc17.1.x86_64
--> Processing Dependency: libOpenIPMIposix.so.0()(64bit) for package: cluster-glue-1.0.6-9.fc17.1.x86_64
--> Processing Dependency: libOpenIPMI.so.0()(64bit) for package: cluster-glue-1.0.6-9.fc17.1.x86_64
---> Package cluster-glue-libs.x86_64 0:1.0.6-9.fc17.1 will be installed
---> Package corosynclib.x86_64 0:2.0.1-1.fc17 will be installed
--> Processing Dependency: librdmacm.so.1(RDMACM_1.0)(64bit) for package: corosynclib-2.0.1-1.fc17.x86_64
--> Processing Dependency: libibverbs.so.1(IBVERBS_1.1)(64bit) for package: corosynclib-2.0.1-1.fc17.x86_64
--> Processing Dependency: libibverbs.so.1(IBVERBS_1.0)(64bit) for package: corosynclib-2.0.1-1.fc17.x86_64
--> Processing Dependency: librdmacm.so.1()(64bit) for package: corosynclib-2.0.1-1.fc17.x86_64
--> Processing Dependency: libibverbs.so.1()(64bit) for package: corosynclib-2.0.1-1.fc17.x86_64
---> Package libqb.x86_64 0:0.14.2-2.fc17 will be installed
---> Package libtool-ltdl.x86_64 0:2.4.2-3.1.fc17 will be installed
---> Package libxslt.x86_64 0:1.1.26-10.fc17 will be installed
---> Package net-snmp-libs.x86_64 1:5.7.1-4.fc17 will be installed
---> Package pacemaker-cli.x86_64 0:1.1.7-2.fc17 will be installed
---> Package pacemaker-cluster-libs.x86_64 0:1.1.7-2.fc17 will be installed
---> Package pacemaker-libs.x86_64 0:1.1.7-2.fc17 will be installed
---> Package resource-agents.x86_64 0:3.9.2-2.fc17.1 will be installed
--> Processing Dependency: /sbin/lvm for package: resource-agents-3.9.2-2.fc17.1.x86_64
--> Processing Dependency: /sbin/fsck.xfs for package: resource-agents-3.9.2-2.fc17.1.x86_64
--> Processing Dependency: libnet.so.1()(64bit) for package: resource-agents-3.9.2-2.fc17.1.x86_64
--> Running transaction check
---> Package OpenIPMI-libs.x86_64 0:2.0.18-13.fc17 will be installed
---> Package libibverbs.x86_64 0:1.1.6-2.fc17 will be installed
---> Package libnet.x86_64 0:1.1.5-3.fc17 will be installed
---> Package librdmacm.x86_64 0:1.0.15-1.fc17 will be installed
---> Package lvm2.x86_64 0:2.02.95-6.fc17 will be installed
--> Processing Dependency: lvm2-libs = 2.02.95-6.fc17 for package: lvm2-2.02.95-6.fc17.x86_64
---> Package perl-TimeDate.noarch 1:1.20-6.fc17 will be installed
---> Package xfsprogs.x86_64 0:3.1.8-1.fc17 will be installed
--> Running transaction check
---> Package lvm2-libs.x86_64 0:2.02.95-6.fc17 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
====================================================================================================
Package Arch Version Repository Size
====================================================================================================
Installing:
corosync x86_64 2.0.1-1.fc17 updates 165 k
pacemaker x86_64 1.1.7-2.fc17 fedora 362 k
Installing for dependencies:
OpenIPMI-libs x86_64 2.0.18-13.fc17 fedora 466 k
cluster-glue x86_64 1.0.6-9.fc17.1 fedora 229 k
cluster-glue-libs x86_64 1.0.6-9.fc17.1 fedora 121 k
corosynclib x86_64 2.0.1-1.fc17 updates 96 k
libibverbs x86_64 1.1.6-2.fc17 fedora 44 k
libnet x86_64 1.1.5-3.fc17 fedora 54 k
libqb x86_64 0.14.2-2.fc17 updates 77 k
librdmacm x86_64 1.0.15-1.fc17 fedora 27 k
libtool-ltdl x86_64 2.4.2-3.1.fc17 updates 45 k
libxslt x86_64 1.1.26-10.fc17 updates 419 k
lvm2 x86_64 2.02.95-6.fc17 fedora 608 k
lvm2-libs x86_64 2.02.95-6.fc17 fedora 512 k
net-snmp-libs x86_64 1:5.7.1-4.fc17 fedora 713 k
pacemaker-cli x86_64 1.1.7-2.fc17 fedora 368 k
pacemaker-cluster-libs x86_64 1.1.7-2.fc17 fedora 77 k
pacemaker-libs x86_64 1.1.7-2.fc17 fedora 322 k
perl-TimeDate noarch 1:1.20-6.fc17 fedora 43 k
resource-agents x86_64 3.9.2-2.fc17.1 fedora 466 k
xfsprogs x86_64 3.1.8-1.fc17 fedora 715 k
Transaction Summary
====================================================================================================
Install 2 Packages (+19 Dependent packages)
Total download size: 5.8 M
Installed size: 20 M
Downloading Packages:
(1/21): OpenIPMI-libs-2.0.18-13.fc17.x86_64.rpm | 466 kB 00:00
(2/21): cluster-glue-1.0.6-9.fc17.1.x86_64.rpm | 229 kB 00:00
(3/21): cluster-glue-libs-1.0.6-9.fc17.1.x86_64.rpm | 121 kB 00:00
(4/21): corosync-2.0.1-1.fc17.x86_64.rpm | 165 kB 00:00
(5/21): corosynclib-2.0.1-1.fc17.x86_64.rpm | 96 kB 00:00
(6/21): libibverbs-1.1.6-2.fc17.x86_64.rpm | 44 kB 00:00
(7/21): libnet-1.1.5-3.fc17.x86_64.rpm | 54 kB 00:00
(8/21): libqb-0.14.2-2.fc17.x86_64.rpm | 77 kB 00:00
(9/21): librdmacm-1.0.15-1.fc17.x86_64.rpm | 27 kB 00:00
(10/21): libtool-ltdl-2.4.2-3.1.fc17.x86_64.rpm | 45 kB 00:00
(11/21): libxslt-1.1.26-10.fc17.x86_64.rpm | 419 kB 00:00
(12/21): lvm2-2.02.95-6.fc17.x86_64.rpm | 608 kB 00:00
(13/21): lvm2-libs-2.02.95-6.fc17.x86_64.rpm | 512 kB 00:00
(14/21): net-snmp-libs-5.7.1-4.fc17.x86_64.rpm | 713 kB 00:00
(15/21): pacemaker-1.1.7-2.fc17.x86_64.rpm | 362 kB 00:00
(16/21): pacemaker-cli-1.1.7-2.fc17.x86_64.rpm | 368 kB 00:00
(17/21): pacemaker-cluster-libs-1.1.7-2.fc17.x86_64.rpm | 77 kB 00:00
(18/21): pacemaker-libs-1.1.7-2.fc17.x86_64.rpm | 322 kB 00:00
(19/21): perl-TimeDate-1.20-6.fc17.noarch.rpm | 43 kB 00:00
(20/21): resource-agents-3.9.2-2.fc17.1.x86_64.rpm | 466 kB 00:00
(21/21): xfsprogs-3.1.8-1.fc17.x86_64.rpm | 715 kB 00:00
----------------------------------------------------------------------------------------------------
Total 1.4 MB/s | 5.8 MB 00:04
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : libtool-ltdl-2.4.2-3.1.fc17.x86_64 1/21
Installing : cluster-glue-libs-1.0.6-9.fc17.1.x86_64 2/21
Installing : libqb-0.14.2-2.fc17.x86_64 3/21
Installing : libxslt-1.1.26-10.fc17.x86_64 4/21
Installing : libibverbs-1.1.6-2.fc17.x86_64 5/21
Installing : 1:net-snmp-libs-5.7.1-4.fc17.x86_64 6/21
Installing : librdmacm-1.0.15-1.fc17.x86_64 7/21
Installing : corosync-2.0.1-1.fc17.x86_64 8/21
error reading information on service corosync: No such file or directory
Installing : corosynclib-2.0.1-1.fc17.x86_64 9/21
Installing : pacemaker-libs-1.1.7-2.fc17.x86_64 10/21
Installing : pacemaker-cluster-libs-1.1.7-2.fc17.x86_64 11/21
Installing : pacemaker-cli-1.1.7-2.fc17.x86_64 12/21
Installing : 1:perl-TimeDate-1.20-6.fc17.noarch 13/21
Installing : lvm2-libs-2.02.95-6.fc17.x86_64 14/21
Installing : lvm2-2.02.95-6.fc17.x86_64 15/21
Installing : OpenIPMI-libs-2.0.18-13.fc17.x86_64 16/21
Installing : cluster-glue-1.0.6-9.fc17.1.x86_64 17/21
Installing : xfsprogs-3.1.8-1.fc17.x86_64 18/21
Installing : libnet-1.1.5-3.fc17.x86_64 19/21
Installing : resource-agents-3.9.2-2.fc17.1.x86_64 20/21
Installing : pacemaker-1.1.7-2.fc17.x86_64 21/21
Verifying : libqb-0.14.2-2.fc17.x86_64 1/21
Verifying : 1:net-snmp-libs-5.7.1-4.fc17.x86_64 2/21
Verifying : pacemaker-1.1.7-2.fc17.x86_64 3/21
Verifying : libtool-ltdl-2.4.2-3.1.fc17.x86_64 4/21
Verifying : cluster-glue-1.0.6-9.fc17.1.x86_64 5/21
Verifying : resource-agents-3.9.2-2.fc17.1.x86_64 6/21
Verifying : libnet-1.1.5-3.fc17.x86_64 7/21
Verifying : pacemaker-cluster-libs-1.1.7-2.fc17.x86_64 8/21
Verifying : lvm2-2.02.95-6.fc17.x86_64 9/21
Verifying : xfsprogs-3.1.8-1.fc17.x86_64 10/21
Verifying : libxslt-1.1.26-10.fc17.x86_64 11/21
Verifying : OpenIPMI-libs-2.0.18-13.fc17.x86_64 12/21
Verifying : libibverbs-1.1.6-2.fc17.x86_64 13/21
Verifying : cluster-glue-libs-1.0.6-9.fc17.1.x86_64 14/21
Verifying : lvm2-libs-2.02.95-6.fc17.x86_64 15/21
Verifying : pacemaker-libs-1.1.7-2.fc17.x86_64 16/21
Verifying : corosync-2.0.1-1.fc17.x86_64 17/21
Verifying : librdmacm-1.0.15-1.fc17.x86_64 18/21
Verifying : 1:perl-TimeDate-1.20-6.fc17.noarch 19/21
Verifying : corosynclib-2.0.1-1.fc17.x86_64 20/21
Verifying : pacemaker-cli-1.1.7-2.fc17.x86_64 21/21
Installed:
corosync.x86_64 0:2.0.1-1.fc17 pacemaker.x86_64 0:1.1.7-2.fc17
Dependency Installed:
OpenIPMI-libs.x86_64 0:2.0.18-13.fc17 cluster-glue.x86_64 0:1.0.6-9.fc17.1
cluster-glue-libs.x86_64 0:1.0.6-9.fc17.1 corosynclib.x86_64 0:2.0.1-1.fc17
libibverbs.x86_64 0:1.1.6-2.fc17 libnet.x86_64 0:1.1.5-3.fc17
libqb.x86_64 0:0.14.2-2.fc17 librdmacm.x86_64 0:1.0.15-1.fc17
libtool-ltdl.x86_64 0:2.4.2-3.1.fc17 libxslt.x86_64 0:1.1.26-10.fc17
lvm2.x86_64 0:2.02.95-6.fc17 lvm2-libs.x86_64 0:2.02.95-6.fc17
net-snmp-libs.x86_64 1:5.7.1-4.fc17 pacemaker-cli.x86_64 0:1.1.7-2.fc17
pacemaker-cluster-libs.x86_64 0:1.1.7-2.fc17 pacemaker-libs.x86_64 0:1.1.7-2.fc17
perl-TimeDate.noarch 1:1.20-6.fc17 resource-agents.x86_64 0:3.9.2-2.fc17.1
xfsprogs.x86_64 0:3.1.8-1.fc17
Complete!
Configure Corosync
More on Multicast.
Pick a Multicast address and ensure that it doesn't conflict with any other multicast group used on your subnet. Pay particular attention to the ambiguous bits and how multicast IPs overlap with one another.
For this tutorial, we will use multicast group 239.255.1.1 and port 5405
cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
vim /etc/corosync/corosync.conf
| Any questions, feedback, advice, complaints or meanderings are welcome. | |||
| Alteeve's Niche! | Enterprise Support: Alteeve Support |
Community Support | |
| © Alteeve's Niche! Inc. 1997-2024 | Anvil! "Intelligent Availability®" Platform | ||
| legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions. | |||