Anvil! Tutorial 3

AN!Wiki :: How To :: Anvil! Tutorial 3

Warning: This tutorial is incomplete, flawed and generally sucks at this time. Do not follow this and expect anything to work. In large part, it's a dumping ground for notes and little else. This warning will be removed when the tutorial is completed.

Warning: This tutorial is built on a guess of what Red Hat's Enterprise Linux 7 will offer, based on what the author sees happening in Fedora upstream. Red Hat never confirms what a future release will contain until it is actually released. As such, this tutorial may turn out to be inappropriate for the final release of RHEL 7. In such a case, the warning above will remain in place until the tutorial is updated to reflect the final release.

This is the third AN!Cluster tutorial built on Red Hat's Enterprise Linux 7. It improves on the RHEL 5, RHCS stable 2 and RHEL 6, RHCS stable3 tutorials.

As with the previous tutorials, the end goal of this tutorial is a 2-node cluster providing a platform for high-availability virtual servers. It's design attempts to remove all single points of failure from the system. Power and networking are made fully redundant in this version, along with minimizing the node failures which would lead to service interruption. This tutorial also covers the AN!Utilities; AN!Cluster Dashboard, AN!Cluster Monitor and AN!Safe Cluster Shutdown.

As it the previous tutorial, KVM will be the hypervisor used for facilitating virtual machines. The old cman and rgmanager tools are replaced in favour of pacemaker for resource management.

Before We Begin

This tutorial does not require prior cluster experience, but it does expect familiarity with Linux and a low-intermediate understanding of networking. Where possible, steps are explained in detail and rationale is provided for why certain decisions are made.

For those with cluster experience;

Please be careful not to skip too much. There are some major and some subtle changes from previous tutorials.

OS Setup

Warning: I used Fedora 18 at this point, obviously things will change, possibly a lot, once RHEL 7 is released.

Install

Not all of these are required, but most are used at one point or another in this tutorial.

yum install bridge-utils gpm man net-tools network ntp rsync syslinux vim wget

If you want to use your mouse at the node's terminal, run the following;

systemctl enable gpm.service
systemctl start gpm.service

Make the Network Configuration Static

We don't want NetworkManager in our cluster as it tries to dynamically manage the network and we need our network to be static.

yum remove NetworkManager

Note: This assumes that systemd will be used in RHEL7. This may not be the case come release day.

Now to ensure the static network service starts on boot.

systemctl enable network.service

Setting the Hostname

Fedora 18 is very different from EL6.

Note: The '--pretty' line currently doesn't work as there is a bug (rhbz#895299) with single-quotes.

Note: The '--static' option is currently needed to prevent the '.' from being removed. See this bug (rhbz#896756).

Use a format that works for you. For the tutorial, node names are based on the following;

• A two-letter prefix identifying the company/user (an, for "Alteeve's Niche!")
• A sequential cluster ID number in the form of cXX (c01 for "Cluster 01", c02 for Cluster 02, etc)
• A sequential node ID number in the form of nYY

In my case, this is my third cluster and I use the company prefix an, so my two nodes will be;

• an-c03n01 - node 1
• an-c03n02 - node 2

Folks who've read my earlier tutorials will note that this is a departure in naming. I find this method spans and scales much better. Further, it the simply required in order to use the AN! Cluster Dashboard.

hostnamectl set-hostname an-c03n01.alteeve.ca --static
hostnamectl set-hostname --pretty "Alteeve's Niche! - Cluster 01, Node 01"

If you want the new host name to take effect immediately, you can use the traditional hostname command:

hostname an-c03n01.alteeve.ca

Alternatively

If you have trouble with those commands, you can directly edit the files that contain the host names.

The host name is stored in /etc/hostname:

echo an-c03n01.alteeve.ca > /etc/hostname 
cat /etc/hostname

an-c03n01.alteeve.ca

The "pretty" host name is stored in /etc/machine-info as the unquoted value for the PRETTY_HOSTNAME value.

vim /etc/machine-info

PRETTY_HOSTNAME=Alteeves Niche! - Cluster 01, Node 01

If you can't get the hostname command to work for some reason, you can reboot to have the system read the new values.

Optional - Video Problems

On my servers, Fedora 18 doesn't detect or use the video card properly. To resolve this, I need to add nomodeset to the kernel line when installing and again after the install is complete.

Once installed

Edit the /etc/default/grub and append nomodeset to the end of the GRUB_CMDLINE_LINUX variable.

vim /etc/default/grub

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_CMDLINE_LINUX="nomodeset rd.md=0 rd.lvm=0 rd.dm=0 $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rd.luks=0 vconsole.keymap=us nomodeset"
GRUB_DISABLE_RECOVERY="true"
GRUB_THEME="/boot/grub2/themes/system/theme.txt"

Save that. and then rewrite the grub2 configuration file.

grub2-mkconfig -o /boot/grub2/grub.cfg

Next time you reboot, you should get a stock 80x25 character display. It's not much, but it will work on esoteric video cards or weird monitors.

What Security?

This section will be re-added at the end. For now;

setenforce 0
sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
systemctl disable firewalld.service
systemctl stop firewalld.service

Network

We want static, named network devices. Follow this;

• Changing Ethernet Device Names in EL7 and Fedora 15+

Then, use these configuration files;

Build the bridge;

vim /etc/sysconfig/network-scripts/ifcfg-ifn-vbr1

# Internet-Facing Network - Bridge
DEVICE="ifn-vbr1"
TYPE="Bridge"
BOOTPROTO="none"
IPADDR="10.255.10.1"
NETMASK="255.255.0.0"
GATEWAY="10.255.255.254"
DNS1="8.8.8.8"
DNS2="8.8.4.4"
DEFROUTE="yes"

Now build the bonds;

vim /etc/sysconfig/network-scripts/ifcfg-ifn-bond1

# Internet-Facing Network - Bond
DEVICE="ifn-bond1"
BRIDGE="ifn-vbr1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=ifn1"

vim /etc/sysconfig/network-scripts/ifcfg-sn-bond1

# Storage Network - Bond
DEVICE="sn-bond1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=sn1"
IPADDR="10.10.10.1"
NETMASK="255.255.0.0"

vim /etc/sysconfig/network-scripts/ifcfg-bcn-bond1

# Back-Channel Network - Bond
DEVICE="bcn-bond1"
BOOTPROTO="none"
NM_CONTROLLED="no"
ONBOOT="yes"
BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=bcn1"
IPADDR="10.20.10.1"
NETMASK="255.255.0.0"

Now tell the interfaces to be slaves to their bonds;

Internet-Facing Network;

vim /etc/sysconfig/network-scripts/ifcfg-ifn1

# Internet-Facing Network - Link 1
DEVICE="ifn1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="ifn-bond1"

vim /etc/sysconfig/network-scripts/ifcfg-ifn2

# Back-Channel Network - Link 2
DEVICE="ifn2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="ifn-bond1"

Storage Network;

vim /etc/sysconfig/network-scripts/ifcfg-sn1

# Storage Network - Link 1
DEVICE="sn1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="sn-bond1"

vim /etc/sysconfig/network-scripts/ifcfg-sn2

# Storage Network - Link 1
DEVICE="sn2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="sn-bond1"

Back-Channel Network

vim /etc/sysconfig/network-scripts/ifcfg-bcn1

# Back-Channel Network - Link 1
DEVICE="bcn1"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="bcn-bond1"

vim /etc/sysconfig/network-scripts/ifcfg-bcn2

# Storage Network - Link 1
DEVICE="bcn2"
NM_CONTROLLED="no"
BOOTPROTO="none"
ONBOOT="yes"
SLAVE="yes"
MASTER="bcn-bond1"

Now restart the network, confirm that the bonds and bridge are up and you are ready to proceed.

Setup The hosts File

You can use DNS if you prefer. For now, lets use /etc/hosts for node name resolution.

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

# AN!Cluster 01, Node 01
10.255.10.1     an-c01n01.ifn
10.10.10.1      an-c01n01.sn
10.20.10.1      an-c01n01.bcn an-c01n01 an-c01n01.alteeve.ca
10.20.11.1      an-c01n01.ipmi

# AN!Cluster 01, Node 02
10.255.10.2     an-c01n02.ifn
10.10.10.2      an-c01n02.sn
10.20.10.2      an-c01n02.bcn an-c01n02 an-c01n02.alteeve.ca
10.20.11.2      an-c01n02.ipmi

# Foundation Pack
10.20.2.7       an-p03 an-p03.alteeve.ca

Setup SSH

Same as before.

Populating And Pushing ~/.ssh/known_hosts

Same as before.

ssh root@an-c03n01.alteeve.ca

The authenticity of host 'an-c03n01.alteeve.ca (10.20.30.1)' can't be established.
RSA key fingerprint is 7b:dd:0d:aa:c5:f5:9e:a6:b6:4d:40:69:d6:80:4d:09.
Are you sure you want to continue connecting (yes/no)?

Type yes

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'an-c03n01.alteeve.ca,10.20.30.1' (RSA) to the list of known hosts.
Last login: Thu Feb 14 15:18:33 2013 from 10.20.5.100

You will now be logged into the an-c03n01 node, which in this case is the same machine on a new session in the same terminal.

[root@an-c03n01 ~]#

You can logout by typing exit.

exit

logout
Connection to an-c03n01.alteeve.ca closed.

Now we have to repeat the steps for all the other variations on the names of the hosts. This is annoying and tedious, sorry.

ssh root@an-c03n01
ssh root@an-c03n01.bcn
ssh root@an-c03n01.sn
ssh root@an-c03n01.ifn
ssh root@an-c03n02.alteeve.ca
ssh root@an-c03n02
ssh root@an-c03n02.bcn
ssh root@an-c03n02.sn
ssh root@an-c03n02.ifn

Keeping Time in Sync

It's not as critical as it used to be to keep the clocks on the nodes in sync, but it's still a good idea.

systemctl start ntpd.service
systemctl enable ntpd.service

Installing Programs

From this point forward, this tutorial borrows heavily from Pacemaker 1.1, Clusters from Scratch.

yum install pacemaker corosync pcs

Configure Corosync

More on Multicast.

Pick a Multicast address and ensure that it doesn't conflict with any other multicast group used on your subnet. Pay particular attention to the ambiguous bits and how multicast IPs overlap with one another.

For this tutorial, we will use multicast group 239.255.1.1 and port 5405

rpm -Uvh http://people.redhat.com/cfeist/pcs/pcs-0.9.26-1.fc18.noarch.rpm

 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche!	Enterprise Support: Alteeve Support	Community Support
© Alteeve's Niche! Inc. 1997-2024		Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.