2x5 Scalable Cluster Tutorial
Overview
This paper has one goal;
- Creating a 2-node, high-availability cluster hosting Xen virtual machines.
Technologies We Will Use
We will introduce and use the following technologies:
- RHCS, Red Hat Cluster Services version 3, aka "Cluster 3", running on Red Hat Enterprise Linux 6.0 x86_64.
- Fencing devices needed to keep a cluster safe.
- Two fencing types are discussed;
- IPMI; The most common fence method used in servers.
- Node Assassin; A home-brew fence device ideal for learning or as a backup to IPMI.
- Two fencing types are discussed;
- Xen; The virtual server hypervisor.
- Putting all cluster-related daemons under the control of rgmanager.
- Making the VMs highly available.
Prerequisites
It is expected that you are already comfortable with the Linux command line, specifically bash, and that you are familiar with general administrative tasks in Red Hat based distributions. You will also need to be comfortable using editors like vim, nano, gedit, kate or similar. This paper uses vim in examples. Simply substitute your favourite editor in it's place.
You are also expected to be comfortable with networking concepts. You will be expected to understand TCP/IP, multicast, broadcast, subnets and netmasks, routing and other relatively basic networking concepts. Please take the time to become familiar with these concepts before proceeding.
Where feasible, as much detail as is possible will be provided. For example, all configuration file locations will be shown and functioning sample files will be provided.
Platform
Red Hat Cluster Service version 3, also known as "Cluster Stable 3" or "RHCS 3", entered the server distribution world with the release of RHEL 6. It is used by downstream distributions like CentOS and Scientific Linux. This tutorial should be easily adapted to any Red Hat derivative distribution. It is expected that most users will have 64-bit CPUs, thus, we will use the x86_64 distribution and packages.
If you are on an 32-bit system, you should be able to follow along fine. Simply replace x86_64 with i386 or i686 in package names. Be aware though that issues arising from the need for PAE will not be discussed.
If you do not have a Red Hat Network account, you can download CentOS or another derivative of the same release, currently 6.0.
Note: When last checked, down-stream distributions have not yet been released. It is expected that they will be available around mid to late December.
Focus and Goal
Clusters can serve to solve three problems; Reliability, Performance and Scalability.
This paper will build a cluster designed to be more reliable, also known as a High-Availability cluster or simply HA Cluster. At the end of this paper, you should have a fully functioning two-node cluster capable of hosting a "floating" virtual servers. That is, VMs that exist on one node that can be easily moved to the other node with minimal or no down time.
Begin
This paper is admittedly long-winded. There is a "cheat-sheet" version planned, but it will be written only after this main tutorial is complete. Please be patient! Clustering is not inherently difficult, but there are a lot of pieces that need to work together for anything to work. Grab a coffee or tea and settle in.
Hardware
We will need two physical servers each with the following hardware:
- One or more multi-core CPUs with Virtualization support.
- Three network cards; At least one should be gigabit or faster.
- One or more hard drives.
- You will need some form of a fence device. This can be an IPMI-enabled server, a Node Assassin, a fenceable PDU or similar.
This paper uses the following hardware, and would suggest these be "minimum specifications":
- ASUS M4A78L-M
- AMD Athlon II x2 250
- 2GB Kingston DDR2 KVR800D2N6K2/4G (4GB kit split between the two nodes)
- 1x Intel 82540 PCI NICs
- 1x D-Link DGE-560T
- Node Assassin
This is not an endorsement of the above hardware. I bought what was within my budget that would serve the purposes of creating this document. What you purchase shouldn't matter, so long as the minimum requirements are met.
Note: I use three physical NICs, but you can get away with fewer by using VLANs or by simply re-using a given interface. Neither appealed to me given the minimal cost of add-in network cards and the relative complexity of VLANs. If you wish to alter your network setup, please do so.
Pre-Assembly Information
With multiple NICs, it is quite likely that the mapping of physical devices to logical ethX devices may not be ideal. This is a particular issue if you decide to network boot your install media.
There is no requirement, from a clustering point of view, that any given network card be mapped to any given ethX device. However, you will be jumping between servers fairly often and having various setups adds one more level of complexity. For this reason, I strongly recommend you follow this section.
Before you assemble your servers, record their network cards' MAC addresses. I like to keep simple text files like these:
cat an-node01.mac
90:E6:BA:71:82:EA eth0 # Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
00:21:91:19:96:53 eth1 # D-Link System Inc DGE-560T PCI Express Gigabit Ethernet Adapter
00:0E:0C:59:46:E4 eth2 # Intel Corporation 82540EM Gigabit Ethernet Controller
cat an-node02.mac
90:E6:BA:71:82:D8 eth0 # Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
00:21:91:19:96:5A eth1 # D-Link System Inc DGE-560T PCI Express Gigabit Ethernet Adapter
00:0E:0C:59:45:78 eth2 # Intel Corporation 82540EM Gigabit Ethernet Controller
This will prove very handy later.
OS Install
There is no hard and fast rule on how you install the host operating systems. Ultimately, it's a question of what you prefer. There are some things that you should keep in mind though.
- Balance the desire for tools against the reality that all programs have bugs.
- Bugs could be exploited to gain access to your server. If the host is compromised, all of the virtual servers are compromised.
- The host operating system, known as dom0 in Xen, should do nothing but run the hypervisor.
- If you install a graphical interface, like Xorg and Gnome, consider disabling it.
- This paper takes this approach and will cover disabling the graphical interface.
Below is the kickstart script used by the nodes for this paper. You should be able to adapt it easily to suit your needs. All options are documented.
Any questions, feedback, advice, complaints or meanderings are welcome. | |||
Alteeve's Niche! | Enterprise Support: Alteeve Support |
Community Support | |
© Alteeve's Niche! Inc. 1997-2024 | Anvil! "Intelligent Availability®" Platform | ||
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions. |