2x5 Scalable Cluster Tutorial

AN!Wiki :: How To :: 2x5 Scalable Cluster Tutorial

Overview

This paper has one goal;

• Creating a 2-node, high-availability cluster hosting Xen virtual machines.

Technologies We Will Use

We will introduce and use the following technologies:

• RHCS, Red Hat Cluster Services version 3, aka "Cluster 3", running on Red Hat Enterprise Linux 6.0 x86_64.
  • RHCS implements:
    • cman; The cluster manager.
    • corosync; The cluster engine, implementing the totem protocol and cpg and other core cluster services.
    • rgmanager; The resource group manager handles restoring and failing over services in the cluster, including our Xen VMs.
• Fencing devices needed to keep a cluster safe.
  • Two fencing types are discussed;
    • IPMI; The most common fence method used in servers.
    • Node Assassin; A home-brew fence device ideal for learning or as a backup to IPMI.
• Xen; The virtual server hypervisor.
  • Converting the host OS into the special access dom0 virtual machine.
  • Provisioning domU VMs.
• Putting all cluster-related daemons under the control of rgmanager.
  • Making the VMs highly available.

Prerequisites

It is expected that you are already comfortable with the Linux command line, specifically bash, and that you are familiar with general administrative tasks in Red Hat based distributions. You will also need to be comfortable using editors like vim, nano, gedit, kate or similar. This paper uses vim in examples. Simply substitute your favourite editor in it's place.

You are also expected to be comfortable with networking concepts. You will be expected to understand TCP/IP, multicast, broadcast, subnets and netmasks, routing and other relatively basic networking concepts. Please take the time to become familiar with these concepts before proceeding.

Where feasible, as much detail as is possible will be provided. For example, all configuration file locations will be shown and functioning sample files will be provided.

Platform

Red Hat Cluster Service version 3, also known as "Cluster Stable 3" or "RHCS 3", entered the server distribution world with the release of RHEL 6. It is used by downstream distributions like CentOS and Scientific Linux. This tutorial should be easily adapted to any Red Hat derivative distribution. It is expected that most users will have 64-bit CPUs, thus, we will use the x86_64 distribution and packages.

If you are on an 32-bit system, you should be able to follow along fine. Simply replace x86_64 with i386 or i686 in package names. Be aware though that issues arising from the need for PAE will not be discussed.

If you do not have a Red Hat Network account, you can download CentOS or another derivative of the same release, currently 6.0.

Note: When last checked, down-stream distributions have not yet been released. It is expected that they will be available around mid to late December.

Focus and Goal

Clusters can serve to solve three problems; Reliability, Performance and Scalability.

This paper will build a cluster designed to be more reliable, also known as a High-Availability cluster or simply HA Cluster. At the end of this paper, you should have a fully functioning two-node cluster capable of hosting a "floating" virtual servers. That is, VMs that exist on one node that can be easily moved to the other node with minimal or no down time.

Begin

This paper is admittedly long-winded. There is a "cheat-sheet" version planned, but it will be written only after this main tutorial is complete. Please be patient! Clustering is not inherently difficult, but there are a lot of pieces that need to work together for anything to work. Grab a coffee or tea and settle in.

Hardware

We will need two physical servers each with the following hardware:

• One or more multi-core CPUs with Virtualization support.
• Three network cards; At least one should be gigabit or faster.
• One or more hard drives.
• You will need some form of a fence device. This can be an IPMI-enabled server, a Node Assassin, a fenceable PDU or similar.

This paper uses the following hardware, and would suggest these be "minimum specifications":

• ASUS M4A78L-M
• AMD Athlon II x2 250
• 2GB Kingston DDR2 KVR800D2N6K2/4G (4GB kit split between the two nodes)
• 1x Intel 82540 PCI NICs
• 1x D-Link DGE-560T
• Node Assassin

This is not an endorsement of the above hardware. I bought what was within my budget that would serve the purposes of creating this document. What you purchase shouldn't matter, so long as the minimum requirements are met.

Note: I use three physical NICs, but you can get away with fewer by using VLANs or by simply re-using a given interface. Neither appealed to me given the minimal cost of add-in network cards and the relative complexity of VLANs. If you wish to alter your network setup, please do so.

Pre-Assembly Information

With multiple NICs, it is quite likely that the mapping of physical devices to logical ethX devices may not be ideal. This is a particular issue if you decide to network boot your install media.

There is no requirement, from a clustering point of view, that any given network card be mapped to any given ethX device. However, you will be jumping between servers fairly often and having various setups adds one more level of complexity. For this reason, I strongly recommend you follow this section.

Before you assemble your servers, record their network cards' MAC addresses. I like to keep simple text files like these:

cat an-node01.mac

90:E6:BA:71:82:EA	eth0	# Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
00:21:91:19:96:53	eth1	# D-Link System Inc DGE-560T PCI Express Gigabit Ethernet Adapter
00:0E:0C:59:46:E4	eth2	# Intel Corporation 82540EM Gigabit Ethernet Controller

cat an-node02.mac

90:E6:BA:71:82:D8	eth0	# Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
00:21:91:19:96:5A	eth1	# D-Link System Inc DGE-560T PCI Express Gigabit Ethernet Adapter
00:0E:0C:59:45:78	eth2	# Intel Corporation 82540EM Gigabit Ethernet Controller

This will prove very handy later.

OS Install

There is no hard and fast rule on how you install the host operating systems. Ultimately, it's a question of what you prefer. There are some things that you should keep in mind though.

• Balance the desire for tools against the reality that all programs have bugs.
  • Bugs could be exploited to gain access to your server. If the host is compromised, all of the virtual servers are compromised.
• The host operating system, known as dom0 in Xen, should do nothing but run the hypervisor.
• If you install a graphical interface, like Xorg and Gnome, consider disabling it.
  • This paper takes this approach and will cover disabling the graphical interface.

Below is the kickstart script used by the nodes for this paper. You should be able to adapt it easily to suit your needs. All options are documented.

• generic_server_rhel6.ks

 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche!	Enterprise Support: Alteeve Support	Community Support
© Alteeve's Niche! Inc. 1997-2024		Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.