Building an Anvil m2

From Alteeve Wiki
Revision as of 21:07, 26 February 2013 by Digimer (talk | contribs) (→‎Network)
Jump to navigation Jump to search

 AN!Wiki :: How To :: Building an Anvil m2

Warning: This is incomplete. Please do not follow this tutorial until this warning is removed!
Note: This is an updated version of 2-Node Red Hat KVM Cluster Tutorial that adds support for the AN!CDB cluster dashboard. It also uses some tools to simplify the setup of nodes.
AN!CDB Anvil Dashboard

This tutorial will help you build an Anvil Mark-2. It differs from the previous tutorial in that it does not focus on learning the foundation of the cluster. Instead, it's focus is to help you create a reliable platform for your servers.

Think of this tutorial as a recipe for making a great cake, rather than learning the chemistry behind baking.

Before We Start

The Task Ahead

We will be building up two servers, on redundant power and networking, to create a highly-available, highly fault-tolerant platform for virtual servers.

It will be built on 100% open-source software.

Specifically, we will be using;

Hardware

Racked Anvil RN3-m2

The hardware used in this tutorial is;

You may, of course, use other hardware. However, you need to ensure a few things;

  1. Your nodes need to have out-of-band management. This is usually provided via an IPMI controller, or an OEM version, like HP's iLO, Dell's iDRAC or similar.
    1. Note: AN!CM was written to support LSI-brand RAID controllers using the MegaCLI64 tool.
    2. Note: AN!CM was tested against IPMI values returned by Fujitsu-brand servers.
      1. If you use other brands, updates to AN!CM may be required.
  2. Your switches need to support persistent multicast groups. Most simple, unmanaged switches do this. Some higher-end ones require static multicast groups be assigned.
  3. Your switched PDU needs to have a fence agent usable by the cluster. You can check here if yours does.
  4. In order to monitor the UPSes, you need to use a network-able UPS that supports apcupsd.
  5. AN!CDB needs to run on a 64-bit CPU and on a machine that properly supports RHEL6 with at least two network cards.

If you run into problems using other hardware, please contact us and we will do our best to help get the Anvil m2 working on your platform.

Prerequisites

This tutorial assumes it's readers be familiar with basic command-line Linux tools. You will need to be familiar with networking concepts and terms. It uses vim as the default editor, though readers can safely substitute their preferred text editor, provided it uses standard unix-style line wrapping. Where ever possible, commands and shortcuts will be explained in detail.

The Setup

The hardware selection and it's configuration may seem overly complex at first, but every component is there for a reason.

Power

In order to make the power feeding the Anvil fully fault tolerant, we will use two separate UPSes powering two separate switched PDUs. Where facilities exist, we recommend using two separate mains circuits two feed the separate UPSes as well.

If your nodes and devices have redundant power supplies, then you will plug one power supply into the first power rail and then plug the second power supply into the second rail. This way, the loss of power will not interrupt your Anvil at all.

If your devices have just one power supply, then you will plug the first device into the first power rail and the second device into the second power rail. In this case, a power failure will take out half your devices, but the remaining devices will continue to operate on the remaining rail. This will likely cause interruption, so it is not necessarily fault tolerant, but the high-availability nature of the Anvil should mean recovery would be quick and automated.

The need for the switched PDUs, specifically, is to provide a backup fence device. The importance of this will be covered in more detail shortly.

Network

Make our network highly available and fully fault tolerant is a little tricky and complex. Take a look at the map below;

                                                              ______________                                                         
                                                             [___Internet___]                                                        
  _____________________________________________________             |             _____________________________________________________ 
 | [ an-node01 ]                                       |            |            |                                       [ an-node02 ] |
 |                       ____________    ______________|        ____|____        |______________    ____________                       |
 |                      |    vbr2    |--| bond2        |       | [ IFN ] |       |        bond2 |--|   vbr2     |                      |
 |  _________________   | 10.255.0.1 |  | ______       |      _|_________|_      |       ______ |  | 10.255.0.2 |  ................... |
 | | [ vm0001-dev ]  |  |____________|  || eth2 =--\   |     |   Switch 1  |     |   /--= eth2 ||  |____________|  :  [ vm0001-dev ] : |
 | | [ Dev Server ]  |    | | : :       ||_____|    \--=-----|_____________|-----=--/    |_____||       | | : :    :  [ Dev Server ] : |
 | |           ______|    | | : :       | ______    /--=-----|   Switch 2  |-----=--\    ______ |       | | : :    :.......          : |
 | |          | eth0 =----/ | : :       || eth5 =--/   |     |_____________|     |   \--= eth5 ||       | | : :----= eth0 :          : |
 | |          |_____||      | : :       ||_____|       |                         |       |_____||       | | :      ::.....:          : |
 | |      10.254.0.1 |      | : :       |______________|                         |______________|       | | :      :                 : |
 | |_________________|      | : :        ______________|                         |______________        | | :      :.................: |
 |                          | : :       | bond1        |        _________        |        bond1 |       | | :                          |
 |  _________________       | : :       |   10.10.0.1  |       | [ SN  ] |       | 10.10.0.2    |       | | :      ................... |
 | | [ vm0002-web ]  |      | : :       | ______       |      _|_________|_      |       ______ |       | | :      :  [ vm0002-web ] : |
 | | [ Web Server ]  |      | : :       || eth1 =--\   |     |   Switch 1  |     |   /--= eth1 ||       | | :      :  [ Web Server ] : |
 | |           ______|      | : :       ||_____|    \--=-----|_____________|-----=--/    |_____||       | | :      :.......          : |
 | |          | eth0 =------/ : :       | ______    /--=-----|   Switch 2  |-----=--\    ______ |       | | :------= eth0 :          : |
 | |          |_____||        : :       || eth4 =--/   |     |_____________|     |   \--= eth4 ||       | |        ::.....:          : |
 | |      10.254.0.2 |        : :       ||_____|       |                         |       |_____||       | |        :                 : |
 | |_________________|        : :       |______________|                         |______________|       | |        :.................: |
 |                            : :        ______________|                         |______________        | |                            |
 | ...................        : :       | bond0        |        _________        |        bond0 |       | |         _________________  |
 | : [ vm0003-db  ]  :        : :       |   10.20.0.1  |       | [ BCN ] |       | 10.20.0.2    |       | |        |  [ vm0003-db  ] | |
 | : [ DB Server  ]  :        : :       | ______       |      _|_________|_      |       ______ |       | |        |  [ DB Server  ] | |
 | :          .......:        : :       || eth0 =--\   |  /--|   Switch 1  |--\  |   /--= eth0 ||       | |        |______           | |
 | :          : eth0 =--------: :       ||_____|    \--=--+--|_____________|--+--=--/    |_____||       | \--------= eth0 |          | |
 | :          :.....::          :       | ______    /--=--+--|   Switch 2  |--+--=--\    ______ |       |          ||_____|          | |
 | :                 :          :       || eth3 =--/   |  |  |_____________|  |  |   \--= eth3 ||       |          | 10.254.0.3      | |
 | :.................:          :       ||_____|       |  |     |       |     |  |       |_____||       |          |_________________| |
 |                              :       |______________|  |     |       |     |  |______________|       |                              |
 | ...................          :                      |  |     |       |     |  |                      |           _________________  |
 | : [ vm0004-win ]  :          :                      |  |     |       |     |  |                      |          |  [ vm0004-win ] | |
 | : [ MS Server  ]  :          :                      |  |     |       |     |  |                      |          |  [ MS Server  ] | |
 | :          .......:          :                      |  |     |       |     |  |                      |          |______           | |
 | :          : NIC0 =----------:                      |  |     |       |     |  |                      \----------= NIC0 |          | |
 | :          :.....::                           ______|  |     |       |     |  |______                           ||_____|          | |
 | :                 :                  _____   | IPMI =--/     |       |     \--= IPMI |   _____                  | 10.254.0.4      | |
 | :.................:                 [_BMC_]--|_____||        |       |        ||_____|--[_BMC_]                 |_________________| |
 |                                                     |        |       |        |                                                     |
 |                                 ______ ______       |        |       |        |       ______ ______                                 |
 |                                | PSU1 | PSU2 |      |        |       |        |      | PSU2 | PSU1 |                                |
 |________________________________|______|______|______|        |       |        |______|______|______|________________________________|
                                       || ||                ____|_     _|____                || ||                                      
                                       || ||               | PDU1 |   | PDU2 |               || ||                                      
                                       || ||               |______|   |______|               || ||                                      
                                       || ||                 || ||     || ||                 || ||                                      
                                       || \\===[ Power 1 ]===// ||     || \\===[ Power 1 ]===// ||                                      
                                       \\======[ Power 2 ]======||=====//                       ||                                      
                                                                \\=============[ Power 2 ]======//

The goal is to look at any one network component above and ask ourselves, "What happens if this fails? In what ways might this fail?".

Avoiding Network Congestion

The first challenge is to deal with congestion.

Broadly speaking, the Anvil! has three major types of traffic;

  1. Cluster-Specific Traffic
  2. Storage Replication Traffic
  3. Client-Facing Server Traffic

Each one of these traffic groups can fairly easily saturate it's network connection. For this reason, we will create three totally separate networks;

  1. Back-Channel Network - This will be used for two primary purposes;
    1. Inter-node communication, which requires low latency
    2. Live-migration of servers between nodes, which requires high-bandwidth to copy the server's RAM between nodes.
  2. Storage Network - Every time a file is saved in a server, it will be copied to the peer node over this network.
  3. Internet-Facing Network - All traffic between the servers on the Anvil and users on your network will travel over this network.

With this setup, any one network can reach full speed without interfering with or harming the performance of the other two networks. This ensures that your user's experience is never degraded, even under heavy loads.

Making the Network Highly Available

We will be using a concept called bonding network cards with two switches to make each network highly available.

.--------------------.                            .---------------------.
:  Node 01           :                            |            Node 02  :
:  __________________:                            |___________________  :
: |  Bonded Network  |     __________________     |   Bonded Network  | :
: |          ________|    |     Primary      |    |________           | :
: |         | link 1 =xxxx=  Network Switch  =xxxx= link 1 |          | :
: |         |_______||    |__________________|    |________|          | :
: |                  |     ________||________     |                   | :
: |          ________|    |    Secondary     |    |________           | :
: |         | link 2 =----=  Network Switch  =----= link 2 |          | : 
: |         |_______||    |__________________|    ||_______|          | :
: |__________________|                            |___________________| :
:                    :                            :                     :
----------------------                            -----------------------

The way this works is that each network connection on each node uses two network ports. The first one plus into the first switch and the second one plugs into the second switch. The switches, in turn, and linked together as well.

.--------------------.                            .---------------------.
:  Node 01           :                            |            Node 02  :
:  __________________:                            |___________________  :
: |  Bonded Network  |     __________________     |   Bonded Network  | :
: |          ________|    |     Primary      |    |________           | :
: |         | link 1 =X  X=  Network Switch  =xxxx= link 1 |          | :
: |         |_______||    |__________________|    |________|          | :
: |                  |     ________XX________     |                   | :
: |          ________|    |    Secondary     |    |________           | :
: |         | link 2 =xxxx=  Network Switch  =----= link 2 |          | : 
: |         |_______||    |__________________|    ||_______|          | :
: |__________________|                            |___________________| :
:                    :                            :                     :
----------------------                            -----------------------

If a network cable comes lose or fails, the node will nearly instantly fail over to the second cable. This happens extremely fast and programs pushing traffic over the network will not notice any interruption. Later, when the cable is replaced or plugged back in. the node will switch back to use the original link.

.--------------------.                            .---------------------.
:  Node 01           :                            |            Node 02  :
:  __________________:                            |___________________  :
: |  Bonded Network  |     __________________     |   Bonded Network  | :
: |          ________|    X     Primary      X    |________           | :
: |         | link 1 =    =  Network Switch  =    = link 1 |          | :
: |         |_______||    X__________________X    |________|          | :
: |                  |     ________||________     |                   | :
: |          ________|    |    Secondary     |    |________           | :
: |         | link 2 =xxxx=  Network Switch  =xxxx= link 2 |          | : 
: |         |_______||    |__________________|    ||_______|          | :
: |__________________|                            |___________________| :
:                    :                            :                     :
----------------------                            -----------------------

If an entire switch should fail, say it lost it's power as an example, then all three networks on both nodes will instantly fail over to their backup connections that plug into the backup switch. As before, this happens so fast the programs using the network will not notice an interruption.

Providing Network Security

The Anvil's "Foundation Pack", the switches, PDUs, UPSes and IPMI interfaces, provides mechanisms for powering off the nodes. This is a security issue, so we will create VLANs in to isolate the three networks from one another. We will also use three separate subnets.

These are;

Network Subnet VLAN ID
BCN - Back-Channel Network 10.20.0.0/16 1
SN - Storage Network 10.10.0.0/16 100
IFN - Internet-Facing Network 10.255.0.0/16 101

In this tutorial, we will use two separate switched, stacked to act like a single switch. We will then create the three separate VLANs to block traffic from one network from being able to reach the ports on the other networks.

If you do not have two stacked switches, you can use unstacked switched just fine. However, you will need to create a link between each switch. If you unstacked switches support VLANs, then you will need to create your three VLANs and then create three links between each switch, one for each VLAN.

If you are using unmanaged switches, you will not be able to create VLANs at all. In this case, you can leave the VLANs out of the configuration entirely. Be aware though! A malicious user will be able to assign themselves an IP address on the back-channel network and then reach out and talk to your nodes and foundation pack devices. If possible, using six small, unmanaged switches would be safest.

 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche! Enterprise Support:
Alteeve Support
Community Support
© Alteeve's Niche! Inc. 1997-2024   Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.