2x5 Scalable Cluster Tutorial: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
|||
| Line 52: | Line 52: | ||
| | [ iqn.2011-08.com.alteeve:an-clusterA.target02.sdd ] | | | | | [ iqn.2011-08.com.alteeve:an-clusterA.target02.sdd ] | | | ||
_________________ | | [ drbd0 == hdd == vg01 Floating IP ] | | ___________________ | _________________ | | [ drbd0 == hdd == vg01 Floating IP ] | | ___________________ | ||
[ Internet Facing ] | | [____drbd1_==_sdd_== | [ Internet Facing ] | | [____drbd1_==_sdd_==_vg02__________192.168.2.100____] | | /---[ Internal Managed ] | ||
[_____Routers_____] | | | | | | | [ Private Network ] | [_____Routers_____] | | | | | | | [ Private Network ] | ||
| | \-----------\ | | /-----/ | | [_and_fence_devices_] | | | \-----------\ | | /-----/ | | [_and_fence_devices_] | ||
Revision as of 20:20, 1 August 2011
 |
Warning: This tutorial is not even close to complete or accurate. It will be updated later, but so long as this warning is here, consider it defective and unusable. The only up to date clustering tutorial is: Red Hat Cluster Service 2 Tutorial. |
The Design
All nodes have IPs as follows:
* eth0 == Internet Facing Network == 192.168.1.x
* eth1 == Storage Network == 192.168.2.x
* eth2 == Back Channel Network == 192.168.3.x
* Where 'x' = the node ID (ie: an-node01 -> x=1)
* If a node has an IPMI (or similar) interface piggy-backed on a network
interface, it will be shared with eth2. If it has a dedicated interface, it
will be connected to the BCN.
* Node management interfaces will be on 192.168.3.(x+100)
* All subnets are /24 (255.255.255.0)
Storage node use 2x SATA drives ('sda' and 'sdb') plus 2x SSD drives ('sdc'
and 'sdd').
Logical map:
___________________________________________ ___________________________________________
| [ an-node01 ] ______| |______ [ an-node02 ] |
| ______ _____ _______ | eth0 =------\ /------= eth0 | _______ _____ ______ |
| [_sda1_]--[_md0_]--[_/boot_] |_____|| | | ||_____| [_/boot_]--[_md0_]--[_sda1_] |
| [_sdb1_] | | | | [_sdb1_] |
| ______ _____ ______ ______| | | |______ ______ _____ ______ |
| [_sda2_]--[_md1_]--[_swap_] /----| eth1 =----\ | | /----= eth1 |----\ [_swap_]--[_md1_]--[_sda2_] |
| [_sdb2_] | /--|_____|| | | | | ||_____|--\ | [_sdb2_] |
| ______ _____ ___ | | | | | | | | | | ___ _____ ______ |
| [_sda3_]--[_md2_]--[_/_] | | ______| | | | | |______ | | [_/_]--[_md2_]--[_sda3_] |
| [_sdb3_] | | | eth2 =--\ | | | | /--= eth2 | | | [_sdb3_] |
| ______ _____ _______ | | |_____|| | | | | | | ||_____| | | _______ _____ ______ |
| [_sda5_]--[_md3_]--[_drbd0_]--/ | | | | | | | | | | \--[_drbd0_]--[_md3_]--[_sda5_] |
| [_sdb5_] | | | | | | | | | | [_sdb5_] |
| ______ _____ _______ | | | | | | | | | | _______ _____ ______ |
| [_sdc1_]--[_md4_]--[_drbd1_]----/ | | | | | | | | \----[_drbd1_]--[_md4_]--[_sdc1_] |
| [_sdd1_] | | | | | | | | [_sdd1_] |
|___________________________________________| | | | | | | |___________________________________________|
| | | | | |
/---------------------------/ | | |
| | | | | |
| /-------------------------------/ | |
| | | | | \-----------------------\
| | | | | |
| | \-----------------------------------\ |
| | | | | |
| | ____________________|_________|____________________ | |
| | [ iqn.2011-08.com.alteeve:an-clusterA.target01.hdd ] | |
| | [ iqn.2011-08.com.alteeve:an-clusterA.target02.sdd ] | |
_________________ | | [ drbd0 == hdd == vg01 Floating IP ] | | ___________________
[ Internet Facing ] | | [____drbd1_==_sdd_==_vg02__________192.168.2.100____] | | /---[ Internal Managed ]
[_____Routers_____] | | | | | | | [ Private Network ]
| | \-----------\ | | /-----/ | | [_and_fence_devices_]
| \-----------\ | | | | /-----/ |
\---------------\ | | | | | | /-----/
_|_|_|___________ _|_|_____________ _|_|_|___________
[ Storage Cluster ] [ Internet Facing ] [ Storage Network ] [ Back-Channel ]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~[_____Network_____]~~~[_________________]~~~[_____Network_____]~~~~~~~~~~~~~~~
[ VM Cluster ] | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | |
__________________________ | | | | | | | | | | | | | | |
| [ an-node03 ] ______| | | | | | | | | | | | | | | |
| | eth0 =-----/ | | | | | | | | | | | | | |
| |_____|| | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | |
| ______| | | | | | | | | | | | | | |
| | eth1 =---------------------------/ | | | | | | | | |
| |_____|| | | | | | | | | | | | | |
| | | | | | | | | | | | | | |
| ______| | | | | | | | | | | | | |
| | eth2 =-------------------------------------------------/ | | | |
| |_____|| | | | | | | | | | | | |
|__________________________| | | | | | | | | | | | |
| | | | | | | | | | | |
__________________________ | | | | | | | | | | | |
| [ an-node04 ] ______| | | | | | | | | | | | |
| | eth0 =-------/ | | | | | | | | | | |
| |_____|| | | | | | | | | | | |
| | | | | | | | | | | | |
| ______| | | | | | | | | | | |
| | eth1 =-----------------------------/ | | | | | | |
| |_____|| | | | | | | | | | |
| | | | | | | | | | | |
| ______| | | | | | | | | | |
| | eth2 =---------------------------------------------------/ | | |
| |_____|| | | | | | | | | |
|__________________________| | | | | | | | | |
| | | | | | | | |
__________________________ | | | | | | | | |
| [ an-node05 ] ______| | | | | | | | | |
| | eth0 =---------/ | | | | | | | |
| |_____|| | | | | | | | |
| | | | | | | | | |
| ______| | | | | | | | |
| | eth1 =-------------------------------/ | | | | |
| |_____|| | | | | | | |
| | | | | | | | |
| ______| | | | | | | |
| | eth2 =-----------------------------------------------------/ | |
| |_____|| | | | | | |
|__________________________| | | | | | |
| | | | | |
__________________________ | | | | | |
| [ an-node06 ] ______| | | | | | |
| | eth0 =-----------/ | | | | |
| |_____|| | | | | |
| | | | | | |
| ______| | | | | |
| | eth1 =---------------------------------/ | | |
| |_____|| | | | |
| | | | | |
| ______| | | | |
| | eth2 =-------------------------------------------------------/ |
| |_____|| | | |
|__________________________| | | |
| | |
__________________________ | | |
| [ an-node07 ] ______| | | |
| | eth0 =-------------/ | |
| |_____|| | |
| | | |
| ______| | |
| | eth1 =-----------------------------------/ |
| |_____|| |
| | |
| ______| |
| | eth2 =---------------------------------------------------------/
| |_____||
|__________________________|
Install The Cluster Software
If you are using Red Hat Enterprise Linux, you will need to add the RHEL Server Optional (v. 6 64-bit x86_64) channel for each node in your cluster. You can do this in RHN by going the your subscription management page, clicking on each server, clicking on "Alter Channel Subscriptions", click to enable the RHEL Server Optional (v. 6 64-bit x86_64) channel and then by clicking on "Change Subscription".
This actual installation is simple, just use yum to install cman.
yum install cman fence-agents
Initial Config
With these decisions and the information gathered, here is what our first /etc/cluster/cluster.conf file will look like.
touch /etc/cluster/cluster.conf
vim /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster name="an-clusterA" config_version="1">
<cman two_node="1" expected_votes="1" />
<totem secauth="off" rrp_mode="none" />
<clusternodes>
<clusternode name="an-node01.alteeve.com" nodeid="1">
<fence>
<method name="PDU">
<device name="pdu2" action="reboot" port="1" />
</method>
</fence>
</clusternode>
<clusternode name="an-node02.alteeve.com" nodeid="2">
<fence>
<method name="PDU">
<device name="pdu2" action="reboot" port="2" />
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="pdu2" agent="fence_apc" ipaddr="192.168.1.6" login="apc" passwd="secret" />
</fencedevices>
</cluster>
Save the file, then validate it. If it fails, address the errors and try again.
ccs_config_validate
Configuration validates
Push it to the other node:
rsync -av /etc/cluster/cluster.conf root@an-node02:/etc/cluster/
sending incremental file list
cluster.conf
sent 781 bytes received 31 bytes 541.33 bytes/sec
total size is 701 speedup is 0.86
Start:
DO NOT PROCEED UNTIL YOUR cluster.conf FILE VALIDATES!
Unless you have it perfect, your cluster will fail.
Once it validates, proceed.
Starting The Cluster For The First Time
By default, if you start one node only and you've enabled the <cman two_node="1" expected_votes="1"/> option as we have done, the lone server will effectively gain quorum. It will try to connect to the cluster, but there won't be a cluster to connect to, so it will fence the other node after a timeout period. This timeout is 6 seconds by default.
For now, we will leave the default as it is. If you're interested in changing it though, the argument you are looking for is post_join_delay.
This behaviour means that we'll want to start both nodes well within six seconds of one another, least the slower one get needlessly fenced.
Left off here
Note to help minimize dual-fences:
- you could add FENCED_OPTS="-f 5" to /etc/sysconfig/cman on *one* node (ilo fence devices may need this)
iSCSI notes
IET vs tgt pros and cons needed.
default iscsi port is 3260
initiator: This is the client. target: This is the server side. sid: Session ID; Found with iscsiadm -m session -P 1. SID and sysfs path are not persistent, partially start-order based. iQN: iSCSI Qualified Name; This is a string that uniquely identifies targets and initiators.
vim /etc/tgt/targets.conf
Note: Linbit offers 8.3.11 to RHEL6 clients now, use that in this tutorial.
yum install iscsi-initiator-utils scsi-target-utils
/etc/init.d/iscsid start
iscsiadm -m iface -o new --interface=eth1
cp /etc/tgt/targets.conf /etc/tgt/targets.conf.orig
vim /etc/tgt/targets.conf
<source lang="bash">
| Any questions, feedback, advice, complaints or meanderings are welcome. | |||
| Alteeve's Niche! | Enterprise Support: Alteeve Support |
Community Support | |
| © Alteeve's Niche! Inc. 1997-2024 | Anvil! "Intelligent Availability®" Platform | ||
| legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions. | |||