Sheepdog on EL6

AN!Wiki :: How To :: Sheepdog on EL6

Warning: This is essentially just a compilation of notes I made while playing with NTT's Sheepdog. It may or may not become a complete tutorial someday.

Installing, configuring and using Sheepdog in EL6.

Base Cluster Config

Note: Replace bindnetaddr's value with the IP address of the interface you want the cluster communication on (BCN, generally).

vim /etc/corosync/corosync.conf

# Please read the corosync.conf 5 manual page
compatibility: whitetank
totem {
  version: 2
  secauth: off
  threads: 0
  # Note, fail_recv_const is only needed if you're 
  # having problems with corosync crashing under 
  # heavy sheepdog traffic. This crash is due to 
  # delayed/resent/misordered multicast packets. 
  # fail_recv_const: 5000
  interface {
    ringnumber: 0
    bindnetaddr: 192.168.3.{3..7}
    mcastaddr: 226.94.1.1
    mcastport: 5405
  }
}
logging {
  fileline: off
  to_stderr: no
  to_logfile: yes
  to_syslog: yes
  # the pathname of the log file
  logfile: /var/log/cluster/corosync.log
  debug: off
  timestamp: on
  logger_subsys {
    subsys: AMF
    debug: off
  }
}
amf {
  mode: disabled
}

Now start corosync on the first node and then check corosync-objctl | grep member to make sure it came up.

/etc/init.d/corosync start

Starting Corosync Cluster Engine (corosync):               [  OK  ]

corosync-objctl |grep member

runtime.totem.pg.mrp.srp.members.1224976576.ip=r(0) ip(192.168.3.73) 
runtime.totem.pg.mrp.srp.members.1224976576.join_count=1
runtime.totem.pg.mrp.srp.members.1224976576.status=joined

Now start corosync on the other nodes. You should then see the other nodes, five total in my case, come online.

corosync-objctl |grep member

runtime.totem.pg.mrp.srp.members.1224976576.ip=r(0) ip(192.168.3.73) 
runtime.totem.pg.mrp.srp.members.1224976576.join_count=1
runtime.totem.pg.mrp.srp.members.1224976576.status=joined
runtime.totem.pg.mrp.srp.members.1241753792.ip=r(0) ip(192.168.3.74) 
runtime.totem.pg.mrp.srp.members.1241753792.join_count=1
runtime.totem.pg.mrp.srp.members.1241753792.status=joined
runtime.totem.pg.mrp.srp.members.1258531008.ip=r(0) ip(192.168.3.75) 
runtime.totem.pg.mrp.srp.members.1258531008.join_count=1
runtime.totem.pg.mrp.srp.members.1258531008.status=joined
runtime.totem.pg.mrp.srp.members.1275308224.ip=r(0) ip(192.168.3.76) 
runtime.totem.pg.mrp.srp.members.1275308224.join_count=1
runtime.totem.pg.mrp.srp.members.1275308224.status=joined
runtime.totem.pg.mrp.srp.members.1292085440.ip=r(0) ip(192.168.3.77) 
runtime.totem.pg.mrp.srp.members.1292085440.join_count=1
runtime.totem.pg.mrp.srp.members.1292085440.status=joined

Install

Done as root (ya, I know...) on CentOS 6.0;

Install dependencies;

yum -y install nss-devel zlib-devel

Corosync from source;

yum -y remove corosync corosynclib corosynclib-devel
cd ~
git clone git://github.com/corosync/corosync.git
cd corosync
git checkout -b flatiron origin/flatiron
./autogen.sh
./configure --enable-nss
make install
cd ~

QEMU (needs >= v0.13);

cd ~
yum -y remove libvirt
git clone git://git.sv.gnu.org/qemu.git
cd qemu
./configure
make install
cd ~

Sheepdog (we don't build the RPMs for missing dependencies reasons);

cd ~
git clone git://github.com/collie/sheepdog.git
cd sheepdog
./autogen.sh
./configure
make install
cd ~

Using Sheepdog

You need a "store" directory. At this point in development, the /etc/init.d/sheepdog initialization script is hard coded to use /var/lib/sheepdog, so that is what we will use. I formatted this partition on /dev/sda5 as an ext4 partition.

mkfs.ext4 /dev/sda5
mkdir /var/lib/sheepdog
tune2fs -l /dev/sda5 | grep "Filesystem UUID" | sed -e "s/^Filesystem UUID:\s*\(.*\)/UUID=\L\1\E\t\/var/lib/sheepdog\text4\tdefaults,user_xattr\t1 3/" >> /etc/fstab
mount /var/lib/sheepdog
mount

/dev/sda2 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sda5 on /var/lib/sheepdog type ext4 (rw,user_xattr)

Note the (rw,user_xattr), this is important.

Now create the store on all nodes.

sheep /var/lib/sheepdog

You can verify which nodes are in the sheepdog cluster using the following command;

Tell it to create five copies.

 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche!	Enterprise Support: Alteeve Support	Community Support
© Alteeve's Niche! Inc. 1997-2024		Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.