Latest revision as of 02:25, 27 November 2012

AN!Wiki :: How To :: RHCS Release Manager

Warning: These are my notes to help me with my duties as the Red Hat cluster release manager. They are not meant to be a general tutorial and no effort is made to make these notes useful in any general way. Unless you are taking over my duties, this page is probably useless to you.

Test Results

These tests are run by copying over the tarball created on the build machine, untar'ing it and running ./configure && make.

Testing v3.1.9x

Per-release test results.

Distro Build Tests

This covers ./configure && make && make install && make uninstall

Distro	Arch	Date tested (YYYY-MM-DD)
Fedora 16	x86_64	2012-05-27
Fedora 16	i386	2012-05-27
Fedora 15	x86-64	2012-05-27
Fedora 15	i386	2012-05-27
Ubuntu 12.04	amd64	2012-05-27
Ubuntu 12.04	i386	2012-05-27

Cluster Tests

Host Nodes are Fedora 16, x86_64.

Test	Result	Notes
Install via make install	Pass
Start One Node, No quorum or fence	Pass
Start Second Node, Gain quorum and fence third	Pass
Full Cluster Start	Pass
Withdraw One Node, Retain Quorum	Pass
Withdraw Second Node, Drop Quorum
Start rgmanager, Service start
ordered=1 Service migrates when higher priority node joins
Push out updated cluster.conf with new IP, pushes and updates IP
Stop and Start the service via rgmanager
Manual relocate the service
Crash node, fences, recovers service
Release Ready?
General Notes	Notes	qdisk untested. RPM build untested.

Cluster configuration used

The /etc/cluster/cluster.conf file used in the cluster VMs:

<?xml version="1.0"?>
<cluster config_version="3" name="rm-cluster">
	<totem rrp_mode="none" secauth="off"/>
	<clusternodes>
		<clusternode name="t1.alteeve.ca" nodeid="1">
			<fence>
				<method name="xvm">
					<device action="reboot" domain="t1" name="xvm"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="t2.alteeve.ca" nodeid="2">
			<fence>
				<method name="xvm">
					<device action="reboot" domain="t2" name="xvm"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="t3.alteeve.ca" nodeid="3">
			<fence>
				<method name="xvm">
					<device action="reboot" domain="t3" name="xvm"/>
				</method>
			</fence>
		</clusternode>
	</clusternodes>
	<fencedevices>
		<fencedevice agent="fence_xvm" name="xvm"/>
	</fencedevices>
	<fence_daemon post_join_delay="30"/>
	<rm log_level="5">
		<resources>
			<ip address="192.168.122.200"/>
		</resources>
		<failoverdomains>
			<failoverdomain name="virt-ip" nofailback="0" ordered="1" restricted="0">
				<failoverdomainnode name="t1.alteeve.ca" priority="1"/>
				<failoverdomainnode name="t2.alteeve.ca" priority="2"/>
				<failoverdomainnode name="t3.alteeve.ca" priority="3"/>
			</failoverdomain>
		</failoverdomains>
		<service autostart="1" domain="virt-ip" name="float_ip" nfslock="1" recovery="relocate">
			<ip ref="192.168.122.200"/>
		</service>
	</rm>
</cluster>

Note: The host needs to have install yum install fence-virtd fence-virtd-libvirt fence-virtd-multicast and then start the fence-virtd daemon.

The /etc/fence_virt.conf file used on the host.

backends {
        libvirt {
                uri = "qemu:///system";
        }

}

listeners {
        multicast {
                interface = "virbr0";
                port = "1229";
                family = "ipv4";
                address = "225.0.0.12";
        }

}

fence_virtd {
        module_path = "/usr/lib64/fence-virt";
        backend = "libvirt";
        listener = "multicast";
}

Old Tests

Previous cluster Release Tests

Build Environment

This is how to setup a machine for building and releasing new version of RHCS. This requires a proper FAS account.

yum -y groupinstall "Development Libraries" "Development Tools" "Fedora Packager"
yum -y install vim wget gnupg

Remove any old builds.

cd ~/projects/RedHat/release-manager/
rm -rf ./cluster*

Warning: This must be done on the machine with the Fedora Account System - Fedora Project key.

Check-out latest cluster.

cd ~/projects/RedHat/release-manager/
git clone ssh://git.fedorahosted.org/git/cluster.git
cd cluster/
git branch stable32 --track origin/STABLE32
git checkout stable32

Change this to reflect the appropriate versions.

Note: You will need to enter the proper passhrase to sign this TC.

make -f make/release.mk version=3.2.0 oldversion=3.1.93

Test Release Tarball Creation

Push the test candidate to each VM to be tested. Change the version number as appropriate.

cd ../cluster-3.2.0-release-candidate/
ls -lah

total 6.6M
drwxrwxr-x 2 digimer digimer 4.0K Jun  1 15:03 .
drwxrwxr-x 6 digimer digimer 4.0K Jun  1 15:02 ..
-rw-rw-r-- 1 digimer digimer  172 Jun  1 15:03 Changelog-3.2.0
-rw-rw-r-- 1 digimer digimer  619 Jun  1 15:03 cluster-3.2.0.sha256
-rw-rw-r-- 1 digimer digimer  836 Jun  1 15:03 cluster-3.2.0.sha256.asc
-rw-rw-r-- 1 digimer digimer 3.2M Jun  1 15:02 cluster-3.2.0.tar
-rw-rw-r-- 1 digimer digimer 526K Jun  1 15:02 cluster-3.2.0.tar.bz2
-rw-rw-r-- 1 digimer digimer 659K Jun  1 15:02 cluster-3.2.0.tar.gz
-rw-rw-r-- 1 digimer digimer 487K Jun  1 15:03 cluster-3.2.0.tar.xz
-rw-rw-r-- 1 digimer digimer 1.2M Jun  1 15:03 rgmanager-3.2.0.tar
-rw-rw-r-- 1 digimer digimer 190K Jun  1 15:03 rgmanager-3.2.0.tar.bz2
-rw-rw-r-- 1 digimer digimer 234K Jun  1 15:03 rgmanager-3.2.0.tar.gz
-rw-rw-r-- 1 digimer digimer 185K Jun  1 15:03 rgmanager-3.2.0.tar.xz
-rw-rw-r-- 1 digimer digimer    0 Jun  1 15:02 tag-3.2.0

rsync -av cluster-3.2.0.tar root@f16-32-1:/root/
rsync -av cluster-3.2.0.tar root@f16-64-1:/root/
rsync -av cluster-3.2.0.tar root@t1:/root/
rsync -av cluster-3.2.0.tar root@t2:/root/
rsync -av cluster-3.2.0.tar root@t3:/root/

Per-Node Compile Test

64-bit nodes:

yum install -y kernel-devel kernel-headers make gcc libxml2-devel openldap-devel dbus-devel net-snmp-libs libibverbs librdmacm \
               slang-devel ncurses-devel perl-ExtUtils-MakeMaker openaislib-devel fence-agents resource-agents

32-bit nodes:

yum install -y kernel-PAE-devel kernel-headers make gcc libxml2-devel openldap-devel dbus-devel net-snmp-libs libibverbs librdmacm \
               slang-devel ncurses-devel perl-ExtUtils-MakeMaker openaislib-devel fence-agents resource-agents

For distro testing, only ./configure && make are done.

cd ~
rm -rf cluster*
tar -xvf cluster-3.2.0.tar 
cd cluster-3.2.0
./configure
make

Cluster Test

This is done on the final 3-node cluster on.

Copy the test tarball to the test nodes.

cd ~/projects/RedHat/release-manager/cluster
rsync -av cluster-3.2.0-release-candidate/cluster-3.2.0.tar root@t1:/root/
rsync -av cluster-3.2.0-release-candidate/cluster-3.2.0.tar root@t2:/root/
rsync -av cluster-3.2.0-release-candidate/cluster-3.2.0.tar root@t3:/root/

Then, on each node, run the following commands (replace the version with the appropriate one, of course.)

cd ~
tar -xvf cluster-3.2.0.tar
cd cluster-3.2.0
yum -y update
yum -y install vim wget corosynclib-devel openaislib-devel fence-agents resource-agents modcluster ricci
./configure 
make
make install

Pushing Changes to git

Initial Setup of git

Don't have notes, this is from bash's history. Sort this out later.

Misc commands

man git-send-email
git clone http://git.fedorahosted.org/git/cluster.git
man git
git branch http://git.fedorahosted.org/git/cluster.git
man git
git show-branch http://git.fedorahosted.org/git/cluster.git
git clone http://git.fedorahosted.org/git/cluster.git
git show-branch
git branch

Pushing To origin

Confirming the changes before committing.

git diff

Pushing changes;

Commit locally with signature.

git commit -a -s

(open's editor)

[stable31 9be00f8] Changes the kernel version check to handle 3.x.y kernels. Now if the 'x' version of the running kernel is higher than the 'x' version of the minimum kernel, the test passes. Also changed the method of checking that version numbers were gathered so that a version number of '0' would be seen as valid.
 Committer: Digital Mermaid <digimer@lework.alteeve.com>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:

    git config --global user.name "Your Name"
    git config --global user.email you@example.com

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 1 files changed, 3 insertions(+), 2 deletions(-)

Check the current branch name.

git branch

  master
* stable31

This shows that stable31 is active.

Now push up to the main git repo.

git push origin stable31:STABLE31

Counting objects: 5, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 543 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/cluster.git
   991bfb0..9be00f8  stable31 -> STABLE31

An email should have been automatically sent to the appropriate mailing lists.

Distro Testing

Note: Always update the OS before running tests!

Fedora 15, 16 and Rawhide

Packages to install;

yum -y groupinstall "Development Libraries" "Development Tools" "Fedora Packager"
yum -y install vim wget corosynclib-devel openaislib-devel fence-agents resource-agents modcluster ricci fence-virt fence-virtd-libvirt fence-virtd-multicast

32-bit versions also need;

yum -y install kernel-PAE-devel

Now make sure that ricci is running and that selinux, iptables and ip6tables are off.

sed -e "s/SELINUX=enforcing/SELINUX=disabled/" -i /etc/selinux/config
systemctl disable ip6tables.service
systemctl disable iptables.service
systemctl enable modclusterd.service
systemctl enable ricci.service
systemctl stop ip6tables.service
systemctl stop iptables.service
systemctl start modclusterd.service
systemctl restart ricci.service

Reboot if selinux was updated.

Debian 6

Debian 6 does not support RHEL's Cluster 3.1+ as it's included version of corosync is too old. No further compatibility testing will be run for this version of Debian.

Ubuntu 11.10

Packages to install;

apt-get update
apt-get -y dist-upgrade
apt-get -y install linux-headers-$(uname -r) libxml2-dev libcorosync-dev libldap2-dev zlib1g-dev libopenais-dev libdbus-1-dev \
 libslang2-dev libncurses5-dev make

Pushing The Release

Once testing is complete and you're ready to release;

Build the final release

cd ~/projects/RedHat/release-manager/cluster
make -f make/release.mk version=3.2.0 oldversion=3.1.93 release=1

<make noise>

You need a passphrase to unlock the secret key for
user:

"Cluster Release Team <cluster-devel@redhat.com>"
4096-bit RSA key, ID 6CE95CA7, created 2010-02-08

Make sure things built properly.

ls -lah ../

total 24K
drwxrwxr-x   6 digimer digimer 4.0K Jan  3 22:47 .
drwxrwxr-x.  7 digimer digimer 4.0K Nov 28 15:26 ..
drwxr-xr-x  15 digimer digimer 4.0K Dec 30 17:55 cluster
drwxrwxr-x   2 digimer digimer 4.0K Jan  3 22:47 cluster-3.2.0-release

Test build one last time

Take the most recently built tarball and do a final test build.

On the test machine, delete the previous build.

ssh root@an-node03

Last login: Fri Dec 30 13:10:15 2011 from 10.20.0.100

rm -rf cluster-*

Back on the build machine, copy the 'release' tarball over.

rsync -av ../cluster-3.2.0-release/cluster-3.2.0.tar root@t1:/root/

sending incremental file list
cluster-3.2.0.tar

sent 9029 bytes  received 10759 bytes  13192.00 bytes/sec
total size is 3174400  speedup is 160.42

Now back on the test machine, extract and build.

tar -xvf cluster-3.2.0.tar
cd cluster-3.2.0/
./configure 
make
make install

If all goes well, we're ready to release!

Push the release!

Make sure you have a summary of the changes.

cat ~/projects/RedHat/release-manager/cluster-3.2.0-release/Changelog-3.1.93

<changes>

Make a one/two sentence summary of the major points for the email.

Warning: The next step creates the release!

make -f make/release.mk version=3.2.0 oldversion=3.1.93 release=1 publish

Make sure the file was uploaded
Update the wiki

Send the release email.

To: linux clustering <linux-cluster@redhat.com>
Subject: cluster <ver> released
Body:

Welcome to the cluster <ver> release.

<short highlight of changes>

The new source tarball can be downloaded here:

https://fedorahosted.org/releases/c/l/cluster/cluster-<ver>.tar.xz

ChangeLog:

https://fedorahosted.org/releases/c/l/cluster/Changelog-<ver>

To report bugs or issues:

   https://bugzilla.redhat.com/

Would you like to meet the cluster team or members of its community?

   Join us on IRC (irc.freenode.net #linux-cluster) and share your
   experience  with other sysadministrators or power users.

Thanks/congratulations to all people that contributed to achieve this
great milestone.

Happy clustering,
Digimer

Any questions, feedback, advice, complaints or meanderings are welcome.
`Alteeve's Niche!`	`Enterprise Support: Alteeve Support`	`Community Support`
© Alteeve's Niche! Inc. 1997-2024		Anvil! "Intelligent Availability®" Platform
`legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.`

RHCS Release Manager: Difference between revisions