Difference between revisions of "DRBD on Fedora 13"

From AN!Wiki
Jump to: navigation, search
(Configuration Files)
(Starting The DRBD Resource)
Line 218: Line 218:
 
</source>
 
</source>
  
 +
'''''FILL THIS IN'''''.
 +
 +
= Setting up CLVM =
 +
 +
The goal of DRBD in the cluster is to provide clustered [[LVM]] to the nodes. This is done by turning the DRBD partition into an LVM physical volume.
 +
 +
So now we will "stack" LVM by creating a [[PV]] on top of the new [[DRBD]] partition, <span class="code">/dev/drbd0</span>, that we created in the previous step. Since this new LVM [[PV]] will exist on top of the shared DRBD partition, whatever get written to it's logical volumes will be immediately available on either node, regardless of which node actually initiated the write.
 +
 +
This capability is the underlying reason for creating this cluster; Neither machine is truly needed so if one machine dies, anything on top of the DRBD partition will still be available. When the failed machine returns, the surviving node will have a list of what blocks changed while the other node was gone and can use this list to quickly re-sync the other server.
 +
 +
== Making LVM Cluster-Aware ==
 +
 +
Normally, LVM is run on a single server. This means that at any time, the LVM can write data to the underlying drive and not need to worry if any other device might change anything. In clusters, this isn't the case. The other node could try to write to the shared storage, so then nodes need to enable "locking" to prevent the two nodes from trying to work on the same bit of data at the same time.
 +
 +
The process of enabling this locking is known as making LVM "cluster-aware".
 +
 +
=== Updating '/etc/lvm/lvm.conf' ===
 +
 +
'''Note''': After CentOS 5.5 is released, this step should no longer be needed. See [https://bugzilla.redhat.com/show_bug.cgi?id=530881 here] for details.
 +
 +
By default, LVM ignores DRBD devices as candidate physical volumes. To enable DRBD, we need to change the <span class="code">filter</span> in <span class="code">/etc/lvm/lvm.conf</span> to include a regular expression that matches the name of our DRBD device. We created our DRBD device as <span class="code">/dev/drbd0</span>, so changing the <span class="code">filter</span> to <span class="code">filter = [ "a|drbd.*|", "a|sd.*|", "r|.*|" ]</span> ('''a'''ccept <span class="code">drbd</span> devices and devices with the <span class="code">sd*</span> "scsi" names, '''r'''eject everything else) will do this. Edit <span class="code">lvm.conf</span> and change it to match this:
 +
 +
<source lang="bash">
 +
vim /etc/lvm/lvm.conf
 +
</source>
 +
<source lang="text">
 +
    # By default we accept every block device:
 +
    #filter = [ "a/.*/" ]
 +
    filter = [ "a|drbd.*|", "a|sd.*|", "r|.*|" ]
 +
</source>
 +
 +
=== Enabling Cluster Locking ===
 +
 +
LVM has a built-in tool called <span class="code">lvmconf</span> that can be used to enable LVM locking. This is provided as part of the <span class="code">lvm2-cluster</span> package. Install it and then run:
 +
 +
<source lang="bash">
 +
yum install lvm2-cluster.x86_64
 +
</source>
 +
 +
Before you continue, we want to tell <span class="code">clvmd</span> that it should start '''after''' the <span class="code">drbd</span> daemon. To do this, edit it's <span class="code">/etc/init.d/clvmd</span> init script and add <span class="code">drbd</span> to it's start block.
 +
 +
<source lang="bash">
 +
vim /etc/init.d/clvmd
 +
</source>
 +
 +
Change the header by adding <span class="code">drbd</span> to <span class="code">Required-Start</span> and <span class="code">Required-Stop</span>:
 +
 +
<source lang="text">
 +
### BEGIN INIT INFO
 +
# Provides: clvmd
 +
# Required-Start: $local_fs drbd
 +
# Required-Stop: $local_fs drbd
 +
# Default-Start:
 +
# Default-Stop: 0 1 6
 +
# Short-Description: Clustered LVM Daemon
 +
### END INIT INFO
 +
</source>
 +
 +
Change the start order by removing and re-adding all cluster-related daemons:
 +
 +
'''Note'''': This has extra entries to make sure all relevant entries are sorted when re-added. If you do not have some of the daemons listed below installed yet, simply skip them.
 +
 +
<source lang="bash">
 +
chkconfig xend off; chkconfig cman off; chkconfig drbd off; chkconfig clvmd off; chkconfig xendomains off
 +
chkconfig xend on; chkconfig cman on; chkconfig drbd on; chkconfig clvmd on; chkconfig xendomains on
 +
</source>
 +
 +
Now to enable cluster awareness in LVM, simply run:
 +
 +
<source lang="bash">
 +
lvmconf --enable-cluster
 +
</source>
 +
 +
There won't be any output from that command.
 +
 +
By default, <span class="code">clvmd</span>, the cluster lvm daemon, is stopped and not set to run on boot. Now that we've enabled LVM locking, we need to start it:
 +
 +
<source lang="bash">
 +
/etc/init.d/clvmd status
 +
</source>
 +
<source lang="text">
 +
clvmd is stopped
 +
active volumes: lv00 lv01 lv02
 +
</source>
 +
 +
As expected, it is stopped, so lets start it and then use <span class="code">chkconfig</span> to enable it at boot.
 +
 +
<source lang="bash">
 +
/etc/init.d/clvmd start
 +
</source>
 +
<source lang="text">
 +
Stopping clvm:                                            [  OK  ]
 +
Starting clvmd:                                            [  OK  ]
 +
Activating VGs:  3 logical volume(s) in volume group "an-lvm01" now active
 +
                                                          [  OK  ]
 +
</source>
 +
 +
<source lang="bash">
 +
chkconfig clvmd on
 +
ls -lah /etc/rc3.d/ |grep clvmd
 +
</source>
 +
<source lang="text">
 +
lrwxrwxrwx  1 root root  15 Mar 16 12:48 S24clvmd -> ../init.d/clvmd
 +
</source>
 +
 +
We can see that it is now set to start at position <span class="code">24</span>.
 +
 +
== Creating a new PV using the DRBD Partition ==
 +
 +
We can now proceed with setting up the new DRBD-based LVM physical volume. Once the PV is created, we can create a new volume group and start allocating space to logical volumes.
 +
 +
'''Note''': As we will be using our DRBD device, and as it is a shared block device, most of the following commands only need to be run on one node. Once the block device changes in any way, those changes will near-instantly appear on the other node. For this reason, unless explicitly stated to do so, only run the following commands on one node.
 +
 +
To setup the DRBD partition as an LVM PV, run <span class="code">pvcreate</span>:
 +
 +
<source lang="bash">
 +
pvcreate /dev/drbd0
 +
</source>
 +
<source lang="text">
 +
  Physical volume "/dev/drbd0" successfully created
 +
</source>
 +
 +
Now, on both nodes, check that the new physical volume is visible by using <span class="code">pvdisplay</span>:
 +
 +
<source lang="bash">
 +
pvdisplay
 +
</source>
 +
<source lang="text">
 +
  --- Physical volume ---
 +
  PV Name              /dev/sda2
 +
  VG Name              san01
 +
  PV Size              465.51 GB / not usable 14.52 MB
 +
  Allocatable          yes
 +
  PE Size (KByte)      32768
 +
  Total PE              14896
 +
  Free PE              1407
 +
  Allocated PE          13489
 +
  PV UUID              IpySTY-a9BY-31XE-Bxd4-H9sp-OEJG-kP7dtg
 +
 
 +
  "/dev/drbd0" is a new physical volume of "399.99 GB"
 +
  --- NEW Physical volume ---
 +
  PV Name              /dev/drbd0
 +
  VG Name             
 +
  PV Size              399.99 GB
 +
  Allocatable          NO
 +
  PE Size (KByte)      0
 +
  Total PE              0
 +
  Free PE              0
 +
  Allocated PE          0
 +
  PV UUID              S6OkVh-NlwQ-BaUn-k5LI-1iTo-pu8V-Uq3qE2
 +
</source>
 +
 +
If you see <span class="code">PV Name /dev/drbd0</span> on both nodes, then your DRBD setup and LVM configuration changes are working perfectly!
 +
 +
== Creating a VG on the new PV ==
 +
 +
Now we need to create the volume group using the <span class="code">vgcreate</span> command:
 +
 +
<source lang="bash">
 +
vgcreate -c y drbd_vg0 /dev/drbd0
 +
</source>
 +
<source lang="text">
 +
  Clustered volume group "drbd_vg0" successfully created
 +
</source>
 +
 +
Now we'll check that the new VG is visible on both nodes using <span class="code">vgdisplay</span>:
 +
 +
<source lang="bash">
 +
vgdisplay
 +
</source>
 +
<source lang="text">
 +
  --- Volume group ---
 +
  VG Name              an-lvm01
 +
  System ID           
 +
  Format                lvm2
 +
  Metadata Areas        1
 +
  Metadata Sequence No  4
 +
  VG Access            read/write
 +
  VG Status            resizable
 +
  MAX LV                0
 +
  Cur LV                3
 +
  Open LV              3
 +
  Max PV                0
 +
  Cur PV                1
 +
  Act PV                1
 +
  VG Size              465.50 GB
 +
  PE Size              32.00 MB
 +
  Total PE              14896
 +
  Alloc PE / Size      13489 / 421.53 GB
 +
  Free  PE / Size      1407 / 43.97 GB
 +
  VG UUID              C0kHFA-OTo8-Gshr-3wIw-3Q0I-eT3X-A9Y0NA
 +
 
 +
  --- Volume group ---
 +
  VG Name              drbd_vg0
 +
  System ID           
 +
  Format                lvm2
 +
  Metadata Areas        1
 +
  Metadata Sequence No  2
 +
  VG Access            read/write
 +
  VG Status            resizable
 +
  Clustered            yes
 +
  Shared                no
 +
  MAX LV                0
 +
  Cur LV                1
 +
  Open LV              0
 +
  Max PV                0
 +
  Cur PV                1
 +
  Act PV                1
 +
  VG Size              399.98 GB
 +
  PE Size              4.00 MB
 +
  Total PE              102396
 +
  Alloc PE / Size      5120 / 20.00 GB
 +
  Free  PE / Size      97276 / 379.98 GB
 +
  VG UUID              TmlQmv-eViK-7Ubr-Dyck-0u86-uEWJ-rDOt9i
 +
</source>
 +
 +
 +
If the new VG is visible on both nodes, we are ready to create our first logical volume using the <span class="code">lvcreate</span> tool.
 +
 +
== Creating the First Two LVs on the new VG ==
 +
 +
Now we'll create two simple 20 GiB logical volumes. This first one will be a shared GFS store for source ISOs and the second will be used for our first virtual machine.
 +
<source lang="bash">
 +
lvcreate -L 20G -n iso_store drbd_vg0
 +
lvcreate -L 20G -n vm01 drbd_vg0
 +
</source>
 +
<source lang="text">
 +
  Logical volume "iso_store" created
 +
  Logical volume "vm01" created
 +
</source>
 +
 +
As before, we will check that the new logical volume is visible from both nodes by using the <span class="code">lvdisplay</span> command:
 +
 +
<source lang="bash">
 +
lvdisplay
 +
</source>
 +
<source lang="text">
 +
  --- Logical volume ---
 +
  LV Name                /dev/an-lvm02/lv01
 +
  VG Name                an-lvm02
 +
  LV UUID                Dy2MNa-EUxN-9x6f-ovkj-NCpk-nlV2-kr5QBb
 +
  LV Write Access        read/write
 +
  LV Status              available
 +
  # open                1
 +
  LV Size                19.53 GB
 +
  Current LE            625
 +
  Segments              1
 +
  Allocation            inherit
 +
  Read ahead sectors    auto
 +
  - currently set to    256
 +
  Block device          253:0
 +
 
 +
  --- Logical volume ---
 +
  LV Name                /dev/an-lvm02/lv00
 +
  VG Name                an-lvm02
 +
  LV UUID                xkBu7j-wtOe-ORr3-68qJ-u0ux-Qif4-stw5SY
 +
  LV Write Access        read/write
 +
  LV Status              available
 +
  # open                1
 +
  LV Size                2.00 GB
 +
  Current LE            64
 +
  Segments              1
 +
  Allocation            inherit
 +
  Read ahead sectors    auto
 +
  - currently set to    256
 +
  Block device          253:1
 +
 
 +
  --- Logical volume ---
 +
  LV Name                /dev/an-lvm02/lv02
 +
  VG Name                an-lvm02
 +
  LV UUID                R20GH1-wQKq-WgUR-x1gx-Yzzp-WjND-WHAjEO
 +
  LV Write Access        read/write
 +
  LV Status              available
 +
  # open                2
 +
  LV Size                400.00 GB
 +
  Current LE            12800
 +
  Segments              1
 +
  Allocation            inherit
 +
  Read ahead sectors    auto
 +
  - currently set to    256
 +
  Block device          253:2
 +
 
 +
  --- Logical volume ---
 +
  LV Name                /dev/drbd_vg0/iso_store
 +
  VG Name                drbd_vg0
 +
  LV UUID                svJx35-KDXK-ojD2-UDAA-Ah9t-UgUl-ijekhf
 +
  LV Write Access        read/write
 +
  LV Status              available
 +
  # open                0
 +
  LV Size                20.00 GB
 +
  Current LE            5120
 +
  Segments              1
 +
  Allocation            inherit
 +
  Read ahead sectors    auto
 +
  - currently set to    256
 +
  Block device          253:3
 +
 
 +
  --- Logical volume ---
 +
  LV Name                /dev/drbd_vg0/vm01
 +
  VG Name                drbd_vg0
 +
  LV UUID                sceLmK-ZJIp-fN5g-RMaS-j5sq-NuY5-7hIwhP
 +
  LV Write Access        read/write
 +
  LV Status              available
 +
  # open                0
 +
  LV Size                20.00 GB
 +
  Current LE            5120
 +
  Segments              1
 +
  Allocation            inherit
 +
  Read ahead sectors    auto
 +
  - currently set to    256
 +
  Block device          253:4
 +
</source>
 +
 +
The last two are the new logical volumes.
 +
 +
== Create the Shared GFS FileSystem ==
 +
 +
GFS is a cluster-aware file system that can be simultaneously mounted on two or more nodes at once. We will use it as a place to store ISOs that we'll use to provision our virtual machines.
 +
 +
The following example is designed for the cluster used in this paper.
 +
* If you have more than 2 nodes, increase the <span class="code">-j 2</span> to the number of nodes you want to mount this file system on.
 +
* If your cluster is named something other than <span class="code">an-cluster</span> (as set in the <span class="code">cluster.conf</span> file), change <span class="code">-t an-cluster:iso_store</span> to match you cluster's name. The <span class="code">iso_store</span> can be whatever you like, but it must be unique in the cluster. I tend to use a name that matches the LV name, but this is my own preference and is not required.
 +
 +
To format the partition run:
 +
<source lang="bash">
 +
mkfs.gfs2 -p lock_dlm -j 2 -t an-cluster:iso_store /dev/drbd_vg0/iso_store
 +
</source>
 +
 +
If you are prompted, press <span class="code">y</span> to proceed.
 +
 +
Once the format completes, you can mount <span class="code">/dev/drbd_vg0/iso_store</span> as you would a normal file system.
 +
 +
'''Both''':
 +
 +
To complete the example, lets mount the GFS2 partition we made just now on <span class="code">/shared</span>.
 +
 +
<source lang="bash">
 +
mkdir /shared
 +
mount /dev/drbd_vg0/iso_store /shared
 +
</source>
 +
 +
Done!
 +
 +
== Growing a GFS2 Partition ==
 +
 +
To grow a GFS2 partition, you must know where it is mounted. You can not grow an unmounted GFS2 partition, as odd as that may seem at first. Also, you only need to run grow commands from one node. Once completed, all nodes will see and use the new free space automatically.
 +
 +
This requires two steps to complete:
 +
# Extend the underlying LVM logical volume
 +
# Grow the actual GFS2 partition
 +
 +
=== Extend the LVM LV ===
 +
 +
To keep things simple, we'll just use some of the free space we left on our <span class="code">/dev/drbd0</span> LVM physical volume. If you need to add more storage to your LVM first, please follow the instructions in the article: "[[Adding Space to an LVM]]" before proceeding.
 +
 +
Let's add <span class="code">50GB</span> to our GFS2 logical volume <span class="code">/dev/drbd_vg0/iso_store</span> from the <span class="code">/dev/drbd0</span> physical volume, which we know is available because we left more than that back when we first setup our LVM. To actually add the space, we need to use the <span class="code">lvextend</span> command:
 +
 +
<source lang="bash">
 +
lvextend -L +50G /dev/drbd_vg0/iso_store /dev/drbd0
 +
</source>
 +
 +
Which should return:
 +
 +
<source lang="text">
 +
  Extending logical volume iso_store to 70.00 GB
 +
  Logical volume iso_store successfully resized
 +
</source>
 +
 +
If we run <span class="code">lvdisplay /dev/drbd_vg0/iso_store</span> now, we should see the extra space.
 +
 +
<source lang="text">
 +
  --- Logical volume ---
 +
  LV Name                /dev/drbd_vg0/iso_store
 +
  VG Name                drbd_vg0
 +
  LV UUID                svJx35-KDXK-ojD2-UDAA-Ah9t-UgUl-ijekhf
 +
  LV Write Access        read/write
 +
  LV Status              available
 +
  # open                1
 +
  LV Size                70.00 GB
 +
  Current LE            17920
 +
  Segments              2
 +
  Allocation            inherit
 +
  Read ahead sectors    auto
 +
  - currently set to    256
 +
  Block device          253:3
 +
</source>
 +
 +
You're now ready to proceed.
 +
 +
=== Grow The GFS2 Partition ===
 +
 +
This step is pretty simple, but you need to enter the commands exactly. Also, you'll want to do a dry-run first and address any resulting errors before issuing the final <span class="code">gfs2_grow</span> command.
 +
 +
To get the exact name to use when calling <span class="code">gfs2_grow</span>, run the following command:
 +
 +
<source lang="bash">
 +
gfs2_tool df
 +
</source>
 +
<source lang="text">
 +
/shared:
 +
  SB lock proto = "lock_dlm"
 +
  SB lock table = "an-cluster:iso_store"
 +
  SB ondisk format = 1801
 +
  SB multihost format = 1900
 +
  Block size = 4096
 +
  Journals = 2
 +
  Resource Groups = 80
 +
  Mounted lock proto = "lock_dlm"
 +
  Mounted lock table = "an-cluster:iso_store"
 +
  Mounted host data = "jid=1:id=196610:first=0"
 +
  Journal number = 1
 +
  Lock module flags = 0
 +
  Local flocks = FALSE
 +
  Local caching = FALSE
 +
 +
  Type          Total Blocks  Used Blocks    Free Blocks    use%         
 +
  ------------------------------------------------------------------------
 +
  data          5242304        1773818        3468486        34%
 +
  inodes        3468580        94            3468486        0%
 +
</source>
 +
 +
From this output, we know that GFS2 expects the name "<span class="code">/shared</span>". Even adding something as simple as a trailing slash ''will not work''. The program we will use is called <span class="code">gfs2_grow</span> with the <span class="code">-T</span> switch to run the command as a test to work out possible errors.
 +
 +
For example, if you added the trailing slash, this is the kind of error you would see:
 +
 +
'''Bad command''':
 +
<source lang="bash">
 +
gfs_grow -T /shared/
 +
</source>
 +
<source lang="bash">
 +
GFS Filesystem /shared/ not found
 +
</source>
 +
 +
Once we get it right, it will look like this:
 +
<source lang="bash">
 +
gfs_grow -T /shared
 +
</source>
 +
<source lang="bash">
 +
(Test mode--File system will not be changed)
 +
FS: Mount Point: /shared
 +
FS: Device:      /dev/mapper/drbd_vg0-iso_store
 +
FS: Size:        5242878 (0x4ffffe)
 +
FS: RG size:    65535 (0xffff)
 +
DEV: Size:      18350080 (0x1180000)
 +
The file system grew by 51200MB.
 +
gfs2_grow complete.
 +
</source>
 +
 +
This looks good! We're now ready to re-run the command without the <span class="code">-T</span> switch:
 +
<source lang="bash">
 +
gfs_grow /shared
 +
</source>
 +
<source lang="bash">
 +
FS: Mount Point: /shared
 +
FS: Device:      /dev/mapper/drbd_vg0-iso_store
 +
FS: Size:        5242878 (0x4ffffe)
 +
FS: RG size:    65535 (0xffff)
 +
DEV: Size:      18350080 (0x1180000)
 +
The file system grew by 51200MB.
 +
gfs2_grow complete.
 +
</source>
 +
 +
You can check that the new space is available on both nodes now using a simple call like <span class="code">df -h</span>.
  
Once you see this, you can proceed.
 
  
 
{{footer}}
 
{{footer}}

Revision as of 21:18, 30 August 2010

 AN!Wiki :: How To :: DRBD on Fedora 13

Warning: Until this warning is removed, do not use or trust this document. When complete and tested, this warning will be removed.

This article covers installing and configuring DRBD on a two-node Fedora 13 cluster.

Contents

Why DRBD?

DRBD is useful in small clusters as it provides real-time mirroring of data across two (or more) nodes. In two-node clusters, this can be used to host clustered LVM physical volumes. On these volumes you can create logical volumes to host GFS2 partitions, virtual machines, iSCSI and so forth.

Install

yum install drbd.x86_64 drbd-xen.x86_64

Compile the DRBD module for Xen dom0

If you are running the custom Xen dom0, you will need to build the DRBD module from the source RPM.

Install the build environment:

yum -y groupinstall "Development Libraries"
yum -y groupinstall "Development Tools"

Install the kernel headers and development library for the dom0 kernel:

Note: The following commands use --force to get past the fact that the headers for the 2.6.33 are already installed, thus making RPM think that these are too old and will conflict. Please proceed with caution.

rpm -ivh --force http://fedorapeople.org/~myoung/dom0/x86_64/kernel-headers-2.6.32.17-157.xendom0.fc12.x86_64.rpm http://fedorapeople.org/~myoung/dom0/x86_64/kernel-devel-2.6.32.17-157.xendom0.fc12.x86_64.rpm

Download, prepare, build and install the source RPM:

rpm -ivh http://fedora.mirror.iweb.ca/releases/13/Everything/source/SRPMS/drbd-8.3.7-2.fc13.src.rpm
cd /root/rpmbuild/SPECS/
rpmbuild -bp drbd.spec 
cd /root/rpmbuild/BUILD/drbd-8.3.7/
./configure --enable-spec --with-km
cp /root/rpmbuild/BUILD/drbd-8.3.7/drbd-km.spec /root/rpmbuild/SPECS/
cd /root/rpmbuild/SPECS/
rpmbuild -ba drbd-km.spec
cd /root/rpmbuild/RPMS/x86_64
rpm -Uvh drbd-km-*

You should be good to go now!

Configure

We need to see how much space you have left on you LVM PV. The pvscan tool will show you this.

pvscan
  PV /dev/sda2   VG vg_01   lvm2 [465.50 GiB / 424.44 GiB free]
  Total: 1 [465.50 GiB] / in use: 1 [465.50 GiB] / in no VG: 0 [0   ]

On my nodes, each of which has a single 500GB drive, I've allocated only 20GB to dom0 so I've got over 440GB left free. I like to leave a bit of space unallocated because I never know where I might need it, so I will allocate 400GB even to DRBD and keep the remaining 44GB set aside for future growth. The space you have left and how you want to allocate is an exercise you must settle based on your own needs.

Next, check that the name you will give to the new LV isn't used yet. The lvscan tool will show you what names have been used.

lvscan
  ACTIVE            '/dev/vg_01/lv_root' [39.06 GiB] inherit
  ACTIVE            '/dev/vg_01/lv_swap' [2.00 GiB] inherit

We see from the above output that lv_root and lv_swap are used, so we will use lv_drbd for the DRBD partition. Of course, you can use pretty much any name you want.

Now that we know that we want to create a 400GB logical volume called lv_drbd, we can proceed.

Now to create the logical volume for the DRBD device on each node. The next two commands show what I need to call on my nodes. If you've used different names of have a different amount of free space, be sure to edit the following arguments to match your nodes.

On an-node01:

lvcreate -L 400G -n lv_drbd /dev/vg_01
  Logical volume "lv_drbd" created

If I re-run lvscan now, I will see the new volume:

lvscan
  ACTIVE            '/dev/vg_01/lv_root' [39.06 GiB] inherit
  ACTIVE            '/dev/vg_01/lv_swap' [2.00 GiB] inherit
  ACTIVE            '/dev/vg_01/lv_drbd' [400.00 GiB] inherit

We can now proceed with the DRBD setup!

Configuration Files

DRBD uses a global configuration file, /etc/drbd.d/global_common.conf, and one or more resource files. The resource files need to be created in the /etc/drbd.d/ directory and must have the suffix .res. For this example, we will create a single resource called r0 which we will configure in /etc/drbd.d/r0.res.

Full details on all the drbd.conf configuration file directives and arguments can be found here. Note: That link doesn't show this new configuration format. Please see Novell's link.

These are example files from a working server.

/etc/drbd.d/global_common.conf

vim /etc/drbd.d/global_common.conf
global {
        usage-count yes;
        # minor-count dialog-refresh disable-ip-verification
}
 
common {
        protocol C;
 
        handlers {
                pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
                # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
                # split-brain "/usr/lib/drbd/notify-split-brain.sh root";
                # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
                # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
                # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
        }
 
        startup {
                # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb;
        }
 
        disk {
                # on-io-error fencing use-bmbv no-disk-barrier no-disk-flushes
                # no-disk-drain no-md-flushes max-bio-bvecs
        }
 
        net {
                # snd‐buf-size rcvbuf-size timeout connect-int ping-int ping-timeout max-buffers
                # max-epoch-size ko-count allow-two-primaries cram-hmac-alg shared-secret
                # after-sb-0pri after-sb-1pri after-sb-2pri data-integrity-alg no-tcp-cork
        }
 
        syncer {
                # rate after al-extents use-rle cpu-mask verify-alg csums-alg
                rate 100M;
        }
}

/etc/drbd.d/r0.res

This is the important part. This defines the resource to use, and must reflect the IP addresses and

vim /etc/drbd.d/r0.res
# This is for the initial LV
 
resource r0 {
        device    /dev/drbd0;
 
        net {
                allow-two-primaries;
        }
 
        startup {
                become-primary-on both;
        }
 
        meta-disk       internal;
 
        # The name below must match the output from `uname -n` on the node.
        on xenmaster001.iplink.net {
                # This must be the IP address of the interface on the storage network.
                address         10.249.0.1:7789;
                # This is the underlying partition to use for this resource on this node.
                disk            /dev/vg_01/lv_drbd;
        }
 
        # Repeat as above, but for the other node.
        on xenmaster002.iplink.net {
                address         10.249.0.2:7789;
                disk            /dev/vg_01/lv_drbd;
        }
}

These files must be copied to BOTH nodes and must match before you proceed.

Starting The DRBD Resource

From the rest of this section, pay attention to whether you see

  • Primary
  • Secondary
  • Both

These indicate which node to run the following commands on. There is no functional difference between either node, so just randomly choose one to be Primary and the other will be Secondary. Once you've chosen which is which, be consistent with which node you run the commands on. Of course, if a command block is proceeded by Both, run the following code block on both nodes.

Both

/etc/init.d/drbd start
 

FILL THIS IN.

Setting up CLVM

The goal of DRBD in the cluster is to provide clustered LVM to the nodes. This is done by turning the DRBD partition into an LVM physical volume.

So now we will "stack" LVM by creating a PV on top of the new DRBD partition, /dev/drbd0, that we created in the previous step. Since this new LVM PV will exist on top of the shared DRBD partition, whatever get written to it's logical volumes will be immediately available on either node, regardless of which node actually initiated the write.

This capability is the underlying reason for creating this cluster; Neither machine is truly needed so if one machine dies, anything on top of the DRBD partition will still be available. When the failed machine returns, the surviving node will have a list of what blocks changed while the other node was gone and can use this list to quickly re-sync the other server.

Making LVM Cluster-Aware

Normally, LVM is run on a single server. This means that at any time, the LVM can write data to the underlying drive and not need to worry if any other device might change anything. In clusters, this isn't the case. The other node could try to write to the shared storage, so then nodes need to enable "locking" to prevent the two nodes from trying to work on the same bit of data at the same time.

The process of enabling this locking is known as making LVM "cluster-aware".

Updating '/etc/lvm/lvm.conf'

Note: After CentOS 5.5 is released, this step should no longer be needed. See here for details.

By default, LVM ignores DRBD devices as candidate physical volumes. To enable DRBD, we need to change the filter in /etc/lvm/lvm.conf to include a regular expression that matches the name of our DRBD device. We created our DRBD device as /dev/drbd0, so changing the filter to filter = [ "a|drbd.*|", "a|sd.*|", "r|.*|" ] (accept drbd devices and devices with the sd* "scsi" names, reject everything else) will do this. Edit lvm.conf and change it to match this:

vim /etc/lvm/lvm.conf
    # By default we accept every block device:
    #filter = [ "a/.*/" ]
    filter = [ "a|drbd.*|", "a|sd.*|", "r|.*|" ]

Enabling Cluster Locking

LVM has a built-in tool called lvmconf that can be used to enable LVM locking. This is provided as part of the lvm2-cluster package. Install it and then run:

yum install lvm2-cluster.x86_64

Before you continue, we want to tell clvmd that it should start after the drbd daemon. To do this, edit it's /etc/init.d/clvmd init script and add drbd to it's start block.

vim /etc/init.d/clvmd

Change the header by adding drbd to Required-Start and Required-Stop:

### BEGIN INIT INFO
# Provides: clvmd
# Required-Start: $local_fs drbd
# Required-Stop: $local_fs drbd
# Default-Start:
# Default-Stop: 0 1 6
# Short-Description: Clustered LVM Daemon
### END INIT INFO

Change the start order by removing and re-adding all cluster-related daemons:

Note': This has extra entries to make sure all relevant entries are sorted when re-added. If you do not have some of the daemons listed below installed yet, simply skip them.

chkconfig xend off; chkconfig cman off; chkconfig drbd off; chkconfig clvmd off; chkconfig xendomains off
chkconfig xend on; chkconfig cman on; chkconfig drbd on; chkconfig clvmd on; chkconfig xendomains on

Now to enable cluster awareness in LVM, simply run:

lvmconf --enable-cluster

There won't be any output from that command.

By default, clvmd, the cluster lvm daemon, is stopped and not set to run on boot. Now that we've enabled LVM locking, we need to start it:

/etc/init.d/clvmd status
clvmd is stopped
active volumes: lv00 lv01 lv02

As expected, it is stopped, so lets start it and then use chkconfig to enable it at boot.

/etc/init.d/clvmd start
Stopping clvm:                                             [  OK  ]
Starting clvmd:                                            [  OK  ]
Activating VGs:   3 logical volume(s) in volume group "an-lvm01" now active
                                                           [  OK  ]
chkconfig clvmd on
ls -lah /etc/rc3.d/ |grep clvmd
lrwxrwxrwx  1 root root   15 Mar 16 12:48 S24clvmd -> ../init.d/clvmd

We can see that it is now set to start at position 24.

Creating a new PV using the DRBD Partition

We can now proceed with setting up the new DRBD-based LVM physical volume. Once the PV is created, we can create a new volume group and start allocating space to logical volumes.

Note: As we will be using our DRBD device, and as it is a shared block device, most of the following commands only need to be run on one node. Once the block device changes in any way, those changes will near-instantly appear on the other node. For this reason, unless explicitly stated to do so, only run the following commands on one node.

To setup the DRBD partition as an LVM PV, run pvcreate:

pvcreate /dev/drbd0
  Physical volume "/dev/drbd0" successfully created

Now, on both nodes, check that the new physical volume is visible by using pvdisplay:

pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               san01
  PV Size               465.51 GB / not usable 14.52 MB
  Allocatable           yes 
  PE Size (KByte)       32768
  Total PE              14896
  Free PE               1407
  Allocated PE          13489
  PV UUID               IpySTY-a9BY-31XE-Bxd4-H9sp-OEJG-kP7dtg
 
  "/dev/drbd0" is a new physical volume of "399.99 GB"
  --- NEW Physical volume ---
  PV Name               /dev/drbd0
  VG Name               
  PV Size               399.99 GB
  Allocatable           NO
  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               S6OkVh-NlwQ-BaUn-k5LI-1iTo-pu8V-Uq3qE2

If you see PV Name /dev/drbd0 on both nodes, then your DRBD setup and LVM configuration changes are working perfectly!

Creating a VG on the new PV

Now we need to create the volume group using the vgcreate command:

vgcreate -c y drbd_vg0 /dev/drbd0
  Clustered volume group "drbd_vg0" successfully created

Now we'll check that the new VG is visible on both nodes using vgdisplay:

vgdisplay
  --- Volume group ---
  VG Name               an-lvm01
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               465.50 GB
  PE Size               32.00 MB
  Total PE              14896
  Alloc PE / Size       13489 / 421.53 GB
  Free  PE / Size       1407 / 43.97 GB
  VG UUID               C0kHFA-OTo8-Gshr-3wIw-3Q0I-eT3X-A9Y0NA
 
  --- Volume group ---
  VG Name               drbd_vg0
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  Clustered             yes
  Shared                no
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               399.98 GB
  PE Size               4.00 MB
  Total PE              102396
  Alloc PE / Size       5120 / 20.00 GB
  Free  PE / Size       97276 / 379.98 GB
  VG UUID               TmlQmv-eViK-7Ubr-Dyck-0u86-uEWJ-rDOt9i


If the new VG is visible on both nodes, we are ready to create our first logical volume using the lvcreate tool.

Creating the First Two LVs on the new VG

Now we'll create two simple 20 GiB logical volumes. This first one will be a shared GFS store for source ISOs and the second will be used for our first virtual machine.

lvcreate -L 20G -n iso_store drbd_vg0
lvcreate -L 20G -n vm01 drbd_vg0
  Logical volume "iso_store" created
  Logical volume "vm01" created

As before, we will check that the new logical volume is visible from both nodes by using the lvdisplay command:

lvdisplay
  --- Logical volume ---
  LV Name                /dev/an-lvm02/lv01
  VG Name                an-lvm02
  LV UUID                Dy2MNa-EUxN-9x6f-ovkj-NCpk-nlV2-kr5QBb
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                19.53 GB
  Current LE             625
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
 
  --- Logical volume ---
  LV Name                /dev/an-lvm02/lv00
  VG Name                an-lvm02
  LV UUID                xkBu7j-wtOe-ORr3-68qJ-u0ux-Qif4-stw5SY
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                2.00 GB
  Current LE             64
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
 
  --- Logical volume ---
  LV Name                /dev/an-lvm02/lv02
  VG Name                an-lvm02
  LV UUID                R20GH1-wQKq-WgUR-x1gx-Yzzp-WjND-WHAjEO
  LV Write Access        read/write
  LV Status              available
  # open                 2
  LV Size                400.00 GB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2
 
  --- Logical volume ---
  LV Name                /dev/drbd_vg0/iso_store
  VG Name                drbd_vg0
  LV UUID                svJx35-KDXK-ojD2-UDAA-Ah9t-UgUl-ijekhf
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                20.00 GB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3
 
  --- Logical volume ---
  LV Name                /dev/drbd_vg0/vm01
  VG Name                drbd_vg0
  LV UUID                sceLmK-ZJIp-fN5g-RMaS-j5sq-NuY5-7hIwhP
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                20.00 GB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:4

The last two are the new logical volumes.

Create the Shared GFS FileSystem

GFS is a cluster-aware file system that can be simultaneously mounted on two or more nodes at once. We will use it as a place to store ISOs that we'll use to provision our virtual machines.

The following example is designed for the cluster used in this paper.

  • If you have more than 2 nodes, increase the -j 2 to the number of nodes you want to mount this file system on.
  • If your cluster is named something other than an-cluster (as set in the cluster.conf file), change -t an-cluster:iso_store to match you cluster's name. The iso_store can be whatever you like, but it must be unique in the cluster. I tend to use a name that matches the LV name, but this is my own preference and is not required.

To format the partition run:

mkfs.gfs2 -p lock_dlm -j 2 -t an-cluster:iso_store /dev/drbd_vg0/iso_store

If you are prompted, press y to proceed.

Once the format completes, you can mount /dev/drbd_vg0/iso_store as you would a normal file system.

Both:

To complete the example, lets mount the GFS2 partition we made just now on /shared.

mkdir /shared
mount /dev/drbd_vg0/iso_store /shared

Done!

Growing a GFS2 Partition

To grow a GFS2 partition, you must know where it is mounted. You can not grow an unmounted GFS2 partition, as odd as that may seem at first. Also, you only need to run grow commands from one node. Once completed, all nodes will see and use the new free space automatically.

This requires two steps to complete:

  1. Extend the underlying LVM logical volume
  2. Grow the actual GFS2 partition

Extend the LVM LV

To keep things simple, we'll just use some of the free space we left on our /dev/drbd0 LVM physical volume. If you need to add more storage to your LVM first, please follow the instructions in the article: "Adding Space to an LVM" before proceeding.

Let's add 50GB to our GFS2 logical volume /dev/drbd_vg0/iso_store from the /dev/drbd0 physical volume, which we know is available because we left more than that back when we first setup our LVM. To actually add the space, we need to use the lvextend command:

lvextend -L +50G /dev/drbd_vg0/iso_store /dev/drbd0

Which should return:

  Extending logical volume iso_store to 70.00 GB
  Logical volume iso_store successfully resized

If we run lvdisplay /dev/drbd_vg0/iso_store now, we should see the extra space.

  --- Logical volume ---
  LV Name                /dev/drbd_vg0/iso_store
  VG Name                drbd_vg0
  LV UUID                svJx35-KDXK-ojD2-UDAA-Ah9t-UgUl-ijekhf
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                70.00 GB
  Current LE             17920
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3

You're now ready to proceed.

Grow The GFS2 Partition

This step is pretty simple, but you need to enter the commands exactly. Also, you'll want to do a dry-run first and address any resulting errors before issuing the final gfs2_grow command.

To get the exact name to use when calling gfs2_grow, run the following command:

gfs2_tool df
/shared:
  SB lock proto = "lock_dlm"
  SB lock table = "an-cluster:iso_store"
  SB ondisk format = 1801
  SB multihost format = 1900
  Block size = 4096
  Journals = 2
  Resource Groups = 80
  Mounted lock proto = "lock_dlm"
  Mounted lock table = "an-cluster:iso_store"
  Mounted host data = "jid=1:id=196610:first=0"
  Journal number = 1
  Lock module flags = 0
  Local flocks = FALSE
  Local caching = FALSE
 
  Type           Total Blocks   Used Blocks    Free Blocks    use%           
  ------------------------------------------------------------------------
  data           5242304        1773818        3468486        34%
  inodes         3468580        94             3468486        0%

From this output, we know that GFS2 expects the name "/shared". Even adding something as simple as a trailing slash will not work. The program we will use is called gfs2_grow with the -T switch to run the command as a test to work out possible errors.

For example, if you added the trailing slash, this is the kind of error you would see:

Bad command:

gfs_grow -T /shared/
GFS Filesystem /shared/ not found

Once we get it right, it will look like this:

gfs_grow -T /shared
(Test mode--File system will not be changed)
FS: Mount Point: /shared
FS: Device:      /dev/mapper/drbd_vg0-iso_store
FS: Size:        5242878 (0x4ffffe)
FS: RG size:     65535 (0xffff)
DEV: Size:       18350080 (0x1180000)
The file system grew by 51200MB.
gfs2_grow complete.

This looks good! We're now ready to re-run the command without the -T switch:

gfs_grow /shared
FS: Mount Point: /shared
FS: Device:      /dev/mapper/drbd_vg0-iso_store
FS: Size:        5242878 (0x4ffffe)
FS: RG size:     65535 (0xffff)
DEV: Size:       18350080 (0x1180000)
The file system grew by 51200MB.
gfs2_grow complete.

You can check that the new space is available on both nodes now using a simple call like df -h.


 

Any questions, feedback, advice, complaints or meanderings are welcome.
Us: Alteeve's Niche! Support: Mailing List IRC: #clusterlabs on Freenode   © Alteeve's Niche! Inc. 1997-2019
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.
Personal tools
Namespaces

Variants
Actions
Navigation
projects
Toolbox