Dual Primary DRBD on CentOS 6: GFS2 & Pacemaker

This guide describes how to create a pair of redundant file servers using DRBD for replication, RedHat GFS2 (Global File System), and Pacemaker for cluster management. In this case we are also using RackSpace Cloud Servers and associated OpenStack features, so we will use the nova client to create the networks, servers, and storage before logging on to finish the configuration.

Once completed you will have dual primary DRBD configuration which allows reads and writes to both nodes at the same time – enabling load balanced NFS for example.

Network Architecture

This is a rough diagram of the network architecture. Each file server is attache to a block storage device, and the file servers will synchronize their disks using DRBD on a private storage network. On a different internal network the NFS servers are fronted by a load balancer (or multiple) which is then attached to your servers. Those servers are then fronted by load balancers, firewalls, etc, or serve content to your internal network. You could also have clients connect directly to the NFS share exposed on the load balancer.

Create RackSpace Cloud Network

First we will need to create a private Cloud Network that we can dedicate to replication. We will call the network “storage-replication” and give it 192.168.1.x. You will need to capture the id that is returned so that you can attach it to a Cloud Server.

nova network-create storage-replication 192.168.1.0/24

+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| cidr     | 192.168.1.0/24                       |
| id       | 7c99ba74-c28c-4c52-9c5a-xxxxxxxxxxxx |
| label    | storage-replication                  |
+----------+--------------------------------------+

Create RackSpace Cloud Servers

Now that we have a network setup, let’s create two servers and assign them to it. You will need to specify an --image (the one below is for CentOS 6.5 PVHVM) however you can get a list of them using nova image-list. We are going to use a 4GB standard Cloud Server, but again you can use a different --flavor value from the results of nova flavor-list. The --nic net-id is where you place your private network id, and the --file option lets us insert an SSH key so we don’t have to bother with the password after the server is built. We put the script to sleep for 30 seconds to encourage the VM to be provisioned on a different hypervisor to provide additional redundancy. When all is said and done we have two new cloud servers named fileserver-1 and fileserver-2, and once again you should capture the id‘s for later use.

for i in 1 2; \
	do nova boot \
	--image 41e59c5f-530b-423c-86ec-13b23de49288 \
	--flavor 5 \
	--nic net-id=7c99ba74-c28c-4c52-9c5a-xxxxxxxxxxxx \
	--file /root/.ssh/authorized_keys=/Users/justinsilver/.ssh/id_dsa.pub \
	fileserver-${i}; \
	sleep 30; \
done

+------------------------+--------------------------------------+
| Property               | Value                                |
+------------------------+--------------------------------------+
| status                 | BUILD                                |
| updated                | 2014-02-26T08:16:59Z                 |
| OS-EXT-STS:task_state  | scheduling                           |
| key_name               | None                                 |
| image                  | CentOS 6.5 (PVHVM)                   |
| hostId                 |                                      |
| OS-EXT-STS:vm_state    | building                             |
| flavor                 | 4GB Standard Instance                |
| id                     | 69f66617-44e2-4cbc-8a34-xxxxxxxxxxxx |
| user_id                | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx     |
| name                   | fileserver-1                         |
| adminPass              | xxxxxxxxxxxx                         |
| tenant_id              | xxxxxx                               |
| created                | 2014-02-26T08:16:58Z                 |
| OS-DCF:diskConfig      | MANUAL                               |
| accessIPv4             |                                      |
| accessIPv6             |                                      |
| progress               | 0                                    |
| OS-EXT-STS:power_state | 0                                    |
| config_drive           |                                      |
| metadata               | {}                                   |
+------------------------+--------------------------------------+
+------------------------+--------------------------------------+
| Property               | Value                                |
+------------------------+--------------------------------------+
| status                 | BUILD                                |
| updated                | 2014-02-26T08:17:32Z                 |
| OS-EXT-STS:task_state  | scheduling                           |
| key_name               | None                                 |
| image                  | CentOS 6.5 (PVHVM)                   |
| hostId                 |                                      |
| OS-EXT-STS:vm_state    | building                             |
| flavor                 | 4GB Standard Instance                |
| id                     | 5f0a5c3e-0dfa-4583-bddc-xxxxxxxxxxxx |
| user_id                | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx     |
| name                   | fileserver-2                         |
| adminPass              | xxxxxxxxxxxx                         |
| tenant_id              | xxxxxx                               |
| created                | 2014-02-26T08:17:31Z                 |
| OS-DCF:diskConfig      | MANUAL                               |
| accessIPv4             |                                      |
| accessIPv6             |                                      |
| progress               | 0                                    |
| OS-EXT-STS:power_state | 0                                    |
| config_drive           |                                      |
| metadata               | {}                                   |
+------------------------+--------------------------------------+

Create Cloud Block Storage

Next we can create a pair of Cloud Block Storage devices to attach to our newly created Cloud Servers. We are going to use SSD instead of SATA for improved read performance and size it to 200GB – you can choose whatever value you need for your purposes – and remember that since we will be using LVM we can resize later though some care should be take up front. Again pay attention to the id‘s that are returned so that we can attach them to the Cloud Servers.

for i in 1 2; \
	do nova volume-create \
	--display-name fileserver-${i} \
	--volume-type SSD 200; \
done;

+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| status              | available                            |
| display_name        | fileserver-1                         |
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| created_at          | 2014-02-26T07:11:37.000000           |
| display_description | None                                 |
| volume_type         | SSD                                  |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| size                | 200                                  |
| id                  | db75fdd8-da9f-48df-861a-xxxxxxxxxxxx |
| metadata            | {}                                   |
+---------------------+--------------------------------------+
+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| status              | available                            |
| display_name        | fileserver-2                         |
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| created_at          | 2014-02-26T07:11:40.000000           |
| display_description | None                                 |
| volume_type         | SSD                                  |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| size                | 200                                  |
| id                  | 28a2905e-49e0-426c-8b33-xxxxxxxxxxxx |
| metadata            | {}                                   |
+---------------------+--------------------------------------+

Now use the ID’s of the Cloud Servers and Cloud Block Storage to attach the storage device to the appropriate server.

nova volume-attach 69f66617-44e2-4cbc-8a34-xxxxxxxxxxxx db75fdd8-da9f-48df-861a-xxxxxxxxxxxx /dev/xvdb && \
nova volume-attach 5f0a5c3e-0dfa-4583-bddc-xxxxxxxxxxxx 28a2905e-49e0-426c-8b33-xxxxxxxxxxxx /dev/xvdb

+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/xvdb                            |
| serverId | 69f66617-44e2-4cbc-8a34-xxxxxxxxxxxx |
| id       | db75fdd8-da9f-48df-861a-xxxxxxxxxxxx |
| volumeId | db75fdd8-da9f-48df-861a-xxxxxxxxxxxx |
+----------+--------------------------------------+
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/xvdb                            |
| serverId | 5f0a5c3e-0dfa-4583-bddc-xxxxxxxxxxxx |
| id       | 28a2905e-49e0-426c-8b33-xxxxxxxxxxxx |
| volumeId | 28a2905e-49e0-426c-8b33-xxxxxxxxxxxx |
+----------+--------------------------------------+

DRBD Cloud Server Configuration

Once the servers are provisioned, connect to them both to continue the configuration. Since this is a dual primary DRBD configuration you will eventually be able to read and write to both, however for the initial sync we will use fileserver-1 as the primary.

As these are brand new servers it’s recommended to fetch all updates, allow traffic on your private network via iptables and reboot in case there were any kernel patches. Make sure to save your iptables changes or they will be lost after rebooting.

DRBD is in the EL repository which is not included by default but it can be fetched via an RPM. Once this RPM has been installed you can use yum to install both the DRBD driver and utils.

rpm -ivh http://elrepo.org/elrepo-release-6-5.el6.elrepo.noarch.rpm
yum -y update
iptables -A INPUT -m iprange --src-range 192.168.1.1-192.168.1.255 -j ACCEPT
service iptables save
yum install -y kmod-drbd84 drbd84-utils
modprobe drbd

Protect yourself from DNS outages, lookups, and the pain of using the wrong interface by assigning a hostname in the hosts file of each server. By entering the hostnames here we can ensure that corosync and cman use the correct interface for cluster management.

192.168.1.1 fileserver-1
192.168.1.2 fileserver-2

Create a partition on both hosts of the same size. You can use the full Cloud Block Storage or a piece of it (hint: 1GB will sync a lot faster if you’re just doing a test) but for the rest of this guide make sure to assign it to /dev/xvdb1.

In this guide we are going to use LVM to manage the volume so that we can shrink/grow it as needed. To do this you will need to press “T” after the partition is created and then choose “8e” for “Linux LVM”. Press “w” to save and exit or just “q” to exit and lose your changes.

fdisk /dev/xvdb

Disk /dev/xvdb: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x1568dcd9

	Device Boot      Start         End      Blocks   Id  System
/dev/xvdb1               1       26108   209712478+  8e  Linux LVM

pvcreate /dev/xvdb1
	Physical volume "/dev/xvdb1" successfully created
vgcreate fileserver /dev/xvdb1
	Volume group "fileserver" successfully created
lvcreate --name r0 --size 50G fileserver
	Logical volume "r0" created

Now we can create a DRBD resource called r0 using a configuration file called /etc/drbd.d/r0.res. Copy this file onto both nodes and make sure it is the same including whitespace. For the “on” directive you need to use the actual hostname of the server that was entered in /etc/hosts.

resource r0 {
        protocol C;
        startup {
        	become-primary-on both;
        }
        disk {
                fencing resource-and-stonith;
        }
        handlers {
                fence-peer              "/usr/lib/drbd/crm-fence-peer.sh";
                after-resync-target     "/usr/lib/drbd/crm-unfence-peer.sh";
        }
        net {
                cram-hmac-alg sha1;
                shared-secret "DRBD Super Secret Password";
                timeout 180;
                ping-int 3;
                ping-timeout 9;
                allow-two-primaries;
                after-sb-0pri discard-zero-changes;
                after-sb-1pri discard-secondary;
                after-sb-2pri disconnect;
        }
        on fileserver-1 {
                device /dev/drbd1;
                disk /dev/fileserver/r0;
                address 192.168.1.1:7788;
                meta-disk internal;
        }
        on fileserver-2 {
                device /dev/drbd1;
                disk /dev/fileserver/r0;
                address 192.168.1.2:7788;
                meta-disk internal;
        }
}

We are now ready to start the DRBD service, but we want to make sure it doesn’t start on boot as it will be managed by pacemaker.

chkconfig drbd off
service drbd start
drbdadm create-md r0

Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success

From fileserver-1 only run the following command to force synchronization for its disk. Check the status and wait until it is 100% complete before continuing.

drbdadm primary --force r0
service drbd status

drbd driver loaded OK; device status:
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by phil@Build64R6, 2013-10-14 15:33:06
m:res          cs          ro               ds                     p  mounted  fstype
...            sync'ed:    18.3%            (33656/41188)M
1:r0           SyncSource  Primary/Secondary UpToDate/Inconsistent  C

Now we are ready to promote both DRBD nodes to primary status. You can check the status again and once the sync has completed you are ready to move on.

drbdadm adjust r0
drbdadm primary r0
service drbd status

drbd driver loaded OK; device status:
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by phil@Build64R6, 2013-10-14 15:33:06
m:res          cs         ro               ds                 p  mounted  fstype
1:r0           Connected  Primary/Primary  UpToDate/UpToDate  C

Create Clustered Filesystem

Now that DRBD is configured we need to install a clustered filesystem – GFS2 in this case. It handles the details of your filesystem being written and read on multiple nodes as the same time without getting trashed. Using EXT3/4 for example just won’t work properly.

yum -y install gfs2-utils cman pacemaker pacemaker-cli fence-agents resource-agents openais

We are now going to create a /etc/cluster/cluster.conf configuration for a cluster named “pacemaker” (note the max length is 15 characters for corosync.conf). Because DRBD only supports two nodes we can’t reach a quorum in the traditional sense – we must set the special two_node="1" parameter that lets the cluster reach quorum even if a node has failed. Specify that we want to use pacemaker for fencing and we should be good to go. Note the fileserver-1 and fileserver-2 from /etc/hosts.

<?xml version="1.0"?>
<cluster config_version="1" name="pacemaker">
    <cman two_node="1" expected_votes="1"/>
    <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
    <clusternodes>
        <clusternode name="fileserver-1" nodeid="1" votes="1">
            <fence>
                <method name="pcmk-redirect">
                    <device name="pcmk" port="fileserver-1"/>
                </method>
            </fence>
        </clusternode>
        <clusternode name="fileserver-2" nodeid="2" votes="1">
            <fence>
                <method name="pcmk-redirect">
                    <device name="pcmk" port="fileserver-2"/>
                </method>
            </fence>
        </clusternode>
    </clusternodes>
    <fencedevices>
        <fencedevice name="pcmk" agent="fence_pcmk"/>
    </fencedevices>
</cluster>

This is a fairly standard /etc/corosync/corosync.conf configuration however I had to remove the interface > mcastaddress parameter and replace it with broadcast: yes to get the nodes to see each other. We want to bind to 192.168.1.0.

totem {
    version: 2
    token: 3000
    token_retransmits_before_loss_const: 10
    join: 60
    consensus: 3600
    vsftype: none
    max_messages: 20
    clear_node_high_bit: yes
    secauth: off
    threads: 0
    rrp_mode: none
    interface {
        ringnumber: 0
        bindnetaddr: 192.168.1.0
        broadcast: yes
        mcastport: 5405
    }
}
amf {
    mode: disabled
}
service {
    ver:       1
    name:      pacemaker
}
aisexec {
    user:   root
    group:  root
}
logging {
    fileline: off
    to_stderr: yes
    to_logfile: no
    to_syslog: yes
    syslog_facility: daemon
    debug: off
    timestamp: on
    logger_subsys {
        subsys: AMF
        debug: off
        tags: enter|leave|trace1|trace2|trace3|trace4|trace6
    }
}

Starting pacemaker on one node at a time will allow you to monitor the status with cman_tool status and cman_tool nodes. Once everything is started up we can create the GFS2 filesystem on the DRBD resource. Make sure to specify -p lock_dlm for locking management and one journal per node with -j 2. Mounting the devices with -o noatime,nodiratime will give us a performance boost as we don’t really care about access times.

chkconfig pacemaker on
service pacemaker start
mkfs.gfs2 -t pacemaker:storage -p lock_dlm -j 2 /dev/drbd1
mkdir -p /mnt/storage
mount -t gfs2 -o noatime,nodiratime /dev/drbd1 /mnt/storage

STONITH: Shoot The Other Node In The Head

STONITH is not a suggestion when dealing with Dual Primaries, it is an absolute requirement.

– LINBIT support

So now that we have everything up and running this is the not so small task of making sure we are able to deliver uncorrupted data. What happens when the synchronization is lost between nodes? How can we ensure that the misbehaving node is taken out of service? This is what fencing and STONITH accomplish – making sure that data isn’t written to the good nodes until the bad node is out of service, ensuring data integrity.

Install CRM Shell

We are going to use the CRM Shell to control Pacemaker, and we can get it from yum if we install the proper repository.

cd /etc/yum.repos.d/
wget http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/network:ha-clustering:Stable.repo
yum -y install crmsh graphviz

crm configure
property stonith-enabled="false"
commit
exit

crm configure
primitive p_drbd_r1 ocf:linbit:drbd \
	params drbd_resource="drbd1" \
	op start interval="0" timeout="240" \
	op stop interval="0" timeout="100" \
	op monitor interval="29s" role="Master" \
	op monitor interval="31s" role="Slave"
ms ms_drbd_r1 p_drbd_r1 \
	meta master-max="2" \
	master-node-max="1" \
	clone-max="2" \
	clone-node-max="1" \
	notify="true"
primitive p_fs_r1 ocf:heartbeat:Filesystem \
	params device="/dev/drbd1" \
	directory="/mnt/storage" \
	fstype="gfs2" \
	op start interval="0" timeout="60" \
	op stop interval="0" timeout="60" \
	op monitor interval="60" timeout="40"
clone cl_fs_r1 p_fs_r1 meta interleave="true"
colocation co_fs_with_drbd inf: cl_fs_r1 ms_drbd_r1:Master
order o_drbd_before_fs inf: ms_drbd_r1:promote cl_fs_r1
primitive stonith_fence_virsh_fileserver1 stonith:fence_virsh \
	params action="reboot" ipaddr="192.168.1.1" \
	login="root" identity_file="/root/.ssh/id_rsa.pub" \
	port="fileserver-1"
primitive stonith_fence_virsh_fileserver2 stonith:fence_virsh \
	params action="reboot" ipaddr="192.168.1.2" \
	login="root" identity_file="/root/.ssh/id_rsa.pub" \
	port="fileserver-2"
location l_stonith_fence_virsh_machine1_noton_fileserver1 \
	stonith_fence_virsh_fileserver1 -inf: fileserver-1
location l_stonith_fence_virsh_machine1_noton_fileserver2 \
	stonith_fence_virsh_fileserver2 -inf: fileserver-2
property stonith-enabled="true"
commit
exit

Validate the Cluster

Check to make sure that everything is good to go. Now is a good time to make sure things are actually failing over properly before moving to production.

[root@fileserver-1]# service drbd status
drbd driver loaded OK; device status:
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by phil@Build64R6, 2013-10-14 15:33:06
m:res    cs         ro               ds                 p  mounted           fstype
1:r0     Connected  Primary/Primary  UpToDate/UpToDate  C  /mnt/storage      gfs2

[root@fileserver-1]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   1   M    152   2014-02-27 22:23:29  fileserver-1
   2   M     76   2014-02-27 22:23:29  fileserver-2

[root@fileserver-2]# cman_tool status
Version: 6.2.0
Config Version: 3
Cluster Name: pacemaker
Cluster Id: 62570
Cluster Member: Yes
Cluster Generation: 152
Membership state: Cluster-Member
Nodes: 2
Expected votes: 1
Total votes: 2
Node votes: 1
Quorum: 1
Active subsystems: 8
Flags: 2node
Ports Bound: 0
Node name: fileserver-1
Node ID: 1
Multicast addresses: 239.192.244.95
Node addresses: 192.168.1.1

[root@fileserver-1]# fence_tool ls
fence domain
member count  2
victim count  0
victim now    0
master nodeid 1
wait state    none
members       1 2

NFS Server

yum -y install nfs*
service rpcbind start
chkconfig rpcbind on
service nfs start
chkconfig nfs on

/mnt/storage     192.168.1.0/24(rw,async,no_root_squash,no_all_squash,no_subtree_check)

• /mnt/storage – NFS mount point
• 192.168.1.0/24 – IP range of clients
• rw – read / write access
• async – faster asynchronous writes
• no_root_squash – allow root
• no_all_squash – allow user
• no_subtree_check – increases performance but lowers security by preventing parent directory permissions to be checked when accessing shares.

mkdir -p /mnt/storage
mount -t nfs -o rsize=32768,wsize=32768,noatime,nodiratime,async fileserver-1:/mnt/storage /mnt/storage/