High Availability #1

I played with Linux High Availability solutions - on Debian/Ubuntu.

Including Red Hat Cluster Suite, Heartbeat, and the relatively newcomer player, probably the successor, the Pacemaker/Corosync/OpenAIS with heartbeat commons (for scripting resources).

I had the best impressions about Pacemaker. Its configuration and manageability is far the bes in my humble opinion, and therefore should provide the upmost uptime counters.

I started to play with the official ubuntu ha repository on two ubuntu hardy virtualbox virtual machine named by c01 and c02 with will be the part of the cluster service:

root@c01:~# cat /etc/apt/sources.list.d/ubuntu-ha.list
deb http://ppa.launchpad.net/ubuntu-ha/ppa/ubuntu hardy main
deb-src http://ppa.launchpad.net/ubuntu-ha/ppa/ubuntu hardy main

I have put it to the file above to separate it from the other debian sources.

Then I installed:

root@c01:~# apt-get install pacemaker-openais
root@c02:~# apt-get install pacemaker-openais

As you can see, I installed it on both cluster nodes.

Since openais and corosync packages has been split up their main configuration

root@c01:~# ls -la /etc/corosync/authkey /etc/corosync/corosync.conf
lrwxrwxrwx 1 root root 16 2009-09-21 09:16 /etc/corosync/authkey -> /etc/ais/authkey
lrwxrwxrwx 1 root root 21 2009-09-21 09:16 /etc/corosync/corosync.conf -> /etc/ais/openais.conf

And note, that the authkey has been created by corosync-keygen after I linked the /dev/random to point to /dev/urandom unless keygen won't work if you have no /dev/random.

In this scenario I have set up two ethernet links between the two (virtual) boxes since the HA cluster service stringly recommends two independent media for communication, in this example, 192.168.0.0/24 and 192.168.1.0/24.

So my corosync/openais config file /etc/corosync/corosync.conf file looks like this:

totem {
version: 2
# How long before declaring a token lost (ms)
token: 10000
# How many token retransmits before forming a new configuration
token_retransmits_before_loss_const: 20
# How long to wait for join messages in the membership protocol (ms)
join: 60
# How long to wait for consensus to be achieved before starting a new round of membership configuration (ms)
consensus: 4800
# Turn off the virtual synchrony filter
vsftype: none
# Number of messages that may be sent by one processor on receipt of the token
max_messages: 20
# Disable encryption
secauth: off
# How many threads to use for encryption/decryption
threads: 0

# Limit generated nodeids to 31-bits (positive signed integers)
clear_node_high_bit: yes

# Optionally assign a fixed node id (integer)
# nodeid: 1234
rrp_mode: passive
interface {
ringnumber: 0
bindnetaddr: 192.168.0.0
#mcastaddr: 226.94.1.1
mcastaddr: 225.0.0.1
mcastport: 5405
}
interface {
ringnumber: 1
bindnetaddr: 192.168.1.0
mcastaddr: 225.0.1.1
mcastport: 5405
}
}

logging {
fileline: off
to_syslog: yes
to_stderr: no
syslog_facility: daemon
debug: on
timestamp: on
}

amf {
mode: disabled
}
service {
# Load the Pacemaker Cluster Resource Manager
name: pacemaker
ver: 0
use_mgmtd: yes
}
aisexec {
user: root
group: root
}

Note that the service is configured to use pacemaker as CRM (Cluster Resource Manager) and I'll be using the mcast settings for multicast, and I have two interface subconfig since as I mentioned earlier, in this scenario, I have two lan connected boxes.

Then let's stop/start the services on both nodes by:

/etc/init.d/corosync stop
/etc/init.d/corosync start

And check wether we can use the crm command to communicate with the cluster. If you see similar, then we're good:

root@c01:~# crm status

============
Last updated: Sat Sep 26 18:47:41 2009
Stack: openais
Current DC: c01 - partition with quorum
Version: 1.0.4-2609e060ce0c516c95ae31f44a10fed0202abfb6
2 Nodes configured, 2 expected votes
4 Resources configured.
============

Online: [ c01 c02 ]

Watch out, we see two online machines.

Now let's set up a Virtual IPs (abbreviated as VIPs) to the cluster:

root@c01:~# crm
crm(live)# configure
crm(live)configure#
primitive vip ocf:heartbeat:IPaddr \
params ip="10.30.49.254" \
op monitor interval="10s" \
meta target-role="Started"
primitive vip2 ocf:heartbeat:IPaddr2 \
params ip="10.30.49.253" nic="eth0" cidr_netmask="16" \
meta target-role="Started" \
op monitor interval="10s"
crm(live)configure# commit

Where at the first example, named as vip we configure to use the heartbeat kind of Open Cluster Format scripts, located at the physical file system: /usr/lib/ocf/resource.d/heartbeat/IPaddr with will be monitored in every 10th seconds and will start up immediately after we commit the config changes. For the vip2 which is a slightly more advanced script (located at /usr/lib/ocf/resource.d/heartbeat/IPaddr2 ) will use a well defined netmask (cidr) and interface name, where this IP will be located (at the first unallocated one eg. eth0:1).

In this example, as well, since we're using two nodes, the quorum will fail, so there are two options to avoid this pitfall:

crm(live)configure# property no-quorum-policy="ignore"

or

crm(live)configure# property expected-quorum-votes="1"

Let's see how it will migrate (by manual failover) from one cluster node to another one:

crm(live)configure# cd
crm(live)# status


============
Last updated: Sat Sep 26 18:58:48 2009
Stack: openais
Current DC: c01 - partition with quorum
Version: 1.0.4-2609e060ce0c516c95ae31f44a10fed0202abfb6
2 Nodes configured, 2 expected votes
4 Resources configured.
============

Online: [ c01 c02 ]

vip (ocf::heartbeat:IPaddr): Started c01
vip2 (ocf::heartbeat:IPaddr2): Started c01

This shows, that the resource vip and vip2 started successfully, you can check the interface list:

root@c01:~# ip addr
[..]
5: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 08:00:27:63:61:9a brd ff:ff:ff:ff:ff:ff
inet 10.30.49.1/16 brd 10.30.255.255 scope global eth0
inet 10.30.49.253/16 brd 10.30.255.255 scope global secondary eth0
inet 10.30.49.254/16 brd 10.30.255.255 scope global secondary eth0:0

Now let's see with the manual failover!

crm(live)# resource migrate vip c02
crm(live)# status
[..]
Online: [ c01 c02 ]

vip (ocf::heartbeat:IPaddr): Started c02
vip2 (ocf::heartbeat:IPaddr2): Started c01

Voilà, it works, you can see that vip went to c02. You can check it on its own interface list.

Now, let's see how to bond together resources, in this example the VIPs:

crm(live)configure# colocation vip-with-vip2 inf: vip vip2
crm(live)configure# commit
crm(live)configure# cd
crm(live)# status
[..]
vip (ocf::heartbeat:IPaddr): Started c01
vip2 (ocf::heartbeat:IPaddr2): Started c01

From now, if you migrate one of them to another place, the other vip will go with it as well.