Multicast and Red Hat Cluster Suite

Getting members of a Red Hat Cluster Suite to communicate and get quorum can be difficult. This post documents how I verified that multicast was working as I step towards setting up a cluster.

After you start your cluster see which multicast address was assigned:

[root@vserver0 ~]# grep "default multicast" /var/log/messages
Dec 27 16:51:51 vserver0 openais[6953]: [MAIN ] Using default multicast address of 239.192.104.1 
Dec 27 18:26:33 vserver0 openais[3664]: [MAIN ] Using default multicast address of 239.192.104.1 
Dec 28 04:25:28 vserver0 openais[13028]: [MAIN ] Using default multicast address of 239.192.104.1 
Dec 30 14:02:35 vserver0 openais[9533]: [MAIN ] Using default multicast address of 239.192.104.1 
[root@vserver0 ~]# 

I also see that each node is listening on the same address:

[root@vserver0 ~]# netstat -an | grep 239
udp        0      0 239.192.104.1:5405          0.0.0.0:*
[root@vserver0 ~]# 

[root@vserver1 ~]# netstat -an | grep 239
udp        0      0 239.192.104.1:5405          0.0.0.0:*
[root@vserver1 ~]# 

[root@vserver2 ~]# netstat -an | grep 239
udp        0      0 239.192.104.1:5405          0.0.0.0:*
[root@vserver2 ~]# 

The cluster documentation talks about configuring multicast and implies that you might have to enable it on your router. You can verify if your router is passing multicast packets between nodes with tcpdump and iperf as described in taosecurity.blogspot.com. I can reproduce the results described in the taosecurity blog in my cluster as so:

wget ftp://ftp.pbone.net/mirror/ftp.freshrpms.net/pub/freshrpms/pub/dag/redhat/el5/en/x86_64/dries/RPMS/iperf-
2.0.4-1.el5.rf.x86_64.rpm
rpm -ivh iperf-2.0.4-1.el5.rf.x86_64.rpm 

I set up vserver0 as an iperf server listening on multicast address 239.192.104.1:

[root@vserver0 ~]# iperf -s -u -B 239.192.104.1 -i 1
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 239.192.104.1
Joining multicast group  239.192.104.1
Receiving 1470 byte datagrams
UDP buffer size:   126 KByte (default)
------------------------------------------------------------

Now I generate multicast traffic from vserver1.

[root@vserver1 ~]# iperf -c 239.192.104.1 -u -T 32 -t 3 -i 1
------------------------------------------------------------
Client connecting to 239.192.104.1, UDP port 5001
Sending 1470 byte datagrams
Setting multicast TTL to 32
UDP buffer size:   126 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.129 port 48770 connected with 239.192.104.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec    129 KBytes  1.06 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec    128 KBytes  1.05 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec    128 KBytes  1.05 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 3.0 sec    386 KBytes  1.05 Mbits/sec
[  3] Sent 269 datagrams
[root@vserver1 ~]# 

Here is what vserver0 sees:

------------------------------------------------------------
[  3] local 239.192.104.1 port 5001 connected with 192.168.1.129 port 48770
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0- 1.0 sec    134 KBytes  1.09 Mbits/sec  1.109 ms    0/   93 (0%)
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  1.0- 2.0 sec    128 KBytes  1.05 Mbits/sec  0.136 ms    0/   89 (0%)
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0- 3.0 sec    386 KBytes  1.07 Mbits/sec  0.957 ms    0/  269 (0%)

The traffic looks like this:

[root@vserver0 ~]# tcpdump -n -i eth0 -s 1515 udp | grep  239.192.104.1 > muticast.txt
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 1515 bytes
343 packets captured
343 packets received by filter
0 packets dropped by kernel

[root@vserver0 ~]# 

[root@vserver0 ~]# wc -l muticast.txt 
283 muticast.txt
[root@vserver0 ~]# head -5 muticast.txt 
15:09:23.364851 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:23.760086 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:24.156427 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:24.555871 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:24.956558 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
[root@vserver0 ~]# 

So, I seem to have generated multicast traffic and ensured that a member of the multicast group actually received it.

According to RedHat (http://sources.redhat.com/cluster/doc/usage.txt): "CMAN can be configured to use multicast instead of broadcast (broadcast is used by default if no multicast parameters are given)." They also go on to say that you must enable it in /etc/cluster/cluster.conf:

<cman>
    <multicast addr="224.0.0.1"/>
</cman>
<clusternode name="nd1">
    <multicast addr="224.0.0.1" interface="eth0"/>
</clusternode>