Getting members of a
Red Hat Cluster Suite to communicate and get quorum can be difficult. This post documents how I verified that multicast was working as I step towards setting up a cluster.
After you start your cluster see which multicast address was assigned:
[root@vserver0 ~]# grep "default multicast" /var/log/messages
Dec 27 16:51:51 vserver0 openais[6953]: [MAIN ] Using default multicast address of 239.192.104.1
Dec 27 18:26:33 vserver0 openais[3664]: [MAIN ] Using default multicast address of 239.192.104.1
Dec 28 04:25:28 vserver0 openais[13028]: [MAIN ] Using default multicast address of 239.192.104.1
Dec 30 14:02:35 vserver0 openais[9533]: [MAIN ] Using default multicast address of 239.192.104.1
[root@vserver0 ~]#
I also see that each node is listening on the same address:
[root@vserver0 ~]# netstat -an | grep 239
udp 0 0 239.192.104.1:5405 0.0.0.0:*
[root@vserver0 ~]#
[root@vserver1 ~]# netstat -an | grep 239
udp 0 0 239.192.104.1:5405 0.0.0.0:*
[root@vserver1 ~]#
[root@vserver2 ~]# netstat -an | grep 239
udp 0 0 239.192.104.1:5405 0.0.0.0:*
[root@vserver2 ~]#
The cluster documentation talks about configuring multicast and implies that you might have to enable it on your router. You can verify if your router is passing multicast packets between nodes with tcpdump and iperf as described in taosecurity.blogspot.com. I can reproduce the results described in the taosecurity blog in my cluster as so:
wget ftp://ftp.pbone.net/mirror/ftp.freshrpms.net/pub/freshrpms/pub/dag/redhat/el5/en/x86_64/dries/RPMS/iperf-
2.0.4-1.el5.rf.x86_64.rpm
rpm -ivh iperf-2.0.4-1.el5.rf.x86_64.rpm
I set up vserver0 as an iperf server listening on multicast address 239.192.104.1:
[root@vserver0 ~]# iperf -s -u -B 239.192.104.1 -i 1
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 239.192.104.1
Joining multicast group 239.192.104.1
Receiving 1470 byte datagrams
UDP buffer size: 126 KByte (default)
------------------------------------------------------------
Now I generate multicast traffic from vserver1.
[root@vserver1 ~]# iperf -c 239.192.104.1 -u -T 32 -t 3 -i 1
------------------------------------------------------------
Client connecting to 239.192.104.1, UDP port 5001
Sending 1470 byte datagrams
Setting multicast TTL to 32
UDP buffer size: 126 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.129 port 48770 connected with 239.192.104.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 129 KBytes 1.06 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 1.0- 2.0 sec 128 KBytes 1.05 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 2.0- 3.0 sec 128 KBytes 1.05 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 3.0 sec 386 KBytes 1.05 Mbits/sec
[ 3] Sent 269 datagrams
[root@vserver1 ~]#
Here is what vserver0 sees:
------------------------------------------------------------
[ 3] local 239.192.104.1 port 5001 connected with 192.168.1.129 port 48770
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0- 1.0 sec 134 KBytes 1.09 Mbits/sec 1.109 ms 0/ 93 (0%)
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 1.0- 2.0 sec 128 KBytes 1.05 Mbits/sec 0.136 ms 0/ 89 (0%)
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0- 3.0 sec 386 KBytes 1.07 Mbits/sec 0.957 ms 0/ 269 (0%)
The traffic looks like this:
[root@vserver0 ~]# tcpdump -n -i eth0 -s 1515 udp | grep 239.192.104.1 > muticast.txt
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 1515 bytes
343 packets captured
343 packets received by filter
0 packets dropped by kernel
[root@vserver0 ~]#
[root@vserver0 ~]# wc -l muticast.txt
283 muticast.txt
[root@vserver0 ~]# head -5 muticast.txt
15:09:23.364851 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:23.760086 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:24.156427 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:24.555871 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:24.956558 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
[root@vserver0 ~]#
So, I seem to have generated multicast traffic and ensured that a member of the multicast group actually received it.
According to RedHat (http://sources.redhat.com/cluster/doc/usage.txt): "CMAN can be configured to use multicast instead of broadcast (broadcast is used by default if no multicast parameters are given)." They also go on to say that you must enable it in /etc/cluster/cluster.conf:
<cman>
<multicast addr="224.0.0.1"/>
</cman>
<clusternode name="nd1">
<multicast addr="224.0.0.1" interface="eth0"/>
</clusternode>