tcpdump

If you're debugging network services between two hosts don't forget to let tcpdump help you.

E.g. if host1 can't SSH to host2 and you think an external firewall is blocking you, try this:

1. Have host2 display TCP info on port 22 and display only results containing host1:

host2$ tcpdump -i eth0 tcp port 22 | grep host1

2. Try to SSH from host1 to host2.

If an external firewall is blocking them you should see nothing from the command above. However, if you see something like the following, then the external firewall isn't blocking you and it's an issue between the two hosts:

host2$ tcpdump -i eth0 tcp port 22 | grep host1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
14:08:50.412007 IP host1.domain.tld.54403 >
host2.domain.tld.ssh: S 2707949880:2707949880(0) win 5840 
14:08:53.411702 IP host1.domain.tld.54403 >
host2.domain.tld.ssh: S 2707949880:2707949880(0) win 5840 
14:08:59.411422 IP host1.domain.tld.54403 >
host2.domain.tld.ssh: S 2707949880:2707949880(0) win 5840 

In the case above host2 is unable to get back to host1 to complete the TCP hand shake. In this case you could try reaching host1 from host2 and debug the resulting issue (e.g. broken netmask on host2).

A healthy looking tcpdump from host2 would look like this:

host2$ tcpdump -i eth0 tcp port 22 | grep host1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
14:11:12.101381 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: S 1120004832:1120004832(0) win 5840 
14:11:12.101391 IP host2.domain.tld.ssh >
host1.domain.tld.54432: S 320695995:320695995(0) ack 1120004833 win
5792 
14:11:12.101498 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: . ack 1 win 183 
14:11:12.107969 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1:24(23) ack 1 win 1448 
14:11:12.108063 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: . ack 24 win 183 
14:11:12.108165 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 1:23(22) ack 24 win 183 
14:11:12.108173 IP host2.domain.tld.ssh >
host1.domain.tld.54432: . ack 23 win 1448 
14:11:12.108341 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 23:663(640) ack 24 win 183

14:11:12.108347 IP host2.domain.tld.ssh >
host1.domain.tld.54432: . ack 663 win 1768 
14:11:12.109143 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 24:664(640) ack 663 win 1768

14:11:12.109313 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 663:687(24) ack 664 win 223

14:11:12.111342 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 664:816(152) ack 687 win 1768

14:11:12.113385 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 687:831(144) ack 816 win 223

14:11:12.119580 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 816:1280(464) ack 831 win 1768

14:11:12.121828 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 831:847(16) ack 1280 win 263

14:11:12.162117 IP host2.domain.tld.ssh >
host1.domain.tld.54432: . ack 847 win 1768 
14:11:12.162204 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 847:895(48) ack 1280 win 263

14:11:12.162219 IP host2.domain.tld.ssh >
host1.domain.tld.54432: . ack 895 win 1768 
14:11:12.162262 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1280:1328(48) ack 895 win 1768

14:11:12.163845 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 895:959(64) ack 1328 win 263

14:11:12.164001 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1328:1392(64) ack 959 win 1768

14:11:12.166947 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 959:1055(96) ack 1392 win 263

14:11:12.168083 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1392:1456(64) ack 1055 win 1768

14:11:12.170456 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 1055:1151(96) ack 1456 win 263

14:11:12.170493 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1456:1520(64) ack 1151 win 1768

14:11:12.170635 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 1151:1519(368) ack 1520 win 263

14:11:12.171024 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1520:1840(320) ack 1519 win 2088

14:11:12.180209 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 1519:2159(640) ack 1840 win 263

14:11:12.183403 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1840:1872(32) ack 2159 win 2408

-- 14:11:12.183659 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 2159:2223(64) ack 1872 win 263

14:11:12.187702 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1872:1920(48) ack 2223 win 2408

14:11:12.187923 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 2223:2671(448) ack 1920 win 263

14:11:12.201571 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1920:1968(48) ack 2671 win 2728

14:11:12.201596 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1968:2080(112) ack 2671 win 2728

14:11:12.201691 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: . ack 2080 win 263 
14:11:12.229982 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 2080:2144(64) ack 2671 win 2728

14:11:12.239626 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 2144:2208(64) ack 2671 win 2728

14:11:12.239719 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: . ack 2208 win 263 

If you read the above you can see host1 ack'ing 1120004833 from host2 to establish a connection. There are plenty of tcpdump examples on the Interblag.