mail

This is a handy command for testing:

echo -e "sent on `date`" | mail -s "test: `hostname`" foo@domain.tld

It will send mail quickly from the command line and there's no need to muck about with any clients After running this you can go see how your message is doing in /var/spool/mqueue etc.

tar -C

Would you believe that I got this far without knowing the tar -C option?

If you do the following:

tar xzf foo.tar.gz -C /

then /foo will contain the contents of foo.tar.gz.

Note that the -C means change to the specified directory so that the contents of your extract end up there. As the man page says:

       -C, --directory DIR
              change to directory DIR

I normally just put the tarball where I want it extracted and then extract it. My problem tonight was that the partition where I wanted to extract it to was not big enough for both the tarball and its files. I had to extract it to that directory from a different partition. Of course I had to wait until I saw a disk full error to realize this. If I had known about it I would probably be home now and not waiting for a 5G tarball to uncompress into 15G. I'll probably remember it now.

tcpdump

If you're debugging network services between two hosts don't forget to let tcpdump help you.

E.g. if host1 can't SSH to host2 and you think an external firewall is blocking you, try this:

1. Have host2 display TCP info on port 22 and display only results containing host1:

host2$ tcpdump -i eth0 tcp port 22 | grep host1

2. Try to SSH from host1 to host2.

If an external firewall is blocking them you should see nothing from the command above. However, if you see something like the following, then the external firewall isn't blocking you and it's an issue between the two hosts:

host2$ tcpdump -i eth0 tcp port 22 | grep host1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
14:08:50.412007 IP host1.domain.tld.54403 >
host2.domain.tld.ssh: S 2707949880:2707949880(0) win 5840 
14:08:53.411702 IP host1.domain.tld.54403 >
host2.domain.tld.ssh: S 2707949880:2707949880(0) win 5840 
14:08:59.411422 IP host1.domain.tld.54403 >
host2.domain.tld.ssh: S 2707949880:2707949880(0) win 5840 

In the case above host2 is unable to get back to host1 to complete the TCP hand shake. In this case you could try reaching host1 from host2 and debug the resulting issue (e.g. broken netmask on host2).

A healthy looking tcpdump from host2 would look like this:

host2$ tcpdump -i eth0 tcp port 22 | grep host1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
14:11:12.101381 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: S 1120004832:1120004832(0) win 5840 
14:11:12.101391 IP host2.domain.tld.ssh >
host1.domain.tld.54432: S 320695995:320695995(0) ack 1120004833 win
5792 
14:11:12.101498 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: . ack 1 win 183 
14:11:12.107969 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1:24(23) ack 1 win 1448 
14:11:12.108063 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: . ack 24 win 183 
14:11:12.108165 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 1:23(22) ack 24 win 183 
14:11:12.108173 IP host2.domain.tld.ssh >
host1.domain.tld.54432: . ack 23 win 1448 
14:11:12.108341 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 23:663(640) ack 24 win 183

14:11:12.108347 IP host2.domain.tld.ssh >
host1.domain.tld.54432: . ack 663 win 1768 
14:11:12.109143 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 24:664(640) ack 663 win 1768

14:11:12.109313 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 663:687(24) ack 664 win 223

14:11:12.111342 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 664:816(152) ack 687 win 1768

14:11:12.113385 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 687:831(144) ack 816 win 223

14:11:12.119580 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 816:1280(464) ack 831 win 1768

14:11:12.121828 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 831:847(16) ack 1280 win 263

14:11:12.162117 IP host2.domain.tld.ssh >
host1.domain.tld.54432: . ack 847 win 1768 
14:11:12.162204 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 847:895(48) ack 1280 win 263

14:11:12.162219 IP host2.domain.tld.ssh >
host1.domain.tld.54432: . ack 895 win 1768 
14:11:12.162262 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1280:1328(48) ack 895 win 1768

14:11:12.163845 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 895:959(64) ack 1328 win 263

14:11:12.164001 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1328:1392(64) ack 959 win 1768

14:11:12.166947 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 959:1055(96) ack 1392 win 263

14:11:12.168083 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1392:1456(64) ack 1055 win 1768

14:11:12.170456 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 1055:1151(96) ack 1456 win 263

14:11:12.170493 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1456:1520(64) ack 1151 win 1768

14:11:12.170635 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 1151:1519(368) ack 1520 win 263

14:11:12.171024 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1520:1840(320) ack 1519 win 2088

14:11:12.180209 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 1519:2159(640) ack 1840 win 263

14:11:12.183403 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1840:1872(32) ack 2159 win 2408

-- 14:11:12.183659 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 2159:2223(64) ack 1872 win 263

14:11:12.187702 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1872:1920(48) ack 2223 win 2408

14:11:12.187923 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: P 2223:2671(448) ack 1920 win 263

14:11:12.201571 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1920:1968(48) ack 2671 win 2728

14:11:12.201596 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 1968:2080(112) ack 2671 win 2728

14:11:12.201691 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: . ack 2080 win 263 
14:11:12.229982 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 2080:2144(64) ack 2671 win 2728

14:11:12.239626 IP host2.domain.tld.ssh >
host1.domain.tld.54432: P 2144:2208(64) ack 2671 win 2728

14:11:12.239719 IP host1.domain.tld.54432 >
host2.domain.tld.ssh: . ack 2208 win 263 

If you read the above you can see host1 ack'ing 1120004833 from host2 to establish a connection. There are plenty of tcpdump examples on the Interblag.

nice

I experimented with nice values and top and I'm pasting my results here.

When running top note the value of PR vs NI:

PR:

The priority number is calculated by the kernel and is used to determine the order in which processes are schedule. The kernel takes many factors in to consideration when calculating this number, and it is not unusual to see large fluctuations in this number over the lifetime of a process.

NI:

This column reflects the "nice" setting of each process. A process's nice is inhereted from its parent. Most user processes run at a nice of 0, indicating normal priority. Users have the option of starting a process with a positive nice value to allow the system to reduce the priority given to that process. This is normally done for long-running cpu-bound jobs to keep them from interfering with interactive processes. The Unix command "nice" controls setting this value. Only root can set a nice value lower than the current value. Nice values can be negative. On most systems they range from -20 to 20. The nice value influences the priority value calculated by the Unix scheduler.

To see these values in action I'll start two CPU intensive processes:

someguy@machine:~$ dd if=/dev/urandom of=/dev/null & 
[1] 31406
someguy@machine:~$ dd if=/dev/urandom of=/dev/null & 
[2] 31415
someguy@machine:~$ 

I see them both at the top of the stack:

  
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
31406 someguy   25   0  9200  772  620 R  100  0.0   3:48.62 dd                 
31415 someguy   25   0  9204  772  620 R  100  0.0   3:28.74 dd                 

Note their priority of 25 and their nice value, let's see how they change as we renice them.

Remember, the nicer a program the lower it's priority. The less nice a program the higher it's priority. For exmple:

• Nice value of 19 is letting people walk on you
• Nice value of 5 is holding the door
• Nice value of -20 is pushing people out of the way

You can nice a program when you start it:

   nice -n -10 xmms 

If it's already running:

   renice 19 -p $pid

In this case I'll move one way up and one way down:

lower 406: renice 19 -p 31406
raise 415: renice -19 -p 31415

The results of the first:

someguy@machine:~$ renice 19 -p 31406
31406: old priority 0, new priority 19
someguy@machine:~$ 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
31415 someguy   25   0  9204  772  620 R  100  0.0   6:50.54 dd                 
31406 someguy   39  19  9200  772  620 R  100  0.0   7:10.48 dd              

So, it's priority went higher and the higher nice value shows. So, even the PR numbers are inversely proportional to nice numbers. Now, I raise the other:

someguy@machine:~$ sudo renice -19 -p 31415
Password:
31415: old priority 0, new priority -19
someguy@machine:~$ 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
31406 someguy   39  19  9200  772  620 R  100  0.0   8:46.89 dd                 
31415 someguy    6 -19  9204  772  620 R  100  0.0   8:26.92 dd               

So, I've raised the priority of the first. I could maximize it:

someguy@machine:~$ sudo renice -20 -p 31415
31415: old priority -19, new priority -20
someguy@machine:~$ 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
31406 someguy   39  19  9200  772  620 R  100  0.0  17:00.51 dd                 
31415 someguy    5 -20  9204  772  620 R  100  0.0  16:41.13 dd              

Note that I still can't get it past 5 with the lowest possible nice value. Note also that these are the two most CPU intensive processes so they both take the top of the stack, even with the most extreme nice values. If I had a third it should be sandwiched between the two.

someguy@machine:~$ dd if=/dev/urandom of=/dev/null & 
[1] 32089
someguy@machine:~$ 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
32089 someguy   25   0  9200  768  620 R  100  0.0   0:21.44 dd                 
31415 someguy    5 -20  9204  772  620 R  100  0.0  18:29.59 dd                 
31406 someguy   39  19  9200  772  620 R  100  0.0  18:48.88 dd                

Now it's scheduling them round robin.

15, 6, 89 :: 15, 89, 6 :: 6, 89, 15 :: 89, 6, 15 :: 6, 15, 89

But 15 is on top 2/3 of the time. Interesting. It still letting the other jobs at the resources.

Now, I'll start a fourth CPU hog with a nice value from the start:

someguy@machine:~$ sudo nice -n -20 dd if=/dev/urandom of=/dev/null &
[2] 32288
someguy@machine:~$ 

I predict that 88 and 15 will stay on top most of the time. Note that it was started with a higher nice value which required sudo so it's running as root:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
31415 someguy    5 -20  9204  772  620 R  100  0.0  30:48.50 dd                 
32288 root       5 -20  9200  772  620 R  100  0.0   4:59.89 dd                 
31406 someguy   39  19  9200  772  620 R   99  0.0  31:05.53 dd                 
32089 someguy   25   0  9200  768  620 R   99  0.0  12:39.09 dd                

Now I'll kill 88 and 15. Then let's try raising the importance of 6 and introducting a new process with a priority just below it:

renice -20 -p 31406 && nice -n -19 dd if=/dev/urandom of=/dev/null &


someguy@machine:~$ sudo su -
root@machine:~# renice -20 -p 31406 && nice -n -19 dd if=/dev/urandom of=/dev/null &
[1] 32656
root@machine:~# 31406: old priority 19, new priority -20

root@machine:~# 

top shows:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
31406 someguy    5 -20  9200  772  620 R  100  0.0  39:28.19 dd                 
32089 someguy   25   0  9200  768  620 R  100  0.0  21:02.10 dd                 
32658 root       6 -19  9200  768  620 R  100  0.0   0:22.50 dd              

Wasn't that fun?

Remember when running top that you can see how all of the CPUs are working by pressing 1.

printing

No, I don't mean having your program print something to the screen:

printf("Hello, world...\n");

I mean sending a PS file (if you're lucky) to a mechanical device which puts its content on paper. What good is that? You can't grep paper. You can't pipe programs to or from it or even copy and paste with it. I guess it works when the power is out, but you can't read it in the dark. A wise man once told me "the only time I've had to print something is when I had to hand it to an idiot". Since this post is about doing something that's not cool I'll describe a non-cool way to approach the issue since it relates to a non-cool part of my job and I'm just going to log it here. Sorry it's lame. I hope the anecdote made this post worth reading on some level.

Anyway, If you're configuring a RedHat box that wasn't minimally installed you can add printers remotely by doing:

ssh -X server

Once you're in you can do:

sudo /usr/sbin/system-config-printer-gui

<sarcasm>love it!</sarcasm> Note that the GUI requires you to have something present for the printer path. This seems like a bug since there are times when I want to print to a network printer like an LaserJet 4000 and this field simply doesn't apply. If this happens you can create the printer with the GUI and then you need to go in and edit the file that it generates (which is /etc/cups/printers.conf) and delete that path.