11th Annual System Administrator Appreciation Day

Here I go again: sysadminday.com.

New Mouse

I just got an Evoluent VM3R2-RSB Vertical Mouse 3. It does feel more natural and I guess you never know how much it hurts your wrists to keep them flat all day, until you try this mouse. I'm still getting used to it however. It's sensitive and I keep over moving it. So far I like it and am looking forward to getting used to it.

New Sorting World Record

There is a new Data sorting world record: 1 terabyte, 1 minute using TritonSort. I think this is really cool! It nicely combines things I found interesting from my good old Algorithms Class with my recent interest in building powerful computers using commodity hardware.

DMCA Exceptions

More often than not there is bad news in the area of copyright that involves people loosing freedom. Yesterday, as a friend of mine pointed out, there was good news.

The Library of Congress introduced new DMCA exceptions documented by the EFF and gizmodo. Also, GE played the role of the good guys and took a case to the 5th Circuit Appeals Court against MGE UPS Systems and won a ruling from a Federal judge that: It's OK to Break DRM for Fair Use.

Network Topology Icons

Cisco hosts charts of their Network Topology Icons including a PDF.

yum confused?

If you run a 'yum upgrade' and end up with a message like this:

Error: failed to retrieve repodata/filelists.xml.gz from rhel-x86_64-server-5 
error was [Errno -1] Metadata file does not match checksum 
 You could try using --skip-broken to work around the problem 
 You could try running: package-cleanup --problems 
                        package-cleanup --dupes 
                        rpm -Va --nofiles --nodigest 

it's likely that yum got confused. This happened to me when I changed a subscription channel, re-registered a system and tried to yum upgrade. I have been able to consistently fix it by running:

 rm -rf /var/cache/yum/*
 yum clean all
 yum update

Qlogic retry interval

I posted earlier about a test with the XIV with where the retry interval was too high and IO was queued longer than it had to be on the system buffer as opposed to being written to disk (see graph). We solved this a while ago, but I forgot to post an update.

Qlogic cards have an on-board BIOS setting, which sets the path failure delay to 45 seconds. In order for multipath to perform properly on these systems, this limit should be lowered. We're currently setting the limit to 0. The options to find in the BIOS are under the advanced settings for each host controller.

You should also set the following in in /etc/modprobe.conf (Fedora/RedHat/CentOS) or /etc/modprobe.d/aliases (Debian/Ubuntu):

options qla2xxx ql2xfailover=0 
options qla2xxx qlport_down_retry=1
options qla2xxx ql2xmaxqdepth=64

Note that the maxqdepth setting on the third line isn't about fail over but about getting more IOPS with respect to the queue depth. This is a standard XIV tuning practice. If you're using Emulex cards you can set the queue depth to 64 with the following line in your modprobe configuration file:

options lpfc lpfc_hba_queue_depth=64

but it is not necessary to change a retry or failover interval.

Debian, Dell m600, XIV SAN

Installing Debian on a Dell m600 for use with an IBM XIV SAN (or any multipath'd SAN) requires a few tricks.

Modified Network Install ISO

The Broadcom NetExtreme II NICs that come in the Dell blades require non-free firmware not included in Lenny. I don't fault Debian for this. The same problem exists for the Qlogic HBA's. You can get around this by making your own custom Lenny install boot ISO. J Snell's howto describes how to add firmware for the NIC, but not the HBA. However, adding the HBA is just as easy. Note that the install will go more smoothly (no ambiguity as to which disk is /dev/sda) if no LUNs are mapped by the SAN. You'll see it question where it can load that firmware (bnx2-06-4.0.5.fw and ql2400_fw.bin) during the install.

Some details on how I created my ISO are:

 
cd /mnt/
mkdir cdrom
mount -o loop debian-testing-amd64-CD-1.iso cdrom
cd /home/$USER/debian
mkdir isocopy
cp -av cdrom isocopy
cd isocopy/
 
dpkg-deb -x firmware-bnx2_0.25_all.deb nic-firmware
cp nic-firmware/lib/firmware/* isocopy/cdrom
 
dpkg-deb -x firmware-qlogic_0.14+lenny2_all.deb hba-firmware
cp hba-firmware/lib/firmware/* isocopy/cdrom
 
cd isocopy/cdrom
mkisofs -o ../modified-debian.iso -b isolinux/isolinux.bin -c isolinux/boot.cat\
 -no-emul-boot -boot-load-size 4 -boot-info-table -J -R -V disks .

Burn the modified-debian.iso to a CD-ROM and boot the server from it. Once booted to the new ISO, you need to immediately switch to another console and copy the files over before the installer looks for them:

 
mkdir lib/firmware
cp /cdrom/*.fw lib/firmware

After that's done, the Debian installer should be able to autoload the bnx2 driver without any problems. Note that you can't do the above until after the system mounts the CD ROM. It will do this automatically. You'll want to do this before you scan for network devices but after the system mounts the CD ROM. I.e. if you try to do the above before the system automounts the CD you'll have trouble.

Mount by UUID

After assigning multiple paths from the SAN the Debian system became confused as to which block device was the local disk where / should be mounted. E.g.

server:~# ls /dev/sd<tab><tab>
sda    sdaa1  sdaf   sdaj   sdan1  sdd1   sdi    sdm1   sdr    sdv1   
sda1   sdab   sdaf1  sdak   sdao   sde    sdi1   sdn    sdr1   sdw    
sda2   sdab1  sdag   sdak1  sdap   sde1   sdj    sdn1   sds    sdw1   
sda5   sdac   sdag1  sdal   sdb    sdf    sdj1   sdo    sds1   sdx    
sda6   sdac1  sdah   sdal1  sdb1   sdg    sdk    sdo1   sdt    sdx1   
sda7   sdad   sdah1  sdam   sdc    sdg1   sdl    sdp    sdt1   sdy    
sda8   sdad1  sdai   sdam1  sdc1   sdh    sdl1   sdq    sdu    sdy1   
sdaa   sdae   sdai1  sdan   sdd    sdh1   sdm    sdq1   sdv    sdz    
server:~# 

Not only that, but it changed after each reboot. So, /dev/sda{1,2,5,6,7,8} was the local disk in the above scenario, but after a reboot it was /dev/sdao{1,2,5,6,7,8} etc. If this happens you'll see a message like the following after waiting for the system to boot:

Target filesystem doesn't have /sbin/init
No init found. Try passing init= bootarg

You'll then be dropped into BusyBox. You should then determine which disk is your local disk and mount it. In my case I was able to know it by the amount of partitions that it had as per above (e.g. only sdX had six partitions). Then mount that block device:

cd /
mkdir /root
mount -t ext3 /dev/sdap1 /root

Then bind /dev into your new root directory:

mount --bind /dev/ /root/dev/

Then chroot into your new root directory:

chroot /root/

then modify your /etc/fstab so you can mount the new block device. BusyBox doesn't have vi so you can use use sed:

sed s/sda/sdap/g -i /etc/fstab

You should then be able to mount everything:

mount -a

I would normally mount by label at this point, but e2label does not work on Debian the way it works on RedHat/Fedora. However, you can mount by UUID with Debian and you can find the UUID with the vol_id command:

vol_id /dev/sdao7

If you grep UUID from the above you should be able to set $UUID to the output and then modify another sed script to bring the UUID into the /etc/fstab. Once you're able to mount a copule of partitions by UUID reboot to see if you can bring the system up with / mounting correctly.

Multipath

Multipath is able to resolve the confusion from the many block devices and allow you to mount by nice names like /dev/mpathX.

apt-get install multipath-tools dmsetup
modprobe dm_mod dm-multipath dm-round-robin

You can optimize the queue depth for the XIV on Debian with Qlogic HBAs by adding the following to /etc/modprobe.d/aliases:

options qla2xxx ql2xmaxqdepth=64
options qla2xxx ql2xfailover=0 
options qla2xxx qlport_down_retry=1

I was then able to test multipath successfully.

Configuration Management and Security: Bellovin & Bush

I started reading a paper on Configuration Management and Security which is good. I should remember to finish it.

IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 27, NO. 3, APRIL 2009
Configuration Management and Security
Steven M. Bellovin and Randy Bush

Abstract:
Proper configuration management is vital for host and network security. We outline the problems, especially for large-scale environments, and discuss the security aspects of a number of different configuration scenarios, including security appliances (e.g., firewalls), desktop and server computers, and PDAs. We conclude by discussing research challenges.

Cisco 2501

My friend let me borrow an old Cisco 2501 Router so I could play with IOS at home. I plugged a rj45 to rs232 adapter into the 2501 and my thinkpad running Xubuntu and set up my serial console with minicom as per the Ubuntu CiscoConsole howto.

After starting minicom back up I see:

Welcome to minicom 2.2

OPTIONS: I18n 
Compiled on Apr 27 2007, 15:50:20.
Port /dev/ttyS0

               Press CTRL-A Z for help on special keys
                                                     
                                                     
                                                     
First, would you like to see the current interface summary? [yes]: AT S7=45 S0=0
% Please answer 'yes' or 'no'.

First, would you like to see the current interface summary? [yes]: yes

Any interface listed with OK? value "NO" does not have a valid configuration

Interface        IP-Address      OK?  Method    Status                 Protocol
Ethernet0        unassigned      NO   not set   up                     down    
Serial0          unassigned      NO   not set   down                   down    
Serial1          unassigned      NO   not set   down                   down    

Configuring global parameters:

  Enter host name [Router]: 
...
Use this configuration? [yes/no]: yes
Building configuration...



Press RETURN to get started!


%LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0, changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0, changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1, changed state to down
%LINK-3-UPDOWN: Interface Ethernet0, changed state to up
%LINK-3-UPDOWN: Interface Serial0, changed state to down
%LINK-3-UPDOWN: Interface Serial1, changed state to down
%LINK-5-CHANGED: Interface Ethernet0, changed state to administratively down
%LINK-5-CHANGED: Interface Serial0, changed state to administratively down
%LINK-5-CHANGED: Interface Serial1, changed state to administratively down
%SYS-5-RESTART: System restarted --
Cisco Internetwork Operating System Software 
IOS (tm) 3000 Software (CPA25-Y-L), Version 11.0(4), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-1995 by cisco Systems, Inc.
Compiled Mon 18-Dec-95 19:21 by alanyu
%SNMP-3-SOCKET: can't open UDP socket
Router>
Router>
Router>sh version
Cisco Internetwork Operating System Software 
IOS (tm) 3000 Software (CPA25-Y-L), Version 11.0(4), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-1995 by cisco Systems, Inc.
Compiled Mon 18-Dec-95 19:21 by alanyu
...
cisco 2500 (68030) processor (revision L) with 2044K/2048K bytes of memory.
Processor board ID 02425768, with hardware revision 00000000
...
Router>
Router>
Router>

All set. Looks like I'll be playing with IOS 11.0.

Mail Clients: mutt + zimbra et al

We run Zimbra where I work. I've been using Thunderbird (and webmail for Calendaring) but I used to use mutt when my mail was just on a server running qmail.

A friend of mine caught me saying I was tempted to get mutt working with zimbra via imap/smtp. He beat me to it. Here's a .muttrc:

# Automatically log in to this mailbox at startup
set spoolfile="imaps://netid@incoming.domain.tld/"
# Define the = shortcut, and the entry point for the folder browser
set folder="imaps://netid@incoming.domain.tld/"
# Use a well-defined SMTP server
set smtp_url="smtps://netid@outgoing.domain.tld:465/"
# Define Zimbra-default folders
set record="=Sent"
set postponed="=Drafts"
# activate TLS if available on the server
set ssl_starttls=yes
# always use SSL when connecting to a server
set ssl_force_tls=yes
# Don't wait to enter mailbox manually
unset imap_passive
# Automatically poll subscribed mailboxes for new mail
set imap_check_subscribed
# Reduce polling frequency to a sane level
set mail_check=60
# And poll the current mailbox more often (not needed with IDLE in post
1.5.11)
set timeout=10
# keep a cache of headers for faster loading (1.5.9+?)
set header_cache=~/.hcache
# Display download progress every 5K
set net_inc=5
# Use emacs (remember to server-start)
set editor="emacsclient -t"
# Set From: address
my_hdr From: First Last 

Note that you must have emacs running in the background to use "emacsclient -t". If you do this add "(server-start)" to your .emacs or use "M-x server-start".

Our discussion on mail clients lead to others worth checking out:

• Claws Mail - Screenshots
• Wanderlust

Droid X bricks if you try to mod it

Someone passed a long a mobilecrunch.com article: Droid X actually self-destructs if you try to mod it. If this is true I'm glad I didn't hold out for one.

Fully Free Android system-image?

The Android Developer's blog recently posted that: "You can now build and boot a fully open-source system image out of the box, for the emulator, as well as for Dream (ADP1), Sapphire (ADP2), and Passion (Nexus One)."

iPhone to Nexus One

I switched from an iPhone to a Nexus One this week. My AT&T contract does not expire for a while. Switching was easy; once I received the phone in the mail I removed the iPhone sim card with a paper clip and inserted it into the Nexus One. After charging the Nexus One everything worked flawlessly.

EFF 20th Anniversary Animation

Amusing EFF 20th Anniversary Animation by Nina Paley.