Debian, Dell m600, XIV SAN

Installing Debian on a Dell m600 for use with an IBM XIV SAN (or any multipath'd SAN) requires a few tricks.

Modified Network Install ISO

The Broadcom NetExtreme II NICs that come in the Dell blades require non-free firmware not included in Lenny. I don't fault Debian for this. The same problem exists for the Qlogic HBA's. You can get around this by making your own custom Lenny install boot ISO. J Snell's howto describes how to add firmware for the NIC, but not the HBA. However, adding the HBA is just as easy. Note that the install will go more smoothly (no ambiguity as to which disk is /dev/sda) if no LUNs are mapped by the SAN. You'll see it question where it can load that firmware (bnx2-06-4.0.5.fw and ql2400_fw.bin) during the install.

Some details on how I created my ISO are:

 
cd /mnt/
mkdir cdrom
mount -o loop debian-testing-amd64-CD-1.iso cdrom
cd /home/$USER/debian
mkdir isocopy
cp -av cdrom isocopy
cd isocopy/
 
dpkg-deb -x firmware-bnx2_0.25_all.deb nic-firmware
cp nic-firmware/lib/firmware/* isocopy/cdrom
 
dpkg-deb -x firmware-qlogic_0.14+lenny2_all.deb hba-firmware
cp hba-firmware/lib/firmware/* isocopy/cdrom
 
cd isocopy/cdrom
mkisofs -o ../modified-debian.iso -b isolinux/isolinux.bin -c isolinux/boot.cat\
 -no-emul-boot -boot-load-size 4 -boot-info-table -J -R -V disks .

Burn the modified-debian.iso to a CD-ROM and boot the server from it. Once booted to the new ISO, you need to immediately switch to another console and copy the files over before the installer looks for them:

 
mkdir lib/firmware
cp /cdrom/*.fw lib/firmware

After that's done, the Debian installer should be able to autoload the bnx2 driver without any problems. Note that you can't do the above until after the system mounts the CD ROM. It will do this automatically. You'll want to do this before you scan for network devices but after the system mounts the CD ROM. I.e. if you try to do the above before the system automounts the CD you'll have trouble.

Mount by UUID

After assigning multiple paths from the SAN the Debian system became confused as to which block device was the local disk where / should be mounted. E.g.

server:~# ls /dev/sd<tab><tab>
sda    sdaa1  sdaf   sdaj   sdan1  sdd1   sdi    sdm1   sdr    sdv1   
sda1   sdab   sdaf1  sdak   sdao   sde    sdi1   sdn    sdr1   sdw    
sda2   sdab1  sdag   sdak1  sdap   sde1   sdj    sdn1   sds    sdw1   
sda5   sdac   sdag1  sdal   sdb    sdf    sdj1   sdo    sds1   sdx    
sda6   sdac1  sdah   sdal1  sdb1   sdg    sdk    sdo1   sdt    sdx1   
sda7   sdad   sdah1  sdam   sdc    sdg1   sdl    sdp    sdt1   sdy    
sda8   sdad1  sdai   sdam1  sdc1   sdh    sdl1   sdq    sdu    sdy1   
sdaa   sdae   sdai1  sdan   sdd    sdh1   sdm    sdq1   sdv    sdz    
server:~# 

Not only that, but it changed after each reboot. So, /dev/sda{1,2,5,6,7,8} was the local disk in the above scenario, but after a reboot it was /dev/sdao{1,2,5,6,7,8} etc. If this happens you'll see a message like the following after waiting for the system to boot:

Target filesystem doesn't have /sbin/init
No init found. Try passing init= bootarg

You'll then be dropped into BusyBox. You should then determine which disk is your local disk and mount it. In my case I was able to know it by the amount of partitions that it had as per above (e.g. only sdX had six partitions). Then mount that block device:

cd /
mkdir /root
mount -t ext3 /dev/sdap1 /root

Then bind /dev into your new root directory:

mount --bind /dev/ /root/dev/

Then chroot into your new root directory:

chroot /root/

then modify your /etc/fstab so you can mount the new block device. BusyBox doesn't have vi so you can use use sed:

sed s/sda/sdap/g -i /etc/fstab

You should then be able to mount everything:

mount -a

I would normally mount by label at this point, but e2label does not work on Debian the way it works on RedHat/Fedora. However, you can mount by UUID with Debian and you can find the UUID with the vol_id command:

vol_id /dev/sdao7

If you grep UUID from the above you should be able to set $UUID to the output and then modify another sed script to bring the UUID into the /etc/fstab. Once you're able to mount a copule of partitions by UUID reboot to see if you can bring the system up with / mounting correctly.

Multipath

Multipath is able to resolve the confusion from the many block devices and allow you to mount by nice names like /dev/mpathX.

apt-get install multipath-tools dmsetup
modprobe dm_mod dm-multipath dm-round-robin

You can optimize the queue depth for the XIV on Debian with Qlogic HBAs by adding the following to /etc/modprobe.d/aliases:

options qla2xxx ql2xmaxqdepth=64
options qla2xxx ql2xfailover=0 
options qla2xxx qlport_down_retry=1

I was then able to test multipath successfully.