xfs rhel4

Problem

Make a RHEL4 system mount a partition which can support more directories than ext3's inode max will allow.

Solution

Use a kernel module to use xfs. Note that we don't need a new kernel, just a new kernel module. There are RPMs for this. If you can install them correctly this won't even require any downtime.

Details

Going to test by making an XFS USB thumb drive. First we install the XFS Kernel Module. There is a howto for doing this with kernel modules via RPMs. For details see faqs.org.

I need 3 RPMS: xfsprogs, xfsprogs-devel, and the kernel-module-xfs:

rpm -Uvh xfsprogs-[kernel-version][rpm-version].rpm
rpm -Uvh xfsprogs-devel-[kernel-version][rpm-version].rpm
rpm -ivh kernel-module-xfs-[kernel-version][rpm-version].rpm

Given what I'm running:

# uname -r
2.6.9-67.0.1.EL
#

and a bit of searching I found a mirror which had the 2.6.9-67.0.1.EL kernel-module-xfs. Note that xfsprogs and xfsprogs-devel don't necessarily have to be the exact same version, just the specific kernel module. After following the order above I'm able to load the kernel module and verify that I have the XFS mkfs:

# modprobe xfs
# lsmod | grep xfs
xfs                   526832  0 
# which mkfs.xfs
/sbin/mkfs.xfs
# 

Next I'll look at the partition on the thumb drive (/dev/sda1 as per dmesg) and determine that I can mount it:

# parted
(parted) select /dev/sda1                                                 
Using /dev/fd0
(parted) mklabel msdos                                                    
(parted) print                                                            
Disk geometry for /dev/fd0: 0.000-1.406 megabytes
Disk label type: msdos
Minor    Start       End     Type      Filesystem  Flags
(parted) quit
# mount -t vfat /dev/sda1 /mnt/usb/
# umount /mnt/usb/

Then we format the partition for XFS:

# /sbin/mkfs.xfs -f -i size=512,maxpct=0 /dev/sda1 
meta-data=/dev/sda1              isize=512    agcount=3, agsize=4096 blks
         =                       sectsz=512  
data     =                       bsize=4096   blocks=12288, imaxpct=0
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096  
log      =internal log           bsize=4096   blocks=1200, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0
#

Finally we verify that we can mount it:

# mount -t xfs /dev/sda1 /mnt/usb/
# mount | grep xfs
/dev/sda1 on /mnt/usb type xfs (rw)
# 

After doing this you can see how many inodes it can handle and test it empirically. The following perl script will attempt to make an arbitrary number of directories:

#!/usr/bin/perl
$num_dirs = 38000;
system "mkdir test";
for($i=0; $i < $num_dirs; $i++) {
  system "mkdir test/$i";
  print "$i\n";
}

You can then run it in one window while you watch it eat inodes in the other:

# df -i /mnt/usb/
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1              86712   24138   62574   28% /mnt/usb
#
...
# df -i /mnt/usb/
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1              86176   38007   48169   45% /mnt/usb
#

So you can fill up half a drive with nothing by empty dirs:

# df -h /mnt/usb/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              44M   20M   24M  46% /mnt/usb
#

update

When I try to use XFS on an iSCSI LUN I get a kernel panic. All I have to do is mount the LUN, mkdir and then rmdir:

# rmdir 2
Message from syslogd@localhost at Tue Jan 29 10:38:34 2008 ...
kernel: Bad page state at free_hot_cold_page (in process 'iscsi-rx', page c1682a20)
 
Message from syslogd@localhost at Tue Jan 29 10:38:34 2008 ...
kernel: flags:0x20000084 mapping:00000000 mapcount:0 count:0
 
Message from syslogd@localhost at Tue Jan 29 10:38:34 2008 ...
kernel: Backtrace:
 
Message from syslogd@localhost at Tue Jan 29 10:38:35 2008 ...
kernel: Trying to fix it up, but a reboot is needed
 
#

Someone else has this problem too.