su -
/usr/sbin/visudo
/usr/sbin/usermod -a -G wheel $USER
sed -i s/"PasswordAuthentication yes"/"PasswordAuthentication no"/g
/etc/ssh/sshd_config
/etc/init.d/sshd restart
exit
sudo ls
Wednesday, January 30, 2008
standard tightening
I often run the following commands when I set up a new server:
Thursday, January 24, 2008
xfs rhel4
Problem
Make a RHEL4 system mount a partition which can support more directories than ext3's inode max will allow.Solution
Use a kernel module to use xfs. Note that we don't need a new kernel, just a new kernel module. There are RPMs for this. If you can install them correctly this won't even require any downtime.Details
Going to test by making an XFS USB thumb drive. First we install the XFS Kernel Module. There is a howto for doing this with kernel modules via RPMs. For details see faqs.org. I need 3 RPMS: xfsprogs, xfsprogs-devel, and the kernel-module-xfs:rpm -Uvh xfsprogs-[kernel-version][rpm-version].rpm rpm -Uvh xfsprogs-devel-[kernel-version][rpm-version].rpm rpm -ivh kernel-module-xfs-[kernel-version][rpm-version].rpmGiven what I'm running:
# uname -r 2.6.9-67.0.1.EL #and a bit of searching I found a mirror which had the 2.6.9-67.0.1.EL kernel-module-xfs. Note that xfsprogs and xfsprogs-devel don't necessarily have to be the exact same version, just the specific kernel module. After following the order above I'm able to load the kernel module and verify that I have the XFS mkfs:
# modprobe xfs # lsmod | grep xfs xfs 526832 0 # which mkfs.xfs /sbin/mkfs.xfs #Next I'll look at the partition on the thumb drive (/dev/sda1 as per dmesg) and determine that I can mount it:
# parted (parted) select /dev/sda1 Using /dev/fd0 (parted) mklabel msdos (parted) print Disk geometry for /dev/fd0: 0.000-1.406 megabytes Disk label type: msdos Minor Start End Type Filesystem Flags (parted) quit # mount -t vfat /dev/sda1 /mnt/usb/ # umount /mnt/usb/Then we format the partition for XFS:
# /sbin/mkfs.xfs -f -i size=512,maxpct=0 /dev/sda1
meta-data=/dev/sda1 isize=512 agcount=3, agsize=4096 blks
= sectsz=512
data = bsize=4096 blocks=12288, imaxpct=0
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=1200, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
#
Finally we verify that we can mount it:
# mount -t xfs /dev/sda1 /mnt/usb/ # mount | grep xfs /dev/sda1 on /mnt/usb type xfs (rw) #After doing this you can see how many inodes it can handle and test it empirically. The following perl script will attempt to make an arbitrary number of directories:
#!/usr/bin/perl
$num_dirs = 38000;
system "mkdir test";
for($i=0; $i < $num_dirs; $i++) {
system "mkdir test/$i";
print "$i\n";
}
You can then run it in one window while you watch it eat inodes in the other:
# df -i /mnt/usb/ Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 86712 24138 62574 28% /mnt/usb # ... # df -i /mnt/usb/ Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 86176 38007 48169 45% /mnt/usb #So you can fill up half a drive with nothing by empty dirs:
# df -h /mnt/usb/ Filesystem Size Used Avail Use% Mounted on /dev/sda1 44M 20M 24M 46% /mnt/usb #
update
When I try to use XFS on an iSCSI LUN I get a kernel panic. All I have to do is mount the LUN, mkdir and then rmdir:# rmdir 2 Message from syslogd@localhost at Tue Jan 29 10:38:34 2008 ... kernel: Bad page state at free_hot_cold_page (in process 'iscsi-rx', page c1682a20) Message from syslogd@localhost at Tue Jan 29 10:38:34 2008 ... kernel: flags:0x20000084 mapping:00000000 mapcount:0 count:0 Message from syslogd@localhost at Tue Jan 29 10:38:34 2008 ... kernel: Backtrace: Message from syslogd@localhost at Tue Jan 29 10:38:35 2008 ... kernel: Trying to fix it up, but a reboot is needed #Someone else has this problem too.
Tuesday, January 22, 2008
Fibre Channel I/O Calls
A colleague of mine came across the following fact regarding I/O and fabric switches:
- 2 Gbps FC can queue 254 I/O commands
- 4 Gbps FC can queue 2048 I/O commands
# cat /proc/diskstats | grep " sdc " 8 32 sdc 391348542 3811329 642958694 1765819166 212637694 1424571277 438970288 1314722135 1 366113251 3445284834 #As per comp.os.linux.development these fields (starting after the device name) are:
Field 1 -- # of reads issued Field 2 -- # of reads merged, field 6 -- # of writes merged Field 3 -- # of sectors read Field 4 -- # of milliseconds spent reading Field 5 -- # of writes completed Field 7 -- # of sectors written Field 8 -- # of milliseconds spent writing Field 9 -- # of I/Os currently in progress Field 10 -- # of milliseconds spent doing I/Os Field 11 -- weighted # of milliseconds spent doing I/OsOr to put it another way:
391348542 reads issued (4)
3811329 reads merged (5)
642958694 sectors read (6)
1765819166 milliseconds spent reading (7)
212637694 writes completed (8)
1424571277 writes merged (9)
438970288 sectors written (10)
1314722135 milliseconds spent writing (11)
1 I/Os currently in progress (12)
366113251 milliseconds spent doing I/Os (13)
3445284834 weighted milliseconds spent doing I/Os (14)
Note that I've put the awk offset in parentheses above. We can then take more readings and focus on essential columns. E.g. we spend more time reading than writing:
# while [ 1 ]; do grep " sdc " /proc/diskstats |
awk {'print $7 " " $11'}; sleep 1; done
1767053699 1323835167
1767053722 1323835217
1767054231 1323858000
1767054400 1323858477
1767054401 1323859097
1767054420 1323859106
1767055201 1323863662
1767055543 1323863671
1767055666 1323864799
1767056048 1323865700
#
If we look at them every quarter second we can see spikes in the number of I/O along with number of reads and write issues during that time (looking at a larger interval hides the spikes):
# while [ 1 ]; do grep " sdc " /proc/diskstats |
awk {'print $12 "\t" $4 "\t" $8'}; sleep 0.25; done
1 391689249 213184077
1 391689253 213184467
4 391689253 213184912
4 391689253 213185311
4 391689253 213185780
1 391689257 213186170
1 391689257 213186558
2 391689258 213187017
68 391689271 213187319
1 391689271 213187801
2 391689313 213188219
1 391689338 213188481
2 391689379 213188863
44 391689379 213189282
32 391689384 213190180
3 391689400 213190569
3 391689400 213190971
1 391689405 213191429
3 391689407 213192172
#
We can check the math on the last few lines. Because our sampling interval is missing events that occur in between our numbers won't add up exactly, but we can see a general trend in some of these numbers:
1 391689338 213188481 2 391689379 213188863 44 391689379 213189282 32 391689384 213190180There were a lot more writes than reads from the samples taken above
3 391689400 213190569 3 391689400 213190971Nothing was read, but 3 I/O operations seem to have been written. I don't have anything conclusive from the above but I do think I can use this over time and recognize trends on my system.
Monday, January 7, 2008
RHEL5 Install via Ubuntu NFS
After booting a RHEL4 or 5 disk with "linux askmethod" and getting the system online (the Celerra could ping it) and making sure the appropriate IP was in the ACL, I still couldn't mount the ISO images to install:
RPC timeout that directory could not be mounted from the serverI ended up mounting the Celerra from an already installed system on the same network, copying the ISOs over to it and then umounting the Celerra. I then made that system an NFS server and now the five RHEL5 hosts have no problem mounting it for an NFS based install. When setting up an NFS server make sure that portmap is running so that it listens on port 111 for the RPC calls that NFS needs.
# netstat -npl | egrep "111|2049" tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 29969/portmap udp 0 0 0.0.0.0:2049 0.0.0.0:* - udp 0 0 0.0.0.0:111 0.0.0.0:* 29969/portmapIt's annoying that I'm not sure why the Celerra won't serve this purpose and that I don't have any log data to figure out why. If time allows I'll try again with tcpdump but I need to get these hosts installed.
Update
The firewall admin noticed that the host that was booted from the RHEL5 CD was trying to make a UDP connection to port 1234 of the Celerra. Opening this seems to have fixed the problem.nfs-common
Out of the box Ubuntu will support NFS mounting but will take about 90 seconds to do it and not work well. If you check /var/log/messages you'll see errors [1]. To fix this install the nfs-common package:
http://packages.ubuntu.com/feisty/net/nfs-common
Footnote:
[1]
[4661210.004709] portmap: server localhost not responding, timed out [4661210.004745] RPC: failed to contact portmap (errno -5). [4661244.949461] portmap: server localhost not responding, timed out [4661244.949496] RPC: failed to contact portmap (errno -5). [4661244.949513] lockd_up: makesock failed, error=-5 [4661279.894214] portmap: server localhost not responding, timed out [4661279.894248] RPC: failed to contact portmap (errno -5). [4661279.894255] nfs: Starting lockd failed (do you have nfs-common installed?). [4661279.894284] nfs: Continuing anyway, but this workaround will go away soon.
Subscribe to:
Posts (Atom)