quick random password generator

Need a random password in a hurry?

someguy@workstation:~> python
Python 2.5.2 (r252:60911, Jul 31 2008, 17:31:22) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import string
>>> from random import Random
>>> "".join(Random().sample(string.letters+string.digits, 8))
'GAZcEFIy'
>>> 

Update, here it is in a one-liner:

import string; from random import Random; "".join(Random().sample(string.letters+string.digits, 8))

O'Reilly on Twitter

O'Reilly's argument has me thinking that I might try twitter again.

Book: Automating UNIX and Linux Administration

Note to self: I need to get more of this. Update: See about joining LISA.

Take that spammers

My spam filter has not been working as hard recently. I wonder if it's related to the recent spam bust.

OpenSSL CLI

I like to telnet into ports and speak the protocol a little to make sure a server is doing what it's supposed to do. This gives you more information than just configuring a client with that you would like it to use. When testing ports secured with SSL you can still do this with the OpenSSL Command Line tools.

multicore in python as a scripting language

As posted on Python-list:

Use Python threads if you need to run IO operations in parallel. Do not use Python threads if you need to run computations in parallel. [Use mpi4py for parallel computations.]

So if you're using Python as a nicer-than-shell scripting language to trigger CPU intensive processes that the OS could easily pass to different cores via two terminals or just '&', what do you use? Threads.

Inspired by a basic threading tutorial I whipped up a quick script to give a multicore *nix box a quick workout: search the hard drive for random strings via the find command. This is IO intensive as said on Python-list. It's also CPU intensive but in this case Python's interface with the CPU intensive process is to wait while os.popen() does the work, i.e. wait for IO. Thus, simple threads work well for this.

The basic tutorial's first two examples include simple threads and then locks. The locks are there to keep the threads synchronized so that they wait for each other. This is good if each is sharing a resource like writing to the same file but is the opposite of what I'm after in this case. I've combined both examples to make it easy to flip back and forth between them. Running top in another terminal let's you see how the other cores are being used. As a final step I remove the sleep calls from the simple threads.

#!/usr/bin/env python                                                                           
import time
import thread
import os

def simple_function(string,sleeptime,junk,*args):
  while 1:
    print string
    # not sleeping, full attack mode 
    # time.sleep(sleeptime)                     
    cmd = "find / -name " + junk
    output = os.popen(cmd).read()
    print output

def lock_function(string,sleeptime,junk,lock,*args):
  while 1:
    # entering critical                                                             
    lock.acquire()
    print string," Now Sleeping after Lock acquired for ",sleeptime
    time.sleep(sleeptime)
    cmd = "find / -name " + junk
    output = os.popen(cmd).read()
    print output
    print string," Now releasing lock and then sleeping again"
    lock.release()
    #exiting critical                                                              
    time.sleep(sleeptime)                                       

if __name__=="__main__":
  no_multi_core = 0 
  # have 8 corese, so false                                                 
  if (no_multi_core):
    lock=thread.allocate_lock()
    thread.start_new_thread(lock_function,("Thread No:1",2,"foo",lock))
    thread.start_new_thread(lock_function,("Thread No:2",2,"bar",lock))
    thread.start_new_thread(lock_function,("Thread No:3",3,"bar",lock))
    thread.start_new_thread(lock_function,("Thread No:4",4,"baz",lock))
    thread.start_new_thread(lock_function,("Thread No:5",5,"qux",lock))
    thread.start_new_thread(lock_function,("Thread No:6",5,"quux",lock))
    thread.start_new_thread(lock_function,("Thread No:7",5,"corge",lock))
    thread.start_new_thread(lock_function,("Thread No:8",5,"grault",lock))
  else: # imporove w/ loop
    thread.start_new_thread(simple_function,("Thread No:1",2,"foo"))
    thread.start_new_thread(simple_function,("Thread No:2",2,"bar"))
    thread.start_new_thread(simple_function,("Thread No:3",3,"bar"))
    thread.start_new_thread(simple_function,("Thread No:4",4,"baz"))
    thread.start_new_thread(simple_function,("Thread No:5",5,"qux"))
    thread.start_new_thread(simple_function,("Thread No:6",5,"quux"))
    thread.start_new_thread(simple_function,("Thread No:7",5,"corge"))
    thread.start_new_thread(simple_function,("Thread No:8",5,"grault"))
  while 1:pass

Here's a screen cap of top with eight find commands spawned immediately with no_multi_core set to false. This variable name is probably a little chauvinistic but I've emphasized the example for my original goal: use Python to script lots of IO work and use many cores.

Update

I've updated this to optimally run for desired_thread_count users at a time from a file. I originally had a scheduling bug whose fix was to put the actual_thread_count increment in main as opposed to the function call; my original code. Tuns out that more than desired_thread_count threads were running. They were spawned before any thread could update the variable to keep future spawn calls from sleeping.

#!/usr/bin/env python
import time
import thread
import os
# global variables
the_file = "users.txt"
desired_thread_count = 8
actual_thread_count = 0
# -------------------------------------------------------
def find(lock,count,user):
    global desired_thread_count
    global actual_thread_count
    # do the work
    print "T:" + str(count) + " of " + str(actual_thread_count) + " " + user
    cmd = "find / -name " + user + " 2> /dev/null"
    os.popen(cmd)
    lock.acquire() # we're done, let someone else go
    actual_thread_count = actual_thread_count - 1 
    lock.release()
# -------------------------------------------------------
if __name__=="__main__":
    i = 0
    lock = thread.allocate_lock()
    try:
        f = open(the_file)
        try:
            for line in f:
                user = line.strip().split(',')[0]
                i += 1
                while (actual_thread_count >= desired_thread_count):
                    time.sleep(2) # wait to run, if necessary
                lock.acquire() # one update at a time
                actual_thread_count = actual_thread_count + 1
                lock.release()
                thread.start_new_thread(find,(lock,i,user))
        finally:
            while (actual_thread_count > 0):
                time.sleep(2) # wait for threads to finish
            f.close()
    except IOError:
        sys.exit("Unable to open the file " + the_file)

Update 2: Note the addition above of an extra loop after the finally. This keeps the program alive until the threads finish. Without this the program would end while the last threads are still running and they wouldn't finish their computations.

DRDB

DRDB: refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network.

MPIO

Novell has a simple howto for MPIO. I also found more details and how to do failover tests.

I've heard that LVM manages physical slices by device name (/dev/sdX), not disk ID and that since multipathd increments device names (/dev/sdc becomes /dev/sdd) that LVM gets confused and requires a vgscan and vgchange -a y between a remount. I'll try for myself sometime.

Update QLogic has a forum post on failover and LUNs discussing the redundant devices and more information about what they mean.

VirtualBox

VirtualBox is interesting. Ubuntu should make it easy to test.

Google Data Center Power Usage

Data Center Efficiency Measurements

chrome libraries

Code Reuse in Google Chrome Browser.

EFF: Online Speech

Hear! Hear!

iSCSI Enterprise Target via RPMs

This how to is handy. It doesn't end well for the author but my "yum install kmod-iscsitarget" was all I needed. It brought in a 2.6.25 Kernel as a dependency.

Downloading Packages:
(1/5): kmod-iscsitarget-2 100% |=========================|  37 kB    00:00     
(2/5): kmod-iscsitarget-0 100% |=========================| 4.3 kB    00:00     
(3/5): kernel-2.6.25.14-6 100% |=========================|  18 MB    00:34     
(4/5): iscsitarget-0.4.15 100% |=========================|  64 kB    00:00     
(5/5): iwl4965-firmware-2 100% |=========================| 237 kB    00:00    

Beware: This installs iscsitarget-0.4.15 which generates a broken pipe error for certain ietadm commands. This is not obvious until you run strace. 0.4.16 doesn't have this problem.

IronKey USB

IronKey Announces Linux Support. I wonder if this means the "windows initialization" is no longer required. Update Yes, "windows initialization" is not required. I plugged mine into Ubuntu and was up and running. No need for windows with this.

Python urlencode annoyance

O'Reilly has examples of how to urlencode a string for various languages. Every example has the same semantics -- a variable to define a string and a function to encode it -- except Python. Python's urlencode presumes you want to convert a dictionary whose key (which seems to be a name you should make up when you call this function) will be a GET variable that you would append to the tail of the URL. I think this presumes too much. I want to build a URL and simply encode the name of a file. I wish I could do this:

url = "http://foo.com/" + urlencode(file_name)

Instead I use a substring kludge since len("x=") is 2:

tmp = urlencode({'x':file_name})
url = "http://foo.com/" + tmp[2:]

Applying this back to O'Reilly's example:

from urllib import urlencode
artist = "Kruder & Dorfmeister"
tmp = urlencode({'x':artist})
artist = tmp[2:]

I guess Python is promoting a paradigm where URL values are arguments to methods and the syntax of a URL itself is abstracted away. But I still would have kept urlencode like the other languages and offered a variation of this function that does whatever Python is trying to promote.

Zimbra and LVM

The Zimbra Forum has some things to say about LVM:

• LVM snaphots hurt performance. However they are handy for minimizing downtime if you want to do a full backup of /opt/zimbra. Zimbra's built-in backups seem a better trade-off to me. While reading through this there as a kernel trap post on replacing atime with relatime.
• Someone who claims to be knowledge in storage and who posts to the forum every day on average suggested using LVM for the flexibility.
• Someone using LVM suggests that zmvolume makes LVM less necessary.
• There were plenty of posts mentioning LVM on the Forum which were answered by Zimbra employees or experienced admins where LVM was stated as something someone was using or were going to use and no one said not to use it. This included advice for partitioning new servers or I/O tuning (where the most popular thing you shouldn't do is use RAID5).

Update: LVM and multipathd conflict. I prefer Zimbra on ext3 directly on disk; no LVM.

Hadoop & NIST Algorithm Dictionary

The Register had a funny article about haddop. I also found a NIST Dictionary of Algorithms and Data Structures.

CSS Positioning in Ten Steps

Patrick Fitzgerald's CSS Positioning in Ten Steps is a quick way to learn about most CSS positioning.

rsync in 10 seconds

It's like cp and the following produce the same results:

cp -a /tmp .
rsync -a /tmp .

Note the following benefits:

• mirrors a directory along with meta data (like cp -a)
• uses diffs to save time when run again
• can cross servers over SSH

Here's the syntax for that SSH option:

rsync -a -e 'ssh' user@host:/tmp .

Which copies /tmp on a remote host over SSH like scp.

Read Jeremy Mates's tutorial next.

python http auth

Found useful article on HTTP Authentication with Python.

Python Emacs Whitespace Annoyance

Python whitespace slowed me down when I tried to add a conditional around something similar to the following block in emacs22:

cat = ""
list = ['foo', 'bar', 'baz']
for thing in list:
    cat += thing + ','
print cat

I indented everything the entire block under the conditional and ended up with:

if (1):
    cat = ""
    list = ['foo', 'bar', 'baz']
    for thing in list:
        cat += thing + ','
        print cat

instead of (the last line became part of the loop):

if (1):
    cat = ""
    list = ['foo', 'bar', 'baz']
    for thing in list:
        cat += thing + ','
    print cat

I'm used to just doing an M-x indent-region when coding and not having to see the details of the block I'm indenting. This is handy when I'm wrapping a block in a conditional and don't want to think of the details of the block at that moment. Perhaps this has been solved so I'll read more about Python Mode.

[FIX]: Don't indent the region, shift the region. Shifting preserves other indentations.

C-c <: Shift the region to the left
C-c >: Shift the region to the right 

Redhat PHP/MySQL on top of mess

If a prior admin has installed php4/mysql4 on a rhel4 box and wanted to upgrade both to version 5 but didn't know about altering the channel subscription to Web Application Stack 1.0 Beta and instead installed them from source and simply had init start /usr/local/bin/mysqld and just edited /etc/httpd/conf.d/php.conf to include a module he dropped into /etc/httpd/modules/libphp5.so (yuck).... then you can do the following...

* Remove the old packages first
up2date doesn't have a remove option like yum (or better apt) so you'll have to dig around. rpm -ev everything that rpm -qa | grep -i php and rpm -qa | grep -i mysql return. Then up2date -u

* Subscribe to Web Application Stack 1.0 Beta
Install the newer versions: up2date -i mysql-server php php-mysql php-ldap php-gd php-xml. Your /etc/httpd/conf.d/php.conf should now just contain a reference RedHat's /etc/httpd/modules/libphp5.so.

* Fix MySQL
If the prior admin installed from source with the default options your data and binaries should be in different locations and can co-exist as you move them:

RedHat: /var/lib/mysql
local: /usr/local/mysql/data

Backup your databases with the old server running:

/usr/local/mysql/bin/mysqldump -u root -p --all-databases > backup.sql

Stop the old one and start the new one. Then drop in the new data. By default the root user shouldn't have a password:

/usr/bin/mysql -u root < backup.sql

Edit /etc/my.conf and update "old_passwords=1" with "old_passwords=0".

RHEL4/5's MySQL comes by default with the old_passwords flag in /etc/my.conf set to true from a default install (via up2date or yum). A new install doesn't have this value set and just uses the new password hash function. As a result the passwords that were created originally used the newer 41 byte hash and not the older 16 byte hash. Thus when you try to import those 41 byte passwords they will be stored but won't work since the system will expect 16 byte passwords until you update the flag and restart MySQL. For more information on MySQL's password hashing see: http://dev.mysql.com/doc/refman/6.0/en/password-hashing.html.

LVM Confused from Unzoned LUN

I recently had a situation where I confused LVM on a test server.

I had zoned a SAN to serve a block device and added it to the LVM. When I was done with the device I umounted it unzoned it. My mistake: not removing it from the LVM. I should have done:

• lvremove
• vgremove
• pvremove

LVM was confused. I kept seeing /dev/dm-2: read failed after 0 of 4096 at 37580898304: Input/output error or ioctl BLKGETSIZE64 failed: No such device or address errors from all of the commands above. Even though /proc/scsi/scsi didn't list the device. The following didn't help:

• vgreduce --removemissing $name
• rm /etc/lvm/.cache
• vgchange -ay

I ended up rebooting the system so that it could rescan the SCSI bus and it worked fine. I might have gotten away with this trick to rescan the SCSI bus.

Rescan the SCSI bus

Rescan the SCSI bus without rebooting

sysadmin day 2

I've been at this for over a year.

DNS Vulnerability

There was a recently DNS cache poisoning vulnerability. Debian and RedHat released timely fixes and DJB's guarantee still stands. darkoz.com has a concise explanation.

TCP Tuning

A friend sent me two interesting links on tuning TCP to maximize throughput for moving large data:

• http://fasterdata.es.net
• http://dsd.lbl.gov/TCP-tuning

MySQL multi-core performance

According to MySQL, MySQL [5.0] "is fully multi-threaded, and will make use of multiple CPUs, provided that the operating system supports them".

However, there are different ways to use the extra CPUs. Does a single query from one connection use all CPUs (Parallelism) OR does it just distribute processing per connection? In the later option a single query wouldn't run any faster but multiple variations of the query could be run at the same time depending on the number of extra CPUs.

I'm using MySQL 5.0 and a forum poster claims 5.1 might have parallelism. Neither RHEL5 nor Hardy Heron have MySQL 5.1 packages.

In my search for this information I came across the blog of a Google employee (AFAICT) who is working on MySQL. He pointed out some things he doesn't like about parallelism and suggested other things MySQL users can do to get parallel features. I also came across a PostgreSQL vs MySQL benchmark which claimed that MySQL performance only improved 37% from quadrupling the cores while PostgreSQL improved over 200%. However they didn't specify what queries they ran (as mentioned above) to measure this.

Goolge Python Side Effect

While searching with the terms python dictionary for documentation on Python's dictionary data type I found Monty Python's Hungarian Dictionary skit.

OpenMoko for Mortals

OpenMoko Neo Freerunner Finally Available on July 4th

The FreeRunner is here! My current normal phone reports that I'm using GSM 850/1900 with AT&T. The FreeRunner supports this frequency band. My SIM card (63698 A 1002) is reported to be working so I just need to install it to have working voice. However, I'll need to add data service like PDA Personal for Internet access. I should be able to do all of this online (without talking to anybody) in five days for $400 + $30/month.

It comes with a 512M microSD card but 8G chips are ~$40.

What programs can you run? Using the OpenMoko Emulator (video) might be the best way to test the software. Note that qtopia has also been ported.

• Standard iPhone-like stuff
  1. Media Player (video)
  2. Web Browser (video)
  3. GPS (video)
  4. Kinetic scrolling
  5. Smart Keyboard
  6. E-mail
  7. YouTube
  8. Wikipedia
  9. Picture Viewer
  10. Alarm Clock
  11. Weather
• Innovative Stuff (that you may not have without Freedom):
  1. Voice/Text Encryption
  2. Mesh Networks
  3. Low Cost Calling
  4. SMS Mashups
  5. Audio Recorder
  6. Musical Instrument Tuner
  7. Python

Update: arstechnica has a nice overview.

Serialize Python Data to a File

Found a blog post on using pickle (most Pythons have this mod by default) to store persistent data to a file.

Backup Hardware Ideas

• Dell MD3000
• iSCSI
• Nearly 1/4 the price of the DL1500
• 45TB max
• RAID levels 0, 1, 10 and 5
• 4 snapshot virtual disks
• EMC DL1500
• Fibre
• Nearly 4 times the price of the MD3000
• 36TB max
• RAID6
• Do SnapShots require a SnapView license?
• Data Deduplication for only certain types of file systems

No System Beep

I found the following file on my laptop recently:

I'm traveling by plane from San Jose to Philadelphia. It's just me and my laptop (no network) so I'm poking around for things to do.

No System Beep:

I've got some audio files which I'm playing through my headphones but every time I type C-g in emacs a very loud beep hurts my ears...

 sudo xset b off

ahhhh. OK now back to my wanderings.

Perl: hashes of unknown elements

#!/usr/bin/perl
# Filename:                unpack.pl
# Description:             Unpacks hash of unknown types
# Supported Langauge(s):   Perl 5.8.x
# Time-stamp:              <2008-06-26 21:47:59> 
# -------------------------------------------------------- 
use strict;
# This is an array of strings
my @array = ("arr1", "arr2");
# This is a "predicable" hash, each string maps to a string:
my %hash = (
    'key0' => 'val0',
    'key1' => 'val1',
);
# This is an "unpredictable" hash:
my %mixed = (
    'key0' => 'string', 
    'key1' => \@array, 
    'key2' => \%hash,
);
# Note that I had to use '\' to indicate that I'm building 
# this hash with references to the previous array and hash. 
# I.e. not the array or hash itself, but a reference to it. 
# 
# I say this hash is unpredictable because it might seem 
# tricky recurse throuh it if you didn't know the mapping: 
# if key0 mapped to @array and key1 mapped to %hash etc. 
# -------------------------------------------------------
# Perl's dumper works: 
use Data::Dumper;
print Dumper(\@array);
print Dumper(\%hash);
print Dumper(\%mixed);
# -------------------------------------------------------
# Let's recursively loop through each:
print "Array:\n";
foreach my $str (@array) {
    print "\t" . $str . "\n";
}
# -------------------------------------------------------
print "Hash:\n";
foreach my $key (keys %hash) {
    print "\t" . $key . " --> " . $hash{$key} . "\n";
}
# -------------------------------------------------------
print "\nMixed:\n";
foreach my $key0 (keys %mixed) {
    print "\t" . $key0 . " => ";
    my $unknown = $mixed{$key0};
    if (ref($unknown) eq 'HASH') {
 print "{";
 foreach my $key1 (keys %$unknown) {
     # dereference with $$
     print $key1 . " => " . $$unknown{$key1} . ", ";
 }
 print "}";
    }
    elsif (ref($unknown) eq 'ARRAY') {
 print "[";
 foreach my $str (@$unknown) {
     print $str . ", ";
 }
 print "]";
    }
    else { # it's a string
 print $unknown;
    }
    print "\n";
}

LD_LIBRARY_PATH

Enjoyed an explanation on why you should not set LD_LIBRARY_PATH.

fluxbuntu

xubuntu, I think I've found someone new: fluxbuntu.

Make your own iSCSI Server

I found a tech republic article about creating your own iSCSI Server on top of the Linux kernel. The project hosted at iscsitarget.sourceforge.net looks good and Fedora has a howto which makes it look easy enough.

condor & netlogo

I had an interesting chat with someone who is looking to get blades for condor to run netlogo modeling.

Sipser's Theory of Computation

I stayed up late reading Sipser's Introduction to the Theory of Computation looking for high level connections between chapters. When I woke up I remembered that:

Finite Automata (FA) can recognize formal languages. A Deterministic Finite Automata's (DFA) next state for a given input is certain but a Non-deterministic Finite Automata's (NFA) is not: it might have two transitions for the same state. An algorithm exists to convert NFA into DFA. FA can recognize regular languages and an algorithm exists to convert FA into regular expressions. However FA have trouble with non-regular languages like {(0^n|1^n) for all n > 0}. Push down automata (PDA) are like finite automata but they have an infinite stack which they can use to recognize said language. We can describe languages by generating them from a context free grammar (CFG) OR recognizing them with a PDA. A CFG and PDA are equivalent. The church-turing thesis precisely defines what an algorithm is such that it must be computable by the lambda calculus or a turing machine; which are equivalent. A turing machine is like a PDA but has an infinite array for random access, which is less restrictive than a stack, so it is more powerful. If a turing machine will stop while carrying out an algorithm then the algorithm is decidable. The halting problem is undecidable.

O'Reilly's Learning Python

Google Books has a lot of O'Reilly's Learning Python. I found the following useful for getting myself back into Python:

• Dynamic Typing: variables aren't typed, objects are:
  • variables are entries in a system table, with spaces for links to objects
  • objects are pieces of allocated memory, with enough space to represent the values for which they stand
  • references are automatically followed pointers from variables to objects
  "variables are always pointers to objects, not labels of changeable memory areas: setting a variable to a new value does not alter the original object, but rather causes the variable to reference an entirely different object."
• Modules: highest-level program organizational unit which typically correspond to other python files. import finds the module's file, compiles it into byte code and runs the module's code to build the objects it defines (it's not just a textual substitution).

  Python forbids import foo.py or import /bar/foo.py. A platform independent module search path (print sys.path) abstracts this away. An import foo will use foo.py, provided it exists in the program's home directory: where the top-level program resides or where the REPL was started. To cross directories set the PYTHONPATH environment variable in your shell:

  $ cat modules/my_mod.py
  def f():
          print "I'm f"
  $ cat my_program.py
  import my_mod
  my_mod.f()
  $ cat shell_wrapper.sh
  #!/bin/sh
  export PYTHONPATH=${PYTHONPATH}:modules/
  /usr/bin/python my_program.py
  $ 
  

  In the example above running python my_program.py will fail but ./shell_wrapper.sh works. See also Python Tutorial: Modules.
• Class Coding: Python's OOP supports:
  • Inheritance: based on attribute (method or variable) lookup. E.g. in X.name expressions, X is an instance of a class with attribute name. Inheritance comes up with respect to instances of classes as well as derived classes.
  • Polymorphism: in X.method(), the meaning of method depends on the type (class) of X.
  • Encapsulation: Methods and operators implement behavior; data hiding is more of a convention than the rule
  The object self is passed as the first argument of a Method. Because classes generate multiple instances, methods must go through the self argument to get to the instance to be processed.

  The book provides a quick example of defining a class and two instances:

  $ python
  Python 2.5.2 (r252:60911, Apr 21 2008, 11:17:30) 
  [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> class FirstClass:
  ...     def setdata(self, value):
  ...             self.data = value
  ...     def display(self):
  ...             print self.data
  ... 
  >>> x = FirstClass()
  >>> y = FirstClass()
  >>> x.setdata("King Arthur")
  >>> y.setdata(3.14159)
  >>> x.display()
  King Arthur
  >>> y.display()
  3.14159
  

  It then goes on to create a second class which inherits from the first class:

  >>> class SecondClass(FirstClass):
  ...     def display(self):
  ...             print 'Current value = "%s"' % self.data
  ... 
  >>> z = SecondClass()
  >>> z.setdata(42)
  >>> z.display()
  Current value = "42"
  >>> 
  

  We used the inherited setdata() method but that the display() method was overrided.

  Python also supports operator overloading via __<HOOKS>__. For example, __init__ is the constructor hook that most classes should have. You can also get creative and support operators like + or * based on the hooks available:

  >>> class ThirdClass(SecondClass):
  ...     def __init__(self, value):
  ...             self.data = value
  ...     def __add__(self, other):
  ...             return ThirdClass(self.data + other)
  ...     def __mul__(self, other):
  ...             self.data = self.data * other
  ... 
  >>> a = ThirdClass("abc")
  >>> a.display()
  Current value = "abc"
  >>> b = a + 'xyz'
  >>> b.display()
  Current value = "abcxyz"
  >>> a * 3
  >>> a.display()
  Current value = "abcabcabc"
  >>> 
  

  Note that we're not using a setdata() method since we're just using the constructor. What's more interesting is that we're just adding and multiplying on our new object with standard operators and that they have meaning relative to the object.

It seems that Google Books will omit different pages based on a different host somewhat randomly (handy for missing pages).

Python: Iterators

Guido said: "The use of iterators pervades and unifies Python. Behind the scenes, the for statement calls iter() on the container object". But it gets better! Generators make it easy to create iterators. They use yield which calls next() behind the scenes and resumes where it left-off. Then "simple generators can be coded succinctly as expressions": So here's a loop (which is already using an iterator behind the scenes):

>>> for i in range(10):  
...     print i,
... 
0 1 2 3 4 5 6 7 8 9
>>> 

Which I'll use to define a list:

>>> [i for i in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>

Or make a generator object:

>>> (i for i in range(10))
<generator object at 0x7f5fdb38d518>
>>>

Which I can feed to a function:

>>> sum(i for i in range(10))
45
>>> 

Makes Python feel more Functional.

Programming Collective Intelligence

Amazon rather accurately recommended Programming Collective Intelligence by Toby Segaran for me. One reviewer said:

...the author describes how the most popular of all neural networks, backpropagation, can be used to map a set of search terms to a URL... to try and find the page best matching the search terms. Instead of... proving the math behind the backprop training algorithm, he instead mentions what it does, and goes on to present python code that implements the stated goal... The upside of the approach is clear -- if you know the theory of neural networks, and are not sure how to apply it (or want to see an example of how it can be applied), then this book is great for that.

Sounds like something I'd want to play with.

down for everyone?

I'm going to send this to the Help Desk: www.downforeveryoneorjustme.com

Update: Dig the recursion!

Possible Worlds and Software Bugs

When I write a program I try to think of what could go wrong. I then try to address these concerns with conditionals or try/catch blocks. This eliminates a lot of bugs early.

The method described above reminds me of an informal logic book which encourages the reader to imagine possible worlds to find counter examples in order to determine if an argument is valid. This is not rigorous but provides a gentle and intuitive introduction to logic. I don't know of a rigorous method of finding bugs in code and in some cases it's impossible so I like to use this method. Interesting that the non-bugginess of the software will be limited by the programmer's imagination [0].

User testing should flush out bugs that the programmer hasn't thought of and is important. However a programmer should take the time to try to anticipate the bugs first because:

• You might see the bug directly as you write the code that caused it instead of waiting and hoping that a blind test on a larger scale flushes it out
• It's easier to work on code that you just wrote which is fresh in your mind than old code which you need to relearn
• User testing and reporting might not get a bug fixed until it's inconvenienced a lot of users
• The bug may not get discovered until the software is run in a new domain

A friend of mine with a philosophy background said that he often finds it easy to spot bugs in other people's code. I wonder if any of this is related to what he does. I think studying philosophy helped make me a better programmer.

Footnote:
[0]
Which is probably limited by how tired the programmer is or the programmer's mood. Which in turn is probably limited by how much coffee or antidepressants the programmer has consumed.

DNS Timeline

Sean Donelan's website has some interesting content including a design considerations for data centers and a DNS timeline.

Back to Python

I haven't used Python in a while. I'm logging my re-familiarization on the REPL and Doc Strings.

REPL

I write my code in emacs and run it on the REPL instead of just doing ./my_code.py. I use execfile to load my code:

$ ls -l my_code.py 
-rwx------ 1 user user 3305 May 29 10:17 my_code.py
$ python 
Python 2.4.3 (#1, Dec 11 2006, 11:38:52) 
[GCC 4.1.1 20061130 (Red Hat 4.1.1-43)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> execfile("my_code.py")
>>> call_a_function_defined_in_my_code()
>>>

In emacs M-x py-shell will get you a local Python REPL and you can C-x o in a split screen between the REPL and your code (for mouse free development). I like to use tramp to edit remote files. To accommodate this style with the py-shell I just ssh from ansi-term and start the REPL as above.

Doc Strings

I tend to include a lot of comments when I define functions. My typical function looks like this:

# -------------------------------------------------------
def my_function():
    """
    Function:                my_function

    Description:             1-3 line description of 
                             the function... with an 
                             extra line here... 

    Parameters:
       [in] string $url      The URL for x
       [in] array $arr       An associative array of y
       [in] int $num         The amount of z
       (in) boolean $flag    An optional flag to do k
                             (Default: false)
       (in) string $name     An optional name
                             (Default: "Larry")

    Return Values:           
       interger              -1 if an error occured
                             0 if authentication failed
                             1 on sucess

    Remarks:                 
       None                  
    """
    # ---------------------------------------------------
    # first line here

I can view this with Python's __doc__ attribute:

>>> print my_function.__doc__

    Function:                my_function

    Description:             1-3 line description of 
                             the function... with an 
                             extra line here... 

    Parameters:
       [in] string $url      The URL for x
       [in] array $arr       An associative array of y
       [in] int $num         The amount of z
       (in) boolean $flag    An optional flag to do k
                             (Default: false)
       (in) string $name     An optional name
                             (Default: "Larry")

    Return Values:           
       interger              -1 if an error occured
                             0 if authentication failed
                             1 on sucess

    Remarks:                 
       None                  
    
>>>

The following elisp allows me to M-x function-template to easily place the above template in a python-mode buffer.

(defvar py-indent-offset 4
  "*Indentation increment in Python mode")

(defun function-template ()
  "creates a template for a function in:  PHP, Perl or Python"
  (interactive)
  (if (eq major-mode 'python-mode)
      (progn 
 (line line-length-default)
 (insert-string "def my_function():\n")))
  (save-excursion 
    (let ((pos (point)))
      (if (eq major-mode 'python-mode)
   (progn 
     (insert-string "\"\"\"\n")
     (insert-string "Function:                my_function\n\n"))
 (progn
   (line-no-comment line-length-default))
   (insert-string "Function:                Class::MyFunction\n\n"))
      (insert-string "Description:             1-3 line description of \n")
      (insert-string "                         the function... with an \n")
      (insert-string "                         extra line here... \n\n")
      ; (insert-string "Type:                    public\n\n")
      (insert-string "Parameters:\n")
      (insert-string "   [in] string $url      The URL for x\n")
      (insert-string "   [in] array $arr       An associative array of y\n")
      (insert-string "   [in] int $num         The amount of z\n")
      (insert-string "   (in) boolean $flag    An optional flag to do k\n")
      (insert-string "                         (Default: false)\n")
      (insert-string "   (in) string $name     An optional name\n")
      (insert-string "                         (Default: \"Larry\")\n")
      (insert-string "\n")
      (insert-string "Return Values:           \n")
      (insert-string "   interger              -1 if an error occured\n")
      (insert-string "                         0 if authentication failed\n")
      (insert-string "                         1 on sucess\n")
      (insert-string "\n")
      (insert-string "Remarks:                 \n")
      (insert-string "   None                  \n")
      (if (eq major-mode 'python-mode)
   (progn
     (insert-string "\"\"\"\n")
     (line (- line-length-default py-indent-offset))
     (insert-string "# first line here\n")
     (indent-rigidly pos (point) py-indent-offset))
 (progn
   (line-no-comment line-length-default)
   (comment-region pos (point))))
      (if (eq major-mode 'php-mode)
   (progn 
     (insert-string "\nfunction MyFunction() {\n")
     (insert-string "\n}\n\n")))
      (if (eq major-mode 'perl-mode)
   (progn 
     (insert-string "\nsub MyFunction {\n")
     (insert-string "  my ($param1, $param2, ..., $paramn) = @_;\n\n")
     (insert-string "\n}\n\n")))
      (if (not (eq major-mode 'python-mode))
   (indent-region pos (point) nil)))))

I've attempted to incorporate my style into conformance with the Python Style Guide and borrowed a little from Tim Peters' improved Emacs Python mode suggestion.

twelve xterms

I squeeze 12 xterms onto two 19inch monitors. I start them with this script:

#!/bin/sh
# each term is 330 high and 1000 wide

# left side left monitor

# col 1
xterm -fg green -bg black -font 8x13 -geom 78x23-2000+50  & 
xterm -fg green -bg black -font 8x13 -geom 78x23-2000+380 & 
xterm -fg green -bg black -font 8x13 -geom 78x23-2000+710 & 

# col 2
xterm -fg green -bg black -font 8x13 -geom 78x23-1000+50  & 
xterm -fg green -bg black -font 8x13 -geom 78x23-1000+380 & 
xterm -fg green -bg black -font 8x13 -geom 78x23-1000+710 & 

# right side left monitor

# col 3
xterm -fg green -bg black -font 8x13 -geom 78x23-750+50  & 
xterm -fg green -bg black -font 8x13 -geom 78x23-750+380 & 
xterm -fg green -bg black -font 8x13 -geom 78x23-750+710 & 

# col 4
xterm -fg green -bg black -font 8x13 -geom 78x23-0+50  & 
xterm -fg green -bg black -font 8x13 -geom 78x23-0+380 & 
xterm -fg green -bg black -font 8x13 -geom 78x23-0+710 & 

Debian OpenSSH Vulnerability

The openssh predictable random number generator bug means that I might need to push a new key out to a lot of non-Debian-based systems or I'm open to threats and even sarcastic comics. Luckily my main Ubuntu system was installed with Ubuntu 6.10 in Feb 2007 and that's when I generated my keys. I then proceeded with upgrades as they came out so I went to 7.10 and now 8.10. So, did 6.10 have this bug? No, it affected 7.04, 7.10 and 8.04. So the keys on this system should be fine. This is also supported by ssh-vulnkey. So my only bad key came from a laptop running Xubuntu. That key is only used on a Savannah project which the Savannah admins were kind enough to disable.

Update: Russ Cox has good comments on the bug.

While I'm on the topic of SSH key's I 'd like to mention that Ubuntu 8.10 enables the Gnome Keyring SSH Agent. All you have to do is SSH into one system and the agent caches your pass phrase. You might want to make sure you uncache your key before you get lunch or go home:

$ crontab -l
# m h  dom mon dow   command
  0 12 *   *   *     /usr/bin/ssh-add -d
  0 17 *   *   *     /usr/bin/ssh-add -d
$ 

seekdir and data center on wheels

These are just cool links that I found:

• A concise yet detailed explanation of a BSD seekdir bug
• Images of a mobile data center

moving log files

I want to clean up some log files but first I want to be sure they are not being written too. Otherwise I'd move the log file and the process that is writing to them (Apache) would get upset and I might interrupt a production system. Note that this is all just paranoia since I know that log files which end in a number have already been rotated.

Here are the log files which are currently open:

lsof | fgrep /var/log/httpd/ | awk {'print $9'} | sort | uniq > open.txt

Here are the log files which I intend to move:

ls -l /var/log/httpd/*.[0-9] | awk {'print $9'} | sort | uniq > move.txt

Now I compare them and get 100% diff, i.e. no intersections:

$ wc -l move.txt open.txt 
  241 move.txt
   84 open.txt
  325 total
$ diff move.txt open.txt | wc -l
327
$ 

The extra two lines come from "1,241c1,84" and "---" which diff added. So I'll move the logs:

mv /var/log/httpd/*.[0-9] /root/oldlogs/

remount,rw

If you misconfigure your /etc/fstab and your system won't boot you'll want to boot into single user mode by using 'e' to edit grub's kernel line and adding a '1' to end. You can then boot with a 'b'. You'll then have root shell. If your root filesystem is mounted read only just remount it read/write:

mount -o remount,rw /

Ubuntu LVM on Software RAID

I've been playing with LVM and software RAID on Ubuntu. Install the packages:

apt-get install lvm2 dmsetup mdadm 

After doing the above you'll want to reboot with your new initrd so that you have the kernel modules to do an lvcreate later. I've got a cheap USB device which I'll set up with software RAID1 on two partitions for fun:

$ fdisk /dev/sdb
...
Command (m for help): p

Disk /dev/sdb: 257 MB, 257949696 bytes
8 heads, 62 sectors/track, 1015 cylinders
Units = cylinders of 496 * 512 = 253952 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         505      125209   fd  Linux raid autodetect
/dev/sdb2             506        1010      125240   fd  Linux raid autodetect  

Create the RAID1 device from the partitions:

$ mdadm --build /dev/md0 --level=1 --raid-devices=2 /dev/sdb{1,2}
mdadm: array /dev/md0 built and started.
$ 

Make a file system on the new device while I watch the RAID sync:

$ mkfs -j /dev/md0 &
$ while [ 1 ]; do cat /proc/mdstat; sleep 4; done
Personalities : [raid1] 
md0 : active raid1 sdb2[1] sdb1[0]
      125209 blocks super non-persistent [2/2] [UU]
      [==================>..]  resync = 91.0% (114944/125209) finish=0.3min speed=506K/sec
      
unused devices: 
Personalities : [raid1] 
md0 : active raid1 sdb2[1] sdb1[0]
      125209 blocks super non-persistent [2/2] [UU]
      [==================>..]  resync = 92.6% (116928/125209) finish=0.2min speed=504K/sec
      
unused devices: 
....
Personalities : [raid1] 
md0 : active raid1 sdb2[1] sdb1[0]
      125209 blocks super non-persistent [2/2] [UU]
      [===================>.]  resync = 99.1% (125120/125209) finish=0.0min speed=519K/sec
      
unused devices: 
Personalities : [raid1] 
md0 : active raid1 sdb2[1] sdb1[0]
      125209 blocks super non-persistent [2/2] [UU]

Mount the new filesystem to test it. Then umount it.

$ mount /dev/md0 /mnt/data/
$ df -h /mnt/data/
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              119M  5.6M  107M   5% /mnt/data
$ umount !$

Add the RAID to an LVM by creating a physical volume, a volume group then a logical volume:

pvcreate /dev/md0
vgcreate volgroup00 /dev/md0
lvcreate -n logvol00 -l 30 volgroup00

Remember physical volumes get added to volume groups and then we add logical volumes so I'm going through the above steps quickly to make it clear how the above are linked together. You can {pv,vg,lv}display at each step of the above as appropriate (that's how I got 30 extents). Make a filesystem and then mount your volume:

mkfs -j /dev/volgroup00/logvol00
mount /dev/volgroup00/logvol00 /mnt/data/

Now we want to extend it in a very inconvenient way (for sadism). We'll undo everything and build it again but with two smaller RAID1 devices... it's funny if you pretend you're in a Monty Python script. Note the LVM stack going back down in reverse:

lvremove /dev/volgroup00/logvol00
vgremove volgroup00
pvremove /dev/md0

Disassemble your RAID device and verify that it's gone:

mdadm -S /dev/md0
cat /proc/mdstat

Finally go back to fdisk and delete your partitions and rebuild them in quaters:

$ fdisk -l /dev/sdb

Disk /dev/sdb: 257 MB, 257949696 bytes
8 heads, 62 sectors/track, 1015 cylinders
Units = cylinders of 496 * 512 = 253952 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         253       62713   fd  Linux raid autodetect
/dev/sdb2             254         506       62744   fd  Linux raid autodetect
/dev/sdb3             507         759       62744   fd  Linux raid autodetect
/dev/sdb4             760        1012       62744   fd  Linux raid autodetect
$

Now create two RAID1 devices:

$ mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb{1,2}
mdadm: array /dev/md0 started.
$ mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb{3,4}
mdadm: array /dev/md1 started.
$ 

It will be syncing md0 first and then md1, but we can go on:

$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sdb4[1] sdb3[0]
      62656 blocks [2/2] [UU]
        resync=DELAYED
      
md0 : active raid1 sdb2[1] sdb1[0]
      62592 blocks [2/2] [UU]
      [=========>...........]  resync = 48.3% (30720/62592) finish=1.0min speed=498K/sec

Add the first RAID device to a physical volume:

$ pvcreate /dev/md0
  Physical volume "/dev/md0" successfully created

but don't add the second one yet, we'll do that later to extend. Next create a volume group and then a logical volume. Then we make a file system and mount it:

vgcreate volgroup /dev/md0
lvcreate -n data -l 15 volgroup
mkfs -j  /dev/volgroup/data
mount /dev/volgroup/data /mnt/data/

Copy some awesome music to your new LVM (What's LVM practice without early 90's metal?):

cp ~/tunes/disincarnate/Dreams\ of\ the\ Carrion\ Kind/* /mnt/data/

You might even want to play one of the songs on your device as you extend the volume. Add the other RAID device as an available physical volume for LVM and then extend volgroup to use it:

pvcreate /dev/md1
vgextend volgroup /dev/md1

vgdisplay will show that you have more space. Note also the current size of /mnt/data:

$ vgdisplay | grep 15
  Alloc PE / Size       15 / 60.00 MB
  Free  PE / Size       15 / 60.00 MB
$ df -h /mnt/data | grep M
Filesystem            Size  Used Avail Use% Mounted on
                       59M   52M  4.0M  93% /mnt/data
$ 

Extend volgroup into the extra 15 extents:

$ lvextend -l +15 /dev/volgroup/data 
  Extending logical volume data to 120.00 MB
  Logical volume data successfully resized
$

Then extend the filesystem (while listening the the music on the same filesystem):

$ resize2fs /dev/mapper/volgroup-data
resize2fs 1.40.2 (12-Jul-2007)
Filesystem at /dev/mapper/volgroup-data is mounted on /mnt/data; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/mapper/volgroup-data to 122880 (1k) blocks.
The filesystem on /dev/mapper/volgroup-data is now 122880 blocks long.

$

Note the new size:

$ df -h /mnt/data | grep M
Filesystem            Size  Used Avail Use% Mounted on
                      117M   52M   60M  47% /mnt/data
$

Now I've got a USB device with four partitions and each pair is in a RAID1 as well as the member of an LVM full of obscure metal. What else should one have in his pocket?

rhel xen

I've played with xen on Ubuntu and now I'm going to try it the RedHat way. So far I've installed my system with the virtualization option checked. This option seems to install tools like virt-manager and the xen kernel:

$ uname -a
Linux xen0.domain.tld 2.6.18-53.el5xen #1 SMP Wed Oct 10 
16:48:44 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

I'm now reading the RedHat's Virtualization Guide.

extra swap files

It's easy to add extra swap space. Create some 2G blob files (assuming 32 bit):

 dd if=/dev/zero of=/var/xswap/xswap1 bs=1M count=2048

Make them swap files:

 mkswap /var/xswap/xswap1

Tell the OS to swap to them:

 swapon /var/xswap/xswap1

If "swapon -s" shows that it's working, then update /etc/fstab:

/var/xswap/xswap1  swap  swap  defaults  0 0

You can do this for as many swap files as you have disk for. As of the 2.6 kernel swap files are just as fast a swap partitions. So, the flexibility of a swap file is a better choice even with LVM.

things to learn

I would like to know more about the following: udev tune2fs kickstart pam_ldap xm

Darwin Streaming Server on GNU/Linux

Darwin Streaming Server is an Apple Public Source Licensed version of QuickTime Streaming Server technology for streaming QuickTime and MPEG-4 media to clients with RTP and RTSP from a GNU/Linux host. You can untar it in /opt and simply run "./Install". After setting an admin password the web admin interface will be running on port 1220. Then you can point a Quicktime client at:

 rtsp://server.domain.tld:7070/sample_300kbit.mov

Note that it's easy to overlook that you need to use port 7070. Using just the RTSP protocol with no port numbers seems to direct me to port 554 which is easy to confuse as the correct port when searching for network activity from a client (123.456.8.9):

$ netstat -an | grep tcp | fgrep 123.456.8.9
tcp        0      0 123.456.7.8:554            123.456.8.9:1733
    CLOSE_WAIT
tcp        0      0 123.456.7.8:554            123.456.8.9:1735
    CLOSE_WAIT
tcp        0      0 123.456.7.8:80             123.456.8.9:1736
    TIME_WAIT
$ fuser -n tcp 554
here: 554
554/tcp:             13052
$ ps axu | grep 13052
qtss     13052  0.0  0.0 70188 5144 ?        Sl   16:23   0:00
/usr/local/sbin/DarwinStreamingServer
root     13316  0.0  0.0  4224  660 pts/2    S+   17:13   0:00 grep 13052
$ 

Note that the above shows what I think to be reasonable troubleshooting for seeing if a desired service is listening on a port but it mislead me. I tried opening the above in Totem but I would need restricted codecs.

twitter

Some of my friends are using twitter. I'm going to try leaving gTwitter up. I see there's a python-twitter so perhaps I'll write a command line twitter for fun.

fuser

fuser - identify processes using files or sockets. Handy:

• Kick users off of your system. Use 'w' to see what /dev/pts/# they have and then kill that process:

  w
  fuser -k /dev/pts/1
  

• Find out who's using a service by port. E.g. apache processs:

  $ fuser -n tcp 80
  here: 80
  80/tcp:     22502 22505 22506 22507 22508 22509 22510 22511 22512 23548 27933
  

  A lot like ps axu | grep httpd but while loops are like for loops too. Obvious next steps... who's using SSH? fuser -n tcp 22
• Determine why a file system is busy

OSSTMM

Open Source Security Testing Methodology Manual

Cisco SMTP Inspection

I mentioned earlier about strange sendmail bugs. Turns out it was a firewall issue. Cisco firewalls have a problem with ESMTP transactions if fixup smtp (mailguard) is enabled. The IOS command no fixup protocol smtp 25 should disable it.

rhel5 mod_ldap

I just got mod_ldap working with my OpenLDAP server on a RHEL5 system. It was harder than it had to be. The following did the trick using the vanilla Apache from RHEL5:

AuthName "your name here"
AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPUrl ldap://ldap.domain.tld:389/o=options
AuthLDAPBindDN cn=servicedn,o=tld
AuthLDAPBindPassword secret
Require user user0 user1

There are three mod_auth_ldap modules:

• mod_auth_ldap
• mod_authnz_ldap
• mod_authz_ldap

Note how the first two are official from Apache 2.0 or 2.2 respectively and conditionally. The third does not come directly from Apache. RHEL5's vanilla Apache 2.0 comes with the first two.

$ grep ldap /etc/httpd/conf/httpd.conf 
LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so

You can yum install mod_authz_ldap to drop the new module in /etc/httpd/modules/ but you'll still need to edit httpd.conf to include it. However, you don't need it for basic authentication.

Strange Sendmail Bugs

I've mentioned earlier that I have pair of DNS load balanced mail gateways. These boxen run sendmail and I've recently lowered the MX preference number in DNS so that the more powerful system will get the mail first. However I've run into a strange problem where some systems are not respecting the DNS change. It seems that the more powerful system is not accepting mail from these hosts and the less powerful system is then picking up the message instead. These systems are also running sendmail. I'm still trying to figure out why this is, but I'd like to share how I figured this out (with the help of a network-oriented friend).

In two terminals I ran the following commands:

echo -e "sent on `date`" | mail -s "test: `hostname`" me@domain.tld
tcpdump -vv -s 1500 -w sendmail_25w.txt port 25

This produced a log which consistently showed that my non-gateway MTA sent a retransmit and the more powerful gateway (mta1) tore down the connection:

16:44:28.856481 IP server.domain.tld.41432 > mta1.domain.tld.smtp: 
P 135:699(564) ack 367 win 46


16:44:29.057802 IP server.domain.tld.41432 > mta1.domain.tld.smtp: 
P 135:699(564) ack 367 win 46


16:44:29.058142 IP mta1.domain.tld.smtp > server.domain.tld.41432: 
R 2420026399:2420026399(0) win 0

However it had no problem building the connection with mta0 and sending the mail. Why did the non-gateway MTA resend the extra packet and why did the gateway MTA reject it?

My friend points out that Jon Postel would say the gateway MTA was wrong as per RFC793 section 2.10, Robustness Principle. Also, why would mta0 accept it and mta1 reject it?

Here's that same data but in a screen capture from wireshark. I couldn't resist overstriking the image. Note the three red-dots showing the re-transmit: Note that my non-gateway MTAs are using vanilla sendmail from RHEL4/5:

$ /usr/sbin/sendmail -v -d0.1 < /dev/null | head -1
Version 8.13.1
...
$ /usr/sbin/sendmail -v -d0.1 < /dev/null | head -1
Version 8.13.8

When looking at mta1 I see it rejecting the connection with an I/O error:

$ fgrep me@server.domain.tld /var/log/maillog
Mar 19 15:52:44 mta1 sendmail[6255]: m2JJqidH006255: from=, size=558, 
class=0, nrcpts=1, msgid=<200803191952.m2JJqh0r018282@server.domain.tld>, proto=SMTP, daemon=MTA,
relay=server.domain.tld [123.456.78.9]
Mar 19 15:57:47 mta1 sendmail[16727]: m2JJvlnq016727: SYSERR(root): collect: I/O error on 
connection from server.domain.tld, from=
Mar 19 15:57:47 mta1 sendmail[16727]: m2JJvlnq016727: from=, size=566, 
class=0, nrcpts=1, proto=SMTP, daemon=MTA, relay=server.domain.tld [123.456.78.9]

We'll see if I figure this one out or live with mail going the wrong way for a few servers.

How to make a growable LUN

We have a SAN in our department. There have been times when some of us have needed to make certain LUNs larger and the process has been non-optimal: arrange for downtime, make a new LUN and finally move the data there and remount it.

If you have a feeling that other organizations with SANs don't do this you're probably right. The SAN offers the ability to expand LUNs without any downtime so why should we not be able to do this with the filesystem? The problem is that after you magically grow the disk behind the scenes you still need to expand the filesystem to recognize it without any downtime. The solution is to configure the raw disks not with ext3 but with ext3 on top of LVM:

• Create the LUN with the SAN's interface
• When the LUN is presented via iSCSI or HBA setup LVM
• Make ext3 on top of the LVM (not the raw device the SAN presents)

Then if you need to grow the mount point in the future:

• Have SAN do a block-level migration to grow the LUN
• Use lvextend and ext2online to have the filesystem grow into the new blocks

The one thing about LVM that some people are confused about is if I have a disk why wouldn't I just make it the max size it could be from the start? The answer is that you might add more disk later using SAN tricks and then you'll wish you had used LVM in order to grow.

In order to practice this (since you don't just want to do this on production data) you can use a USB thumb drive.

1. Make two partitions on the USB thumb drive
2. Make an LVM on the first partition
3. Make a filesystem on the LVM and store some data
4. Add the second partiton to the LVM
5. Grow the filesystem to use the second partition
6. The data should be unchanged but we'll have more capacity

I'll say that agin but with details:

Make two partitions on the USB thumb drive

$ fdisk /dev/sda

It there are no partitions:

Command (m for help): p

Disk /dev/sda: 65 MB, 65536000 bytes
3 heads, 42 sectors/track, 1015 cylinders
Units = cylinders of 126 * 512 = 64512 bytes

   Device Boot      Start         End      Blocks   Id  System

Create two partitions with 'n':

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1015, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-1015, default 1015): 507
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (508-1015, default 508): 
Using default value 508
Last cylinder or +size or +sizeM or +sizeK (508-1015, default 1015): 
Using default value 1015

Command (m for help): p

Disk /dev/sda: 65 MB, 65536000 bytes
3 heads, 42 sectors/track, 1015 cylinders
Units = cylinders of 126 * 512 = 64512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         507       31920   83  Linux
/dev/sda2             508        1015       32004   83  Linux

Command (m for help): 

Note that both got the System ID "83 Linux". You'll want to change it to type "8e Linux LVM" (you'll see this listed via 'l'). Use 't' to change a partition's system id (you'll see this listed via 'm').

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 8e
Changed system type of partition 2 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/sda: 65 MB, 65536000 bytes
3 heads, 42 sectors/track, 1015 cylinders
Units = cylinders of 126 * 512 = 64512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         507       31920   8e  Linux LVM
/dev/sda2             508        1015       32004   8e  Linux LVM

Command (m for help):

Type 'w' to write the partition to disk. For the rest of this exercise we'll work on /dev/sda1 but near the end we'll include /dev/sda2 as if it's been added after a long time and more disk became available.

Make an LVM on the first partition

To use LVM, partitions or whole disks must first be converted into physical volumes (PVs) using the pvcreate command.

$ pvcreate /dev/sda1
  Physical volume "/dev/sda1" successfully created
$

Once you have one or more physical volumes created, you can create a volume group from these PVs using the vgcreate command.

$ vgcreate lvm_test /dev/sda1
  Volume group "lvm_test" successfully created
$ 

Note that you could have added /dev/sda2 as a third argument and other partitions or physical devices as needed to create the partition with more volumes. However we want to simulate growing a partition with data on it so we'll reserve the following for later:

pvcreate /dev/sda2
vgextend lvm_test /dev/sda2

Remember, don't do the above yet. You can see what you have so far in the volume group with vgdisplay:

$ vgdisplay lvm_test
  --- Volume group ---
  VG Name               lvm_test
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               28.00 MB
  PE Size               4.00 MB
  Total PE              7
  Alloc PE / Size       0 / 0   
  Free  PE / Size       7 / 28.00 MB
  VG UUID               komlLr-rmIb-VUKH-z6Xs-Upl3-eRzY-J1tgSm
   
$

We'll use the Total PE above to create a new logical volume using all available space in lvm_test:

$ lvcreate -n lvm_test_vol0  -l 7 lvm_test
  Logical volume "lvm_test_vol0" created
$

If we had more space we could create a second logical volume but we've used it all. Verify this with vgdisplay and note that the Alloc PE and Free PE have transposed. lvcreate will create a linear allocation by default. If we had multiple disks (not partitions) and we wanted to stripe them for better retrieval then we would use "lvcreate -i2 -I4" to create two stripes and stripe size of 4 KB. You should see a new device in /dev as a result of the lvcreate:

$ ls -l /dev/lvm_test/
total 0
lrwxrwxrwx  1 root root 34 Mar 20 09:19 lvm_test_vol0 
                           -> /dev/mapper/lvm_test-lvm_test_vol0
$

We will use this device (/dev/lvm_test/lvm_test_vol0) as if it were a raw disk. LVM is inserting a layer that will allow us to grow this device later without changing the data on the filesystem.

Make a filesystem on the LVM and store some data

Make an ext3 filesystem on the new logical device and then mount it:

$ /sbin/mkfs.ext3 /dev/lvm_test/lvm_test_vol0
$ mount -t ext3 /dev/lvm_test/lvm_test_vol0 test/

Verify your new mount point and put some data on it.

$ df -h | grep test
/dev/mapper/lvm_test-lvm_test_vol0
                       28M  1.4M   25M   6% /mnt/test
$ cd /mnt/test/
$ echo "please don't break me" > file.txt

Let's put some extra data on there until it's full (this is fair use I own the CD):

$ cp ~tunes/malevolent_creation/Retribution/* .
cp: writing `./06 The Coldest Survive.mp3': No space left on device
cp: writing `./07 Monster.mp3': No space left on device
cp: writing `./08 Mindlock.mp3': No space left on device
cp: writing `./09 Iced.mp3': No space left on device
$ ll
total 24883
-rwx---r-x  1 root root 5230857 Mar 20 09:53 01 Eve of the Apocalypse.mp3
-rwx---r-x  1 root root 4172897 Mar 20 09:53 02 Systematic Execution.mp3
-rwx---r-x  1 root root 4493160 Mar 20 09:53 03 Slaughter of Innocence.mp3
-rwx---r-x  1 root root 6128950 Mar 20 09:53 04 Coronation of Our Domain.mp3
-rw-r--r--  1 root root 5324912 Mar 20 09:53 05 No Flesh Shall Be Spared.mp3
-rw-r--r--  1 root root      22 Mar 20 09:50 file.txt
drwx------  2 root root   12288 Mar 20 09:46 lost+found

So we couldn't fit the rest of the album. Let's try to fix that.

Add the second partiton to the LVM

We're now ready to add /dev/sda2 to lvm_test:

$ pvcreate /dev/sda2
    Physical volume "/dev/sda2" successfully created
$ vgextend lvm_test /dev/sda2
    Volume group "lvm_test" successfully extended
$

This simulates adding more disk provided that it presents a physical device. If we had grown a LUN on our SAN we wouldn't need do the above steps since /dev/sda2 wouldn't be getting added. Instead it would be more like /dev/sda1 had grown behind the scenes and we'd just need to go to the next step.

We'll verify that we have the extra space (you should do this for this exercise or if you've grown the LUN with your SAN):

$ vgdisplay lvm_test
  --- Volume group ---
  VG Name               lvm_test
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               56.00 MB
  PE Size               4.00 MB
  Total PE              14
  Alloc PE / Size       7 / 28.00 MB
  Free  PE / Size       7 / 28.00 MB
  VG UUID               komlLr-rmIb-VUKH-z6Xs-Upl3-eRzY-J1tgSm

$

Note that VG and PE Sizes have doubled and that the Free PE size now shows our extra available space to use this space:

$ lvextend -L+28M /dev/lvm_test/lvm_test_vol0
  Extending logical volume lvm_test_vol0 to 56.00 MB
  Logical volume lvm_test_vol0 successfully resized
$ 

You can see the change from the above with vgdisplay:

$ vgdisplay lvm_test | grep Alloc
  Alloc PE / Size       14 / 56.00 MB
$ 

Note that the mounted filesystem still doesn't see the extra space:

$ df -h | grep test
/dev/mapper/lvm_test-lvm_test_vol0
                       28M   26M   85K 100% /mnt/test
$ 

Grow the filesystem to use the second partition

So far we've added the extra disk to the LVM without umounting the filesystem. Now to we'll extend the filesystem to use the rest of logical volume. We could first umount it and be on safe side according to the LVM howto. However RedHat claims it's possible to expand both the ext3fs and GFS file systems online without bringing the system down:

For this exercise we'll extend the filesystem online. However if downtime is available it's probably better to take advantage of it and use ext2resize instead of ext2online.

Running this command produced interesting results:

$ ext2online /dev/lvm_test/lvm_test_vol0 
ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b
ext2online: ext2_ioctl: No space left on device

ext2online: unable to resize /dev/mapper/lvm_test-lvm_test_vol0
$ 

However I have some extra space:

$ df -h | grep test
/dev/mapper/lvm_test-lvm_test_vol0
                       39M   26M   12M  70% /mnt/test
$ 

It's probably realted to the filesystem being full. How full?

Mounting and remounting produces the same 39M volume and ext2resize isn't available. ext2online complains if the drive isn't mounted. Let's try freeing up some space (metal names make howto's funny):

$ rm 05\ No\ Flesh\ Shall\ Be\ Spared.mp3
$ df -h | grep test
/dev/mapper/lvm_test-lvm_test_vol0
                       39M   21M   17M  56% /mnt/test
$

Same result. Let's see what it looks like:

$ vgdisplay lvm_test
  --- Volume group ---
  VG Name               lvm_test
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               56.00 MB
  PE Size               4.00 MB
  Total PE              14
  Alloc PE / Size       14 / 56.00 MB
  Free  PE / Size       0 / 0   
  VG UUID               komlLr-rmIb-VUKH-z6Xs-Upl3-eRzY-J1tgSm
   
$ 

So all available space is used. We've gone from:

  Alloc PE / Size       7 / 28.00 MB

to:

  Alloc PE / Size       14 / 56.00 MB

So we have extra capacity, but it's not as much as I would have expected.

The data should be unchanged but we'll have more capacity

Yes:

$ diff 01\ Eve\ of\ the\ Apocalypse.mp3 ~tunes/malevolent_creation/Retribution/01\ Eve\ of\ the\ Apocalypse.mp3 

The only issue is that ext2online is a little unpredictable. So far I've seen it grow data and not corrupt it, but it hasn't grown data as much as I'd expect. If I find out more I'll share it.

Update I was able to follow a similar process online as per randombugs.com and it worked well.

FSF Weekend

I attended the FSF Meeting. I had a good time as I often do. Ben Klemens was my favorite speaker. I was sorry to not see Eben Moglen or Gerald Sussman.

It was interesting that some people seemingly new to the organization had misread the philosophy and were driving themselves mad with it. E.g. one person asked if it was unethical for him to use packets that went through routers running non-free software. The consensus was that he'd drive himself crazy worrying about that sort of thing and boycotting businesses that choose to use non-free software is silly. It's the service provider's choice and the non-freedom of their equipment is their problem. They are the victims of proprietary software, not the users. Furthermore if you're at your friend's house and want to borrow his computer to look something up and you use proprietary software you can still be a member of the Church of Emacs. Lighten up. There are better things you can do to help the FSF than seek purity like some sort of ascetic while annoying your friends.

I ran into a friend who shared some cool things that he's into: the Kinesis Advantage Pro and javascript. Everyone knows about javascript but after talking with him about what he's up to with it I realize that I was too dismissive of it in the past. I've known how handy it is when combined with web services (buzzword intentionally omitted) but I didn't know it was such a nice language in itself. It was written by a Lisper and supports higher order functions, lexical closures and other lisp similarities. He showed me some code which was dispatching event handlers within closures and I'm inspired. I'm going to learn more javascript. He said there weren't any great javascript books, especially for a lispy style, but that O'Reilly's book is decent.

I also ran into another friend who mentioned that ruby is not context-free and that its developers were unable to add a new syntax because they didn't understand the consequences of this. He said they even reported a bug to bison.

spf and mail loops

It's a great day for slightly unusual mail questions:

1. SPF
My employer has an SPF record in DNS. A third-party marketer working with a department has been setting the from address to my user's addresses. Some mail servers are actually using our SPF record and bouncing the mail back to the falsified from address. The third-party vendor is now asking me to add their IP to our SPF record. If their goal is for recipients to reply to my users, then they should use the reply-to field. I say "no".

2. Mail Loops
My mailer-daemon sent a user a mail loop: too many hops (too many 'Received:' header fields) failure notice. I sent explanations ranging from friendly to medium to rigorous.

NetScaler

A friend of mine was saying good things about NetScaler (acquired by Citrix). It's a load balancing layer-7 content switch. Their software is mainly a FreeBSD kernel module called the Core Packet Processing Engine which has been tuned to minimize unresponsiveness due to I/O interrupts . He suggested that I might want to buy three (@~$50k each) -- two in my primary data center and one in my backup -- and use it to load balance all my critical services. Apparently it can determine what servers are under utilized or not available, look at user packets and then intelligently reroute them to the optimal server. This made me uncomfortable at first since in the past I've wanted to keep the intelligence in my application. Especially for webapps that write to a database. How could I trust a switch to change the server in between the steps of a transaction? There are white papers on how this works and the Optimizing Web Applications paper has a six page explanation. My friend mentioned that I can customize it to load balance for my application. E.g. I could create specific rules by using a Perl API to make regex's it will follow when processing packets. People are really doing this kind of thing, including Google.

High Capacity Email Whitepaper

Horms has an white paper on High Capacity Email.

PEAR RHEL4

What PHP PEAR packages are installed on a default RHEL4 system?

A vanilla RHEL4 system which has had it's PHP5 installed from the Red Hat Web Application Stack 1.0 Beta (for ES v. 4 x86) channel via up2date -i php contains pear. You can see what modules are installed using the built-in pear command:

$ pear list
Installed packages, channel pear.php.net:
=========================================
Package        Version State
Archive_Tar    1.3.1   stable
Console_Getopt 1.2     stable
PEAR           1.4.6   stable
XML_RPC        1.4.5   stable

Each module, or set of PHP files, is installed in /usr/share/pear.

$ find /usr/share/pear -name \*.php  | wc -l
81

To install a module you can use pear install DB and then the find command above will return 98 files. None of this should interrupt your running LAMP server unless you are overwriting PEAR packages that are already in use. Pear also handles dependencies:

$ pear install HTML_QuickForm 
WARNING: channel "pear.php.net" has updated its protocols, 
use "channel-update pear.php.net" to update
downloading HTML_QuickForm-3.2.10.tgz ...
Starting to download HTML_QuickForm-3.2.10.tgz (101,851 bytes)
.......................done: 101,851 bytes
downloading HTML_Common-1.2.4.tgz ...
Starting to download HTML_Common-1.2.4.tgz (4,519 bytes)
...done: 4,519 bytes
install ok: channel://pear.php.net/HTML_Common-1.2.4
install ok: channel://pear.php.net/HTML_QuickForm-3.2.10

keep_awake

When I SSH into a system, sometimes my connection is dropped by the firewall. I keep my shell open with:

while [ 1 ]; do date +%S; sleep 3; done

open returns -1 EMFILE

One of my systems has been running a program to create an archive of images and ran into the following issue as reported by strace:

open("/path/index.pdf",O_WRONLY|O_CREAT|O_TRUNC, 0644)
   = -1 EMFILE (Too many open files) 

Recall that open takes a path and returns a file descriptor (or -1 on error). In this case the fix is to adjust the file descriptor limit by editing /etc/security/limits.conf and doing a "ulimit -n unlimited" so that it uses the hard limit set in said file.

/root/.dvipsrc: Permission denied

A TeX user on one of my systems mentioned the following error:

/root/.dvipsrc: Permission denied

I looked around and saw that others have this issue too, but no explanation was posted. I'm posting my explanation in hopes that it will help those searching around and not mislead them. Disclaimer: I'm no TeX expert.

So why would the programs that come along with TeX, in this case dvips, be looking for these non existent files? Well, dvips looks along a path of configuration files which were set according to its Makefile from when it was compiled. I see that our dvips came from RedHat:

$ rpm -qa | grep dvi
tetex-dvips-2.0.2-22.0.1.EL4.10

and whoever maintains the RPM probably builds it that way. When it tries to look in root's home it gets permission denied, as it should. We can make the error go away by building our own RPM or chmod'ing ~root open. The later item is not an option and it's not worth it to build our own dvips RPM just for this.

vi paste wrap

When pasting text into a xterm running vi line wraps are added. Ayman points out how to stop this:

:set paste

ext3++

The default limit for subdirectories in a single directory on ext2/ext3 filesystem is 32,000. This can be increased to a maximum of 65,500 by changing the source code for the ext3/ext2 filesystem and building a new kernel.

Preparing to build a RHEL4 Kernel

There are plenty of howtos [0] on building a RHEL4 kernel. My notes on this are:

Get the packages you'll need to build the kernel:

up2date -i kernel-devel redhat-rpm-config ncurses-devel rpm-build

Get the correct source for your current kernel. This just installs an RPM which you should install manually:

up2date --get-source kernel 
rpm -ivh /var/spool/up2date/kernel-2.6.9-67.0.4.EL.src.rpm

rpmbuild to extract and prepare the kernel sources. The prepare phase of rpmbuild will extract the source from the archive and apply RedHat patches:

cd /usr/src/redhat/SPECS/
rpmbuild -bp --target `uname -m` kernel-2.6.spec

Go to the kernel source:

cd /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9.  

Make custom configuration changes:

 make menuconfig

Note that I added support for Resierfs, JFS, XFS, NTFS (for further experimentation).

Modifying the Limit in the Kernel Source

From /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9/ go to the ext2 and ext3 header files where the 32,000 hard limit is defined:

cd include/linux/

Observe what has to be changed:

$ grep 32000 ext2_fs.h ext3_fs.h 
ext2_fs.h:#define EXT2_LINK_MAX         32000
ext3_fs.h:#define EXT3_LINK_MAX         32000

Patch your kernel by modifying the files above such that 32000 is replaced by 65500. I'm told that greater values will not work.

sed -i s/32000/65500/g ext2_fs.h ext3_fs.h

Check that sed did the trick:

$ grep 65500 ext2_fs.h ext3_fs.h 
ext2_fs.h:#define EXT2_LINK_MAX         65500
ext3_fs.h:#define EXT3_LINK_MAX         65500

Build Your New Kernel

I originally tried building an RPM for my new kernel. I ran:

rpmbuild --target=i686 -ba /usr/src/redhat/SPECS/kernel-2.6.spec

and after waiting 5 hours for five types of Kernel RPMs (i686, smp, hugmem and xen) to build on a 1Ghz system and booting the new i686 kernel I found that my patch as above was undone. I.e. the grep returned 32000, not 65500. The new kernel also failed to support more than 32,000 subdirectories. Thus, I'm building my kernel with make.

Go to the original source directory:

cd /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9/

Start building:

make
make modules_install
make install

The new kernel, initrd, and system.map will all be copied into /boot/ and the new kernel should appear in GRUB as "2.6.9-prep" on the list of kernels to boot.

  CHK     include/linux/version.h
make[1]: `arch/i386/kernel/asm-offsets.s' is up to date.
  CHK     include/linux/compile.h
Kernel: arch/i386/boot/bzImage is ready
sh
/usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9/arch/i386/boot/install.sh
2.6.9-prep arch/i386/boot/bzImage System.map ""

In /etc/grub.conf, /vmlinuz-2.6.9-67.0.1.EL was changed from 0 to 1. But 1 remained the default. Changed to 0 manually. The 65500 in the header files remain. Reboot.

Test the Extended Capabilities

When the system comes back online check that you have your kernel:

 
$ uname -r
2.6.9-prep

Make an ext3 file system on some device. In this case I'm testing on a 64M USB thumb drive. Since the amount of inodes you can have is a function of the size of your disk, we need to pay attention to how many blocks per inode we have. Thus, I'm passing mkfs some options to minimize the bytes:inode ratio as well as make the blocks as small as possible.

$ /sbin/mkfs.ext3 -b 1024 -i 1024 /dev/sda1
mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
63744 inodes, 63724 blocks
3186 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=65273856
8 block groups
8192 blocks per group, 8192 fragments per group
7968 inodes per group
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345

I'm using the smallest possible block size for ext3 (must be 1024, 2048 or 4096). Note that with XFS, the block size can theoretically be any power-of-two multiple of 512 bytes up to 64KB.

I'm also passing "-i bytes-per-inode". As per the man page:

Specify  the  bytes:inode  ratio.   mke2fs creates an inode for
every bytes-per-inode bytes of space on the disk.   The  larger
the  bytes-per-inode  ratio,  the fewer inodes will be created.
This value generally shouldn't be smaller than the blocksize of
the  filesystem,  since  then too many inodes will be made.  Be
warned that is not possible to expand the number of inodes on a
filesystem after it is created, so be careful deciding the cor-
rect value for this parameter.

Since the ratio shouldn't be smaller than the blocksize I'm setting it to the lowest possible value; equal to the blocksize. Note that there is also a -N option to pass the number-of-inodes, as per the man page:

       
overrides  the default calculation of the number of inodes that
should be reserved for the filesystem (which is  based  on  the
number  of  blocks and the bytes-per-inode ratio).  This allows
the user to specify the number of desired inodes directly.

However, I'm going to let mkfs compute the inode number based on the best blocksize and bytes:inode ratio for what I want to do.

Mount your new file system:

mount -t ext3 /dev/sda1 /mnt/usb/

and see how many inodes you have available:

$ df -i /mnt/usb/
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1              63744      11   63733    1% /mnt/usb

Note the important difference in the number of available inodes on this small device given the mkfs options. If I had used a 4096 block size and the standard bytes:inode ratio I would have the folllowing and I couldn't even cary out a meaningful test:

$ df -i /mnt/usb/
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1              15936    8804    7132   56% /mnt/usb

Finally, use a script to see how many subdirectories you can fit:

#!/usr/bin/perl
$num_dirs = 63743;
system "mkdir test";
for($i=0; $i < $num_dirs; $i++) {
  system "mkdir test/$i";
  print "$i\n";
}

I was able to break the 32,000 limit:

$ df -i /mnt/usb/
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1              63744   49584   14160   78% /mnt/usb
$ 

and create 49,570 subdirectories on a ext3 USB thumb drive. Note that my script failed before it finished since I used all my disk space:

$ df -h /mnt/usb/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              55M   55M     0 100% /mnt/usb

But I wasn't limited by inodes.

Footnotes:

[0]

Howtos on Building a RHEL4 kernel:
http://kbase.redhat.com/faq/FAQ_85_8254.shtm
http://voidmain.is-a-geek.net/redhat/fedora_3_kernel_build.html
http://www.jukie.net/~bart/blog/20060410102824
http://lists.us.dell.com/pipermail/linux-poweredge/2005-April/020134.html
http://www-theorie.physik.unizh.ch/%7Edpotter/howto/modules