Wednesday, February 27, 2008

Monday, February 25, 2008

PEAR RHEL4

What PHP PEAR packages are installed on a default RHEL4 system?

A vanilla RHEL4 system which has had it's PHP5 installed from the Red Hat Web Application Stack 1.0 Beta (for ES v. 4 x86) channel via up2date -i php contains pear. You can see what modules are installed using the built-in pear command:

$ pear list
Installed packages, channel pear.php.net:
=========================================
Package        Version State
Archive_Tar    1.3.1   stable
Console_Getopt 1.2     stable
PEAR           1.4.6   stable
XML_RPC        1.4.5   stable
Each module, or set of PHP files, is installed in /usr/share/pear.
$ find /usr/share/pear -name \*.php  | wc -l
81
To install a module you can use pear install DB and then the find command above will return 98 files. None of this should interrupt your running LAMP server unless you are overwriting PEAR packages that are already in use. Pear also handles dependencies:
$ pear install HTML_QuickForm 
WARNING: channel "pear.php.net" has updated its protocols, 
use "channel-update pear.php.net" to update
downloading HTML_QuickForm-3.2.10.tgz ...
Starting to download HTML_QuickForm-3.2.10.tgz (101,851 bytes)
.......................done: 101,851 bytes
downloading HTML_Common-1.2.4.tgz ...
Starting to download HTML_Common-1.2.4.tgz (4,519 bytes)
...done: 4,519 bytes
install ok: channel://pear.php.net/HTML_Common-1.2.4
install ok: channel://pear.php.net/HTML_QuickForm-3.2.10

Friday, February 15, 2008

keep_awake

When I SSH into a system, sometimes my connection is dropped by the firewall. I keep my shell open with:
while [ 1 ]; do date +%S; sleep 3; done

Wednesday, February 13, 2008

open returns -1 EMFILE

One of my systems has been running a program to create an archive of images and ran into the following issue as reported by strace:
open("/path/index.pdf",O_WRONLY|O_CREAT|O_TRUNC, 0644)
   = -1 EMFILE (Too many open files) 
Recall that open takes a path and returns a file descriptor (or -1 on error). In this case the fix is to adjust the file descriptor limit by editing /etc/security/limits.conf and doing a "ulimit -n unlimited" so that it uses the hard limit set in said file.

Tuesday, February 12, 2008

/root/.dvipsrc: Permission denied

A TeX user on one of my systems mentioned the following error:
/root/.dvipsrc: Permission denied
I looked around and saw that others have this issue too, but no explanation was posted. I'm posting my explanation in hopes that it will help those searching around and not mislead them. Disclaimer: I'm no TeX expert.

So why would the programs that come along with TeX, in this case dvips, be looking for these non existent files? Well, dvips looks along a path of configuration files which were set according to its Makefile from when it was compiled. I see that our dvips came from RedHat:

$ rpm -qa | grep dvi
tetex-dvips-2.0.2-22.0.1.EL4.10
and whoever maintains the RPM probably builds it that way. When it tries to look in root's home it gets permission denied, as it should. We can make the error go away by building our own RPM or chmod'ing ~root open. The later item is not an option and it's not worth it to build our own dvips RPM just for this.

vi paste wrap

When pasting text into a xterm running vi line wraps are added. Ayman points out how to stop this:
:set paste

Monday, February 11, 2008

ext3++

The default limit for subdirectories in a single directory on ext2/ext3 filesystem is 32,000. This can be increased to a maximum of 65,500 by changing the source code for the ext3/ext2 filesystem and building a new kernel.

Preparing to build a RHEL4 Kernel

There are plenty of howtos [0] on building a RHEL4 kernel. My notes on this are:

Get the packages you'll need to build the kernel:

up2date -i kernel-devel redhat-rpm-config ncurses-devel rpm-build
Get the correct source for your current kernel. This just installs an RPM which you should install manually:
up2date --get-source kernel 
rpm -ivh /var/spool/up2date/kernel-2.6.9-67.0.4.EL.src.rpm
rpmbuild to extract and prepare the kernel sources. The prepare phase of rpmbuild will extract the source from the archive and apply RedHat patches:
cd /usr/src/redhat/SPECS/
rpmbuild -bp --target `uname -m` kernel-2.6.spec
Go to the kernel source:
cd /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9.  
Make custom configuration changes:
 make menuconfig
Note that I added support for Resierfs, JFS, XFS, NTFS (for further experimentation).

Modifying the Limit in the Kernel Source

From /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9/ go to the ext2 and ext3 header files where the 32,000 hard limit is defined:
cd include/linux/
Observe what has to be changed:
$ grep 32000 ext2_fs.h ext3_fs.h 
ext2_fs.h:#define EXT2_LINK_MAX         32000
ext3_fs.h:#define EXT3_LINK_MAX         32000
Patch your kernel by modifying the files above such that 32000 is replaced by 65500. I'm told that greater values will not work.
sed -i s/32000/65500/g ext2_fs.h ext3_fs.h
Check that sed did the trick:
$ grep 65500 ext2_fs.h ext3_fs.h 
ext2_fs.h:#define EXT2_LINK_MAX         65500
ext3_fs.h:#define EXT3_LINK_MAX         65500

Build Your New Kernel

I originally tried building an RPM for my new kernel. I ran:
rpmbuild --target=i686 -ba /usr/src/redhat/SPECS/kernel-2.6.spec
and after waiting 5 hours for five types of Kernel RPMs (i686, smp, hugmem and xen) to build on a 1Ghz system and booting the new i686 kernel I found that my patch as above was undone. I.e. the grep returned 32000, not 65500. The new kernel also failed to support more than 32,000 subdirectories. Thus, I'm building my kernel with make.

Go to the original source directory:

cd /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9/
Start building:
make
make modules_install
make install
The new kernel, initrd, and system.map will all be copied into /boot/ and the new kernel should appear in GRUB as "2.6.9-prep" on the list of kernels to boot.
  CHK     include/linux/version.h
make[1]: `arch/i386/kernel/asm-offsets.s' is up to date.
  CHK     include/linux/compile.h
Kernel: arch/i386/boot/bzImage is ready
sh
/usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9/arch/i386/boot/install.sh
2.6.9-prep arch/i386/boot/bzImage System.map ""
In /etc/grub.conf, /vmlinuz-2.6.9-67.0.1.EL was changed from 0 to 1. But 1 remained the default. Changed to 0 manually. The 65500 in the header files remain. Reboot.

Test the Extended Capabilities

When the system comes back online check that you have your kernel:
 
$ uname -r
2.6.9-prep
Make an ext3 file system on some device. In this case I'm testing on a 64M USB thumb drive. Since the amount of inodes you can have is a function of the size of your disk, we need to pay attention to how many blocks per inode we have. Thus, I'm passing mkfs some options to minimize the bytes:inode ratio as well as make the blocks as small as possible.
$ /sbin/mkfs.ext3 -b 1024 -i 1024 /dev/sda1
mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
63744 inodes, 63724 blocks
3186 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=65273856
8 block groups
8192 blocks per group, 8192 fragments per group
7968 inodes per group
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345
I'm using the smallest possible block size for ext3 (must be 1024, 2048 or 4096). Note that with XFS, the block size can theoretically be any power-of-two multiple of 512 bytes up to 64KB.

I'm also passing "-i bytes-per-inode". As per the man page:

Specify  the  bytes:inode  ratio.   mke2fs creates an inode for
every bytes-per-inode bytes of space on the disk.   The  larger
the  bytes-per-inode  ratio,  the fewer inodes will be created.
This value generally shouldn't be smaller than the blocksize of
the  filesystem,  since  then too many inodes will be made.  Be
warned that is not possible to expand the number of inodes on a
filesystem after it is created, so be careful deciding the cor-
rect value for this parameter.
Since the ratio shouldn't be smaller than the blocksize I'm setting it to the lowest possible value; equal to the blocksize. Note that there is also a -N option to pass the number-of-inodes, as per the man page:
       
overrides  the default calculation of the number of inodes that
should be reserved for the filesystem (which is  based  on  the
number  of  blocks and the bytes-per-inode ratio).  This allows
the user to specify the number of desired inodes directly.
However, I'm going to let mkfs compute the inode number based on the best blocksize and bytes:inode ratio for what I want to do.

Mount your new file system:

mount -t ext3 /dev/sda1 /mnt/usb/
and see how many inodes you have available:
$ df -i /mnt/usb/
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1              63744      11   63733    1% /mnt/usb
Note the important difference in the number of available inodes on this small device given the mkfs options. If I had used a 4096 block size and the standard bytes:inode ratio I would have the folllowing and I couldn't even cary out a meaningful test:
$ df -i /mnt/usb/
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1              15936    8804    7132   56% /mnt/usb
Finally, use a script to see how many subdirectories you can fit:
#!/usr/bin/perl
$num_dirs = 63743;
system "mkdir test";
for($i=0; $i < $num_dirs; $i++) {
  system "mkdir test/$i";
  print "$i\n";
}
I was able to break the 32,000 limit:
$ df -i /mnt/usb/
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1              63744   49584   14160   78% /mnt/usb
$ 
and create 49,570 subdirectories on a ext3 USB thumb drive. Note that my script failed before it finished since I used all my disk space:
$ df -h /mnt/usb/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              55M   55M     0 100% /mnt/usb
But I wasn't limited by inodes.

Footnotes:

[0]

Howtos on Building a RHEL4 kernel:
http://kbase.redhat.com/faq/FAQ_85_8254.shtm
http://voidmain.is-a-geek.net/redhat/fedora_3_kernel_build.html
http://www.jukie.net/~bart/blog/20060410102824
http://lists.us.dell.com/pipermail/linux-poweredge/2005-April/020134.html
http://www-theorie.physik.unizh.ch/%7Edpotter/howto/modules

Sunday, February 10, 2008

Zimbra Book

My review of:

"Zimbra: Implement, Administer, Manage" by Marty Resnick

  • ISBN-10: 1847192084
  • ISBN-13: 978-1847192080

Availabe from Amazon at: http://www.amazon.com/Zimbra-Implement-Administer-Marty-Resnick/dp/1847192084/

Summary: This is currently the only book on Zimbra. It's not bad and is certainly better than nothing. It raises things you might overlook when starting your Zimbra implementation and sets you up to ask the more difficult questions, though it doesn't answer them. Instead it answers the basic questions so that you can get up and running quickly. It's a good book to get you started quickly, but can't be your only reference.


Details:

My organization is evaluating Zimbra so I just spent the weekend reading this book cover to cover. It wasn't too difficult since it's about 200 pages and when you've gotten the general idea you don't need to read every word. E.g. just about every step starts with becoming root or the zimbra user with su and also covers details on using vi: "type 'i' to insert text". To me this is obvious and could have been trimmed. I suspect he's talking down to people who are new to running servers and doesn't want to leave them out, but perhaps a simple reference in the preface could have cut having to hear about this for every how to. Also, there are a lot of screen shots (which are great but take up space when counting pages) and some irrelevant sections. E.g. if you're installing Zimbra on RedHat, SUSE and Mac OS X you might actually read all of chapter 2. However, this doesn't need to be cut since the new deployer will probably choose one of the three example systems.

The point is; not all 200 pages are content thick. At the same time, the book could have been longer to cover things in more detail. That said it was easy to read and I know more about Zimbra than when I started so I'm still happy with it. Besides, what other Zimbra book are you going to read?

I've read some guides that try to set up a system in a more difficult configuration. E.g. "The NetBSD operating system" by Federico Lupi is a succinct introduction to Unix and he configures his example system the hard way, just to make it more instructive. E.g. he partitions the hard drive using sectors, not megabytes just to make it more instructive. This book does the opposite. There are plenty of times when he brings up a certain feature to let you know it exists and then fails to tell you how you might use it. E.g. Chapter 3, Administering Zimbra lists the attributes for a Class of Service (COS) and when it gets to Server Pool says: "...does not apply to us, as we are only using one server for Zimbra. So we could skip this tab". I quote the "So we could skip" part to give you a sense of the book's style; a little chatty. We all could have done without that sentence. This omission hit home for me since I'm particularly interested in the Server Pool feature because I want to define a class of servers to handle a class of users. E.g. all faculty get their mail on system X while all students get their mail on system Y, but both have the same domain name. Luckily this is explained in the Zimbra documentation's Multiple Server Installation so it's not that how this is done is unknown. Also, I knew I wanted to use multiple servers for a single domain, but I didn't know about Zimbra's COS term until I read this book so the book deserves credit. Here's an example of where it introduced me to something so that I'd ask question. The book didn't answer the question, but at least it inspired me to ask and made it easier for me to find the answer on my own since it introduced the terminology that other Zimbra admins will use. Also, I think this book is trying to help the new admin get a server up quickly and doing things the hard way may for teaching purposes not appeal to a majority of this book's audience. At the same time, I'd expect a book on a scalable mail server to cover how to scale the mail system.

I'll spend the rest of this review listing questions that it raised that I recommend the reader try to answer. I'll also hint as some things which you might not know about Zimbra unless you read this book.


Even More Details: (now I'm getting knit picky)

Chapter 2: Installing Zimbra

Configuring DNS, step 6, mentions creating a CNAME "which will create an alias of the server that we will use for our mail server". Aside from this being an awkward way to put it, the screen showing this contains no CNAME.

Chapter 3: Administering Zimbra

Class of Service (COS), step 7, mentions Server Pool but doesn't really explain why you might take advantage of it or how it works.

Creating a Distribution List never talks about security. For now I assume that anyone can post to a distribution list, but if I could limit Distribution List posters to users within a certain COS that would be very handy.

The zmcontrol and zmprov commands seem great. I wonder if there are more features to them.

Chapter 4: The Zimbra Web Client

Note the distinction between Contacts, the Personal and Shared Contacts and the Global Address List.

Tags are very handy and I wouldn't have taken advantage of them without this book. He makes a good case for them by showing two completely unrelated messages related by content only and saying "in our minds, we want to be able to group those messages together".

The sharing for all kinds of Zimbra objects (messages, calendars, contacts, documents) is nicely explained. The system is well designed in that it probably uses the same core object permission system. The documents are less well known. It's a page of HTML that allows me to provide notes along with a hypertext link to documents. Nicer than just a file manager view. The Zimbra Assistant is also a great feature I might have otherwise overlooked.

Chapter 5: Securing Zimbra

There's a little bit of a cop out because he decides that our example is not supporting IMAP or POP, just MAPI, Web and Mobile. That said I guess I'd choose the last three since the first two are obvious, but why should we have to cut any? As a result he doesn't talk much about IMAPs. He also doesn't talk directly about secure MAPI, though it is supported since the MAPI client really translates to a web service which can use HTTPS.

He does mention that the "MTA has built-in security and verification, so that it is a closed relay to be used by authenticated users only". However, there's no mention of which ports this runs on (25?, 587?).

Under Host-Based Firewall he lists six ports to leave open but never says what services they provide. You'd want to know this if you're supporting services on them.

  • 3930 not listed in wiki, syam-webserver?
  • 7025 LMTP (note that he never mentions what LMTP is)
  • 5800 not listed in wiki, vnc-http?
  • 5900 not listed in wiki, vnc?
  • 3895 not listed in wiki, syam-smc?
  • 3894 not listed in wiki, syam-agent?
I'm left guessing based on standard definitions of these ports and they don't seem to be Zimbra related. At least they weren't listed on the Zimbra wiki: http://wiki.zimbra.com/index.php?title=Ports

His mention of making sure /opt has 5G free before upgrading is handy.

Chapter 6: Customizing Zimbra

The POP accounts section in this chapter is not Zimbra's POP service. It's about using the web client to POP from another POP server. It stores the other account's POP credentials so that you can read that mail with a single sign on to the Zimbra web client. I wonder if it stores the credentials in plain text. That aside I can now go to one place to read my work and personal mail if I POP my gmail to the Zimbra web client. Since the web client is powerful enough that you can actually learn it better with short cuts it might be efficient to get used to one interface. The keyboard Shortcuts are great. I'm glad he takes time to explain how to set up your own short cuts to save time. E.g. I can type "v 6" to view all mail in a folder about a topic and then type ". 6" to move all highlighted messages into that folder. The "Group mail by" feature supports a threaded inbox and the "show fragments in conversation" to show more than the subject allows Zimbra to feel even more like gmail.

The Search Builder feature is cool. I like that it shows users a search string which defines the search formally and succinctly in an effort to train them to use the search better. E.g. (attachment:any from: (The Boss) is:anywhere before:3/1/2007) is a saved search and if you understand it you can vary it more quickly than if you just used the search builder view. It's nice that these saved searches can be executed three ways (Advanced Search, Navigation Pane, Shortcut).

The "Enable address for new mail notifications" feature is interesting. It let's the user supply an email address which gets notified when email arrives for another account, without forwarding the message content. This let's you wear multiple hats efficiently. Suppose there's a generic account like help@ but the person responsible for it doesn't want to leave it open all day. This gets more interesting when you add Zimbra Identities since that person could then hop into the other identity without logging out to read and reply to the message. In that context a different signature and reply-to is used for the generic account. The security of this is based on COS. So not anyone with an account could do this, but you'd have to define the COS carefully to prevent this. You can duplicate a default COS to a new COS to inherent features but this sounds like it would introduce data anomalies. It would be better if COS's could be inherited. I'm not sure I want to introduce all of this feature to my users since I didn't intend to COS on a departmental level.

Chapter 7: Zimbra and Outlook

The import wizard seems great. I love the ignore previously imported items feature. Note the migrate private appointments warning since they'll then become public, though this should be fixed by Zimbra5.5. The ZCO's address book integration with LDAP and the GAL is nice and it's nice that Zimbra supports meeting invitations too. Like Exchange Zimbra also supports sharing Outlook folders (contacts and calendars) and delegating access to other users.

Chapter 8: Zimbra on the Road

Zimbra purposely choose to mimic how Exchange interacts with smart phones so any Exchange compatible smart phone will work with Zimbra too. This section of the book covers how to enable Zimbra mobile but doesn't explain how it works. A system diagram in Chapter 1 shows an "Over the Air" Sync to a Protocol Gateway which runs on top of ZCS, on top of Tomcat, but doesn't really explain how it works. He also says that Zimbra Mobile requires an additional licence but doesn't explain how to configure it. I'm left guessing that it will be in the licence XML file. He then explains configuring a Windows Mobile and Palm device to use Zimbra Mobile. I'm sure it works like magic, but I'd at least like a high level explanation.

Chapter 9: The World of Zimlets

It's like adsense for you, not the advertiser. These things read your mail and then try to help. E.g. addresses become hyperlinks which turn into maps when hovered over. Or dates and times become hyperlinks which show your calendar when hovered over. These simple examples show how useful Zimlets are. He explains how to add Zimlets via the command line and after editing an XML file he has you reload it with zmzimletctl. He didn't explain how the command knew which Zimlet to update since he didn't provide the name in the command. I guess the XML file contains that information but I would have guessed that the user would want to know. Also he then suggests that you restart tomact but doesn't explain the user impact. I imagine that all users would have a short interruption in service if you did this. I wonder if anyone's written and an adsense Zimlet for those looking to boost revenue.

Chapter 10: Backup and Restore

By default Zimbra automatically does full backups (including LDAP) weekly and incremental backups (user data diffs) nightly and it retains data for one month. Almost idiot proof. zmbackup and zmrestore along with zmschedulebakup seem like handy commands. The book suggests backing up to a separate server but doesn't show how this is done. Saving the backup in the default /opt/zimbra/backup/ won't work for me. I imagine I'll find a way around it, but it would be nice if it was in the book. The examples for restoring individual accounts are useful.

Chapter 11: Keeping an Eye on Zimbra

He motivates discussing monitoring to maximize availability, scalability and security. I agree that you'll need to know "zmcontrol status" and "zmcontrol start $service" but I have a feeling like the availability talk would need something more than just that. While talking about scalability he mentions using Message Volume and Disk monitors. So, if I know the aggregate size in MB of all the messages received and sent by the server per hour and per day (Message Volume) as well as how much disk space is available per hour (Disk Usage) then I can predict how to scale my system? I guess, but I think I'd need to know some other things as well.

The mail queues section is handy. It has a slight typo suggesting that I use the flush queue button on the right. It's in the screen shot on the left. Perhaps there were updates and the screenshot person got out of sync. I didn't find the explanation of when to flush the queue to be as good as it could be: "when a bottleneck is created from the server experiencing too many errors sending out too many messages and therefore no messages are being sent out from the server".

The log files guide is a good start:

/var/log/zimbra.log         :: if a service is stopped
/opt/zimbra/log/audit.log   :: security and authentication
/opt/zimbra/log/mailbox.log :: mailbox activity 
Overall I agree with the author; you have to do the things in Chapter 11, but I feel there's more to it than that. How about how to setup Zimbra with Cacti or Zenoss? How about at least mentioning SNMP? How about listing all the log files for the current version after emphasizing the most popular three?

The End:

This book slowly makes the case that Zimbra seems to come very close to doing what Exchange can do. Perhaps even closer than other any other product. It also has the benefit of being cheaper, easier to administer, more extensible and more reliable. Of course it's also Open Source software with a few proprietary add-ons to keep your users with an Exchange background happy. I can imagine some organizations just using the Open Source version and doing very well. Don't let anything I say that might seem critical of the book taint your view on Zimbra.

And don't get me wrong: I'm grateful for this book. If you're considering deploying Zimbra buy it. Even if you read it quickly it will tell you everything you need to know to get started. It will help you better understand the online documentation which you will still have to read.

Thursday, February 7, 2008

truecrypt

TrueCrypt has relevant features for my organization:
  • What sounds like whole disk encryption - prompting for a password before even windows loads
  • A Mac OS X version
It does have it's own License which some Debian developers seem to find acceptable. See truecrypt.org.

Wednesday, February 6, 2008

cr.yp.to mail survey

I read Daniel J. Bernstein's 2001 Internet host SMTP server survey. I also found a nice mail server comparison.