*nix hacks

emacs usb foot switch

2012-02-01T09:09:00.002-05:00

Found an entry on emacswiki.org about foot switches. I see that xkeys now has a GPL'd SDK. I think it's time to get my feet in on the action...

DNS TTL Cheat Sheet

2012-01-24T10:14:00.002-05:00

When migrating a DNS from an old server to a new server I lower the TTL 24 hours before so that when I make the change external users get the new site within an hour instead of the old site because the DNS information is cached for 24 hours.

When lowering the TTL I end up having to ask someone or look it up enough that I'm posting this cheat sheet for myself to search in the future. So if a longer TTL is inherited from the top of the zone file and foo.tld inherits that time it would look like the following:

foo                        A       123.456.7.8

To lower it's TTL to one hour I simply insert the time with units (else it's seconds) in between the hostname and the A:

foo        1h              A       123.456.7.8

Then after reloading the zone file query that DNS server for the name:

dig @123.456.2.1 calendar.lafayette.edu

and make sure the ANSWER section contains 3600, which is the number of seconds in an hour:

;; ANSWER SECTION:
foo.tld. 3600    IN      A       123.456.7.8

svn2git

2012-01-16T17:52:00.004-05:00

Nothingmuch's Migrating from Subversion to Git worked for me for two SVN repos that I'm actively working on.

Quick notes:

yum install git-svn
git svn clone --prefix=svn/ --stdlayout --authors-file=authors.txt 
git svn-abandon-fix-refs
git svn-abandon-cleanup

This makes a clean git repo complete with tags, branches, and history. Now set the "central" repo location :

git remote add origin 
git config branch.master.remote origin
git config branch.master.merge refs/heads/master

finally, push it up to the server :

git push origin master

Mac 10.7, python-ldap, MySQL-python, homebrew

2011-12-18T17:15:00.003-05:00

I configured a Mac 10.7 machine to resume writing a Python program I had to put on hold. I needed python-ldap and found a guide on how to get it working on Lion. I was also introduced to homebrew in the process. I also found a workaround for MySQL-python on Lion.

RHEL6 latency on Dell M610: C States

2011-11-09T08:37:00.003-05:00

We ran into latency problems on Dell M610 servers using RHEL6 while RHEL5 performed perfectly. The RHEL6 stock kernel has a module called intel_idle which ignores Dell's BIOS settings to disable C-states. We've set the following in the tail of the boot line in /etc/grub.conf and so far so good.

intel_idle.max_cstate=0 processor.max_cstate=1

When I started here six years ago we had a DNS server with a similar problem solved basically the same way; with a "acpi=off" to disable ACPI.

Update: The above seems to have helped servers connected to the SAN but not one which just uses the local disk. RedHat had some interesting tweaks to try with MegaCli, hdparm, and the scheduler as they think it's IO related:

Set policy to cache data:

# MegaCli64 -LDSetProp -Cached -LAll -aAll

Enable disks' cache:

# MegaCli64 -LDSetProp EnDskCache -LAll -aAll
# MegaCli64 -LDSetProp ADRA -LALL -aALL

Enable write cache, but first verify if the controller is battery backed:

# MegaCli64 -AdpBbuCmd -GetBbuStatus -a0 | grep -e '^isSOHGood' -e '^Charger Status'

If the above results display no BBU, do not proceed.
If we have a battery backed controller, enable write cache.

# MegaCli64 -LDSetProp WB -LALL -aALL

If battery fails or is discharged, disable write cache:

# MegaCli64 -LDSetProp NoCachedBadBBU -LALL -aALL

Verify policy is Enabled:

# MegaCli64 -LDInfo -LAll -aAll |grep 'Disk Cache'

Disk Cache Policy: Enabled
Disable local device caching:

# hdparm -W0 /dev/sda

- Add the above to /etc/rc.local to persist reboots.

Edit /etc/fstab and modify the mount options, ie: 
/dev/mapper/vg_root-lv_root / ext4 defaults,nobarriers,data=writeback 1 1
...
...

The filesystem(s) would need be remounted for changes to take effect.
Next, we will change the scheduler to Deadline.

# echo 'deadline' > /sys/block/sda/queue/scheduler
# echo 200 > /sys/block/sda/queue/iosched/read_expire
# echo 500 > /sys/block/sda/queue/iosched/write_expire

- Add the above commands to /etc/rc.local to persist reboots.

KVM PCI device assignment

2011-10-31T09:10:00.004-04:00

My coworker is setting up KVM PCI device assignment to get around a problem he had with assigning more than four drives per server. Normally we'd just use clustered LVM and assign drives directly from the SAN by editing that VMs XML file contained in /etc/libvirt/qemu/ . However, this is the first time we've needed more than four drives per VM and KVM will only allow 4 drives per server since we're limited to a single disk controller.

Update: Turns out this was not a good idea since the VM would monopolize the device on the physical machine; e.g. the VM would get an HBA to the SAN.

Update 2: Using NPIV seems to be the right thing for this.

MySQL: replace into

2011-10-10T22:41:00.002-04:00

I am having one of those nights where I appreciate Shlomi Noach's REPLACE INTO: think twice. "REPLACE INTO" is not a conditional insert or update based on if the same key exists. It's better thought of as a conditional insert OR delete then insert query. This will get you if you're trying to do a field_name=field_name in your query to preserve it's value as you would do with an update since the original field_value would be lost. I re-wrote my query to do a "INSERT ... ON DUPLICATE KEY UPDATE" instead.

web.py on Mac OS X 10.7

2011-09-26T23:29:00.002-04:00

Lion comes with Python 2.7.1 and easy_install is working so all you have to do is:

$ sudo easy_install web.py
Password:
Searching for web.py
Reading http://pypi.python.org/simple/web.py/
Reading http://webpy.org/
Best match: web.py 0.36
Downloading http://pypi.python.org/packages/source/w/web.py/web.py-0.36.tar.gz#md5=3f9ee778c5c34357a0233c1f0e024d00
Processing web.py-0.36.tar.gz
Running web.py-0.36/setup.py -q bdist_egg --dist-dir /tmp/easy_install-qC65Ak/web.py-0.36/egg-dist-tmp-CjDoLl
zip_safe flag not set; analyzing archive contents...
web.application: module references __file__
web.debugerror: module references __file__
Adding web.py 0.36 to easy-install.pth file

Installed /Library/Python/2.7/site-packages/web.py-0.36-py2.7.egg
Processing dependencies for web.py
Finished processing dependencies for web.py
$

You might also be interested in Setting up PHP & MySQL on OS X 10.7 Lion to get it working with MySQL.

Claws Mail Client

2011-09-09T11:32:00.005-04:00

Thunderbird has been performing so badly that I've switched to Claws. The important features for me from Thunderibrd were satisfied by Claws:

Quick response (Thunderbird was getting too slow)
IMAP/SMTP over SSL
Threaded View
Emacs as my editor
GPG Plugin
Support for Zimbra tags
Easy view of mail headers
Some type of scripting support (Python and Perl)

It also has a calendar plugin I might try; I've been sticking with Zimbra webmail for calendars/contacts.

On Fedora you can easily install it with its plugins:

yum install claws-mail
 claws-mail-plugins-smime
 claws-mail-plugins-pgp 
 claws-mail-plugins-python

Whiptail in New York Times

2011-08-23T15:08:00.002-04:00

Whiptail was mentioned in the New York Times.

pychart & web.py

2011-08-17T11:55:00.004-04:00

I'm writing a web interface in web.py to display statistics on a migration of data that will take days to run. The migration will be done by several scripts which rsync on a loop to keep the data fresh and I expect that each run of the program will take less time than the previous run. The migration scripts are logging this and I want my web interface to display the shrinking synchronization window graphically. I used PyChart to create a graph within a web.py GET class and used PyChart's canvas and Python's StringIO to get web.py to display the image dynamically.

class pychart:
    def GET(self):
        # sample data, that will be passed as an argument (left for this demo)
        data = [(1, 6), # the first run took 6 days
                (2, 3), # the second run took 3 days
                (3, 1)] # the third run too
        import cStringIO
        from pychart import theme, axis, area, line_plot, line_style, tick_mark, canvas
        f = cStringIO.StringIO()
        can = canvas.init(f, format="png")
        theme.use_color = 1
        theme.scale_factor = 2
        theme.reinitialize()
        theme.get_options()
        xaxis = axis.X(format="/a-60/hL%d", tic_interval = 1, label="Runs")
        yaxis = axis.Y(tic_interval = 1, label="Days")
        ar = area.T(x_axis=xaxis, y_axis=yaxis, y_range=(0,None))
        plot = line_plot.T(label="Time to run", data=data,
                           line_style=line_style.red,
                           ycol=1, tick_mark=tick_mark.square)
        ar.add_plot(plot)
        ar.draw(can) 
        can.close()
        f.seek(0)
        web.header('Content-Type', 'image/png')
        return f

Aeolus todo

2011-07-22T17:34:00.002-04:00

RHEL is to Fedora as CloudForms is to Aeolus. So rather than wait to be approved for the CloudForms beta we're looking into trying Aeolus now.

Pwn Plug Wireless

2011-07-13T14:16:00.001-04:00

A friend shared this link to a commercial-grade wireless pentesting drop box with me.

python configparser

2011-07-12T16:33:00.002-04:00

I'm writing a Python program and as it grows I realize that it would be best to have certain variables in a configuration file. Python ConfigParser to the rescue! It's nice that this is a built-in. I just followed the examples and I was up and running quickly.

php:include "bar.php" 2 python:?

2011-07-10T14:30:00.003-04:00

PHP programmers are used to having code in one file they can use in another file simply by calling include. Python's module system supports more than just including files, but if you want to just include a file the following code provides an example:

> ls 
foo.py mod/
> ls mod/
bar.py
> 
> cat mod/bar.py 
def f():
    print("I am bar")
> 
> cat foo.py 
print("I am foo")
import sys
sys.path.append("mod/")
import bar
bar.f()
> 
> python foo.py 
I am foo
I am bar
>

Simple python inheritance example

2011-07-07T22:05:00.009-04:00

Python's Classes documentation has a bag class. Since I'm writing a program in which I want to do inheritance but haven't done it in a while and need a refresh, I thought I'd extend bag into a wet paper bag; which looses things you put in it nearly half the time. I came up with this:

class Bag(object):
    def __init__(self):
        self.data = []
    def add(self, x):
        self.data.append(x)
    def addtwice(self, x):
        self.add(x)
        self.add(x)

class WetPaper(Bag):
    def __init__(self):
        super(WetPaper, self).__init__()
    def add(self, x):
        from random import randint
        if (randint(1,10) > 4):
            super(WetPaper, self).add(x)

if __name__ == '__main__':
    bag = Bag()
    bag.addtwice(1)
    print "Bag:", bag.data
    wet = WetPaper()
    wet.addtwice(1)
    print "WetPaper:", wet.data

Since I'm extending Bag, defining the constructor was simplified and I didn't have to worry about how add() was implemented; I could just flip a coin with rand to see if the inherited add() should be called. I also didn't have to define addtwice() and it inherited the unreliable aspect of WetPaper's add().

When writing the above, the first thing I had to do was update the original Bag definition to a new style class descended from object. Until I did this I got a "TypeError: must be type, not classobj" error when I used super and I had to directly use the parent class name instead:

    def __init__(self):
        Bag.__init__(self)

While working on this I found Python's Super is nifty, but you can't use it. I also found out that in Python 3 self will become implicit in super calls so instead of:

  super(WetPaper, self).__init__()

I will just be able to do:

  super().__init__()

Cloud Computing Definition

2011-07-07T11:28:00.004-04:00

I read an article in the FSF Bulletin about Merlin. It contained a good definition of Cloud Computing, which I think is worth posting since there is a lot of confusion about what Cloud Computing is.

"Cloud Computing is about (a) aggregating server, network, and storage resources into a seemingly contiguous system ("the cloud"), (b) providing some kind of interface for the user to request or release these resources, and (c) making these resources network or location agnostic, so that the resources are accessible from anywhere, even in the face of system or network failures." -- Justin Baugh

Python Logging

2011-07-04T09:56:00.001-04:00

I read Doug Hellmann's blogpost on logging with Python.

zfone

2011-06-27T15:24:00.000-04:00

"Zfone is a new secure VoIP phone software product which lets you make encrypted phone calls over the Internet. Its principal designer is Phil Zimmermann, the creator of PGP."

web.py, mod_wsgi , RHEL6

2011-06-04T21:48:00.005-04:00

web.py can be run in a production environment on RHEL6 with mod_wsgi using stock packages from yum with the exception of web.py itself (which can be inserted into the system Python library).

1. Install the packages that RHEL6 provides:

 yum install httpd mod_ssl mod_wsgi mysql-devel MySQL-python

2. Insert web.py directly into RHEL's python2.6 site-packages:

cd /tmp
wget http://pypi.python.org/packages/source/w/web.py/web.py-0.35.tar.gz
tar xzf web.py-0.35.tar.gz
cd /usr/lib/python2.6/site-packages
mv /tmp/web.py-0.35/web .

3. Configure a place to host your application:

mkdir /var/www/myapp

4. Install some basic code in /var/www/myapp/code.py

import web
urls = (
        '/.*', 'hello',
        )
class hello:
        def GET(self):
                return "Hello World"
application = web.application(urls, globals()).wsgifunc()

5. Configure mod_wsgi directives for Apache in /etc/httpd/conf.d/wsgi.conf:

LoadModule wsgi_module modules/mod_wsgi.so
WSGIScriptAlias /myapp /var/www/myapp/code.py/
<Directory /var/www/tentacle/>
  Order allow,deny
  Allow from all
</Directory>

When configuring the above I used the webpy.org cookbook as a reference. Note that my previous example was development only.

web.py

2011-06-04T17:52:00.007-04:00

I tried web.py.

I like it because it's simple and I found it intuitive and easy to learn. You can set it up quickly on Fedora with:

yum install python-webpy
yum install MySQL-python

It was straight forward to follow the tutorial, except it's not obvious in the tutorial how to specify a different DB server besides the default localhost. The keyword argument is 'host' just like in MySQLdb.connect(), which is provided by the Python package MySQLdb; i.e. the MySQL-python yum package. Here's the example from the tutorial in which an external DB server is declared:

db = web.database(dbn='mysql',\
                  user='user',\
                  pw='password',\
                  db='todo',\
                  host='mysql.example.com')

Also, the PostgreSQL example translates into MySQL like this:

CREATE TABLE todo (   id serial primary key,   title text,   
                    created timestamp default now(),   
        `done` boolean default '0'    );

specifically boolean is an synonym for tinyint in MySQL so it uses not 'f' but '0' to represent false.

The output code from my tutorial is:

code.py:

#!/usr/bin/env python
# Filename:                code.py
# Description:             intro to web.py
# Supported Langauge(s):   Python 2.7.x
# Time-stamp:              <2011-06-04 18:15:06 someguy> 
# -------------------------------------------------------
import web
# -------------------------------------------------------
render = web.template.render('templates/')
urls = (
    '/', 'index',
    '/add', 'add'
)
app = web.application(urls, globals())
db = web.database(dbn='mysql',\
                  user='todo',\
                  pw='redacted',\
                  db='todo',\
                  host='mysql.example.com')
# -------------------------------------------------------
class index:        
    def GET(self):
        todos = db.select('todo')
        return render.index(todos)

class add:
    def POST(self):
        i = web.input()
        n = db.insert('todo', title=i.title)
        raise web.seeother('/')
# -------------------------------------------------------
if __name__ == "__main__":
    app.run()

templates/index.html

$def with (todos)
<ul>
$for todo in todos:
    <li id="t$todo.id">$todo.title</li>
</ul>

<form method="post" action="add">
<p><input type="text" name="title" /> 
<input type="submit" value="Add" /></p>
</form>

AI TouchBook

2011-05-23T11:07:00.003-04:00

Always Innovating (AI) sells a a tablet + pluggable keybaord that runs AIOS, Android, Ubuntu, and ChromiumOS for $550 which is open {source, hardware, community}.

goodbye openoffice, hello abiword

2011-05-15T16:20:00.004-04:00

I don't often use Word Processors. Instead I edit text. I like emacs for lots of editing and vi for quickly tweaking a configuration file. If I had to make something look nice I'd type set it with LaTeX. However, I have recently been collaborating with people sending me .doc files. I used to save them as text, edit them in emacs, and import them back into openoffice but the overhead from loss of formatting was making it not worth it so I started just using openoffice but the key-bindings were the hardest part; e.g. my muscle memory thinks that Ctrl-f moves the cursor forward and shouldn't open up a Find/Replace dialog. Openoffice refuses to accommodate vi or emacs key bindings but I found that AbiWord does (successfully using AbiWord 2.8.6). Also, AbiWord is smaller than openoffice so it's faster. I didn't have luck changing the key-bindings by just doing what's described on the AbiWord FAQ, instead I got it working as described in a linuxgazette.net mailing list.

tried spice

2011-04-25T17:25:00.003-04:00

I tried a spice desktop today from my Fedora client:

sudo yum install spice-client
spicec -h $server -p $port -w $passwd
shift-ctrl-f12

It really works. The image of my VM was less than 100M since it's stored in qcow2 format. This also allows images given to each user (one per port) to inherit software updates to the master image (requires qcow2 images to go offline).

Link: How To Safely Store A Password

2011-04-25T16:07:00.000-04:00

http://codahale.com/how-to-safely-store-a-password/

flymake & pyflakes

2011-04-07T22:12:00.004-04:00

I am writing Python using flymake in Emacs with pyflakes as described in chrism's blog entry on Flymake Mode for Emacs / Python.

cd ~/elisp
wget http://cvs.savannah.gnu.org/viewvc/*checkout*/emacs/emacs/lisp/progmodes/flymake.el?revision=1.2.4.41
mv flymake.el\?revision\=1.2.4.41 flymake.el
easy_install pyflakes

add the following to .emacs:

  (when (load "flymake" t) 
         (defun flymake-pyflakes-init () 
           (let* ((temp-file (flymake-init-create-temp-buffer-copy 
                              'flymake-create-temp-inplace)) 
              (local-file (file-relative-name 
                           temp-file 
                           (file-name-directory buffer-file-name)))) 
             (list "pyflakes" (list local-file)))) 

         (add-to-list 'flymake-allowed-file-name-masks 
                  '("\\.py\\'" flymake-pyflakes-init))) 

   (add-hook 'find-file-hook 'flymake-find-file-hook)

todo: try Mongo GridFS

2011-03-30T10:57:00.003-04:00

I've been thinking about traditional file systems mkfs'd on block devices vs distributed file systems and I would like to set some time aside to play with Mongo GridFS. It looks like given a sharded mongo installation you can use the API to save a large file and it will split it across multiple servers. This looks like a nice way to store a large data set across multiple machines as opposed to having a large file system attached to one server which requires an occasional file system check.

persistent connections within cisco firewall

2011-03-30T10:04:00.003-04:00

Our cisco firewall tears down old connections so I sometimes modify the kernel to send keep alives on tcp connections:

[root@server ~]# tail -2 /etc/sysctl.conf
# keep persistent connection (so firewall doesn't tear down)
net.ipv4.tcp_keepalive_time = 900
[root@server ~]#

prgmr.com

2011-03-29T16:58:00.002-04:00

I like the approach that prgmr.com offers to being a hosting company. Also, their prices seem very reasonable.

Disclaimer: I am not yet a customer. All I can say so far is that this looks like the kind of hosting company that I'd like for my personal server.

at

2011-03-28T14:58:00.004-04:00

Good old at is handy:

me@box:~> at now + 15 minutes
at> echo -e "remember that at(1) is your friend" | mail -s "at: `hostname`" me@tld.com
at> ^D
job 2 at Mon Mar 28 15:11:00 2011
me@box:~>

Pylons

2011-03-26T21:34:00.005-04:00

I am experimenting with pylons. Fedora made this easy:

yum install python-sqlalchemy python-pylons
paster create -t pylons HelloWorld
cd HelloWorld
paster serve --reload development.ini
firefox4 http://127.0.0.1:5000

and then I was able to directly go to the examples in Chapter 3 of Definitive Guide to Pylons.

firefox4

2011-03-26T21:27:00.004-04:00

I am now using fedora's firefox4 package as my main browser with the following plugins:

firebug
https-everywhere
tree style tab
vimperator
web developer

Professional IT Community Conference

2011-03-25T15:55:00.003-04:00

I'll be attending the Professional IT Community Sysadmin Conference

Cisco Firewall Translation, RPC Portmapper, and NFS

2011-02-22T17:33:00.007-05:00

I've posted before about problems using NFS with a Firewall which occur because of RPC. I think the correct way to solve the problem is to configure the NFS server so that RPC services like nlockmgr, rquotad, and mountd are hard coded. On a RedHat based system this comes down to uncommenting the following in /etc/sysconfig/nfs:

 RQUOTAD_PORT=875 
 LOCKD_TCPPORT=32803
 LOCKD_UDPPORT=32769
 MOUNTD_PORT=892
 STATD_PORT=662

and then configuring your firewall to allow only the above ports through in addition to RPC:tcp/udp 111 and NFS:tcp/udp 2049, for NFS.

Today I learned that if you are using a Cisco firewall it is possible to not do the above but to enable inspect rpc so that when port mapper tells the client to use random ports for services like nlockmgr, rquotad, and mountd, that the firewall will then dynamically open the same random port exclusively between the NFS client and NFS server. This surprised me as it seems odd to imagine a server asking the firewall to open a port because it wants to use it. What also surprised me is that for FWSM versions older than 3.2, this won't work if you use xlate-bypass. So, if you thought along the lines of "I don't need NAT, let's turn off xlates" and enabled xlate-bypass then you will break sunrpc. I am personally in favor of not using xlates nor sunrpc.

Freedom Box Foundation

2011-02-18T14:39:00.002-05:00

I posted about Eben Moglen's Freedom in the Cloud Talk. There is now a Feedom Box Foundation with a kickstarter project.

RedHat Cloud Offerings: Update

2011-02-10T14:28:00.002-05:00

I previously posted about RedHat Cloud Offerings and emphasized what my organization is lacking: Satellite, RHEV-M, and MRG Grid.

We're now running Satellite and will be registering all of our servers to use it. We've also been able to use Satellite alone to kickstart a VM on our KVM servers without touching the hypervisors. We're going to keep experimenting with Satellite and roll it into production very soon.

We're less interested in RHEV-H and RHEV-M since we are already happy with KVM running on RHEL and prefer to use our command line tools for management as opposed to the "vmware-like" RHEV-M GUI. We're also less interested in MRG Grid since HPC is not our focus.

We are interested in Deltacloud but in a development sense only. We don't expect it to be production-ready for a while. For now we're going to use a by-the-hour EC2 account with our development KVM cluster and see what it's like creating and migrating VMs between EC2 and our cluster (or private cloud (if you must)).

me, make, meet

2010-12-03T11:06:00.002-05:00

I evolved a variation of the calendar scheduling described in me, make, meet when I was in college. Glad to see others have a similar system. Some people thought I was weird for scheduling so much of my day.

iptables state and cisco firewalls

2010-11-30T15:56:00.002-05:00

RedHat's {G,T}UI allows one to easily run iptables to allow connections only on certain ports. So if you allowed only HTTPS the following rule would be added to /etc/sysconfig/iptables:

- -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT

The above however is for NEW connections only. However, a rule like the following is also added by RedHat's tool which allows anything related to an existing state entry to pass:

-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

This all seems reasonable but I'm having a problem where packets are in an INVALID state and are getting blocked by iptables (confirmed with an iptables log). The workaround that my colleague came up with is to drop state out of the equation by simply having:

- -A RH-Firewall-1-INPUT -m tcp -p tcp --dport 443 -j ACCEPT

The same colleague also found a blog post on Cisco PIX mangled packets and iptables state tracking which offers are much more satisfying explanation. We're looking into this further.

RedHat Cloud Offerings

2010-10-29T16:02:00.011-04:00

My organization has server clusters that host about 100 VMs and more are coming. These systems were built using RedHat's cluster manager and Xen or KVM virtualization. Do we have what some would call a private cloud? Not yet, but we're considering getting closer. Since we're already using RedHat we're looking into what they recommend first.

According to a RedHat sponsored IDC paper: "Cloud services are shared, standardized service and are available from a self-service catalog; able to scale in an 'elsatic' fashion as needed; [can be] priced on actual usage; accessible via the Internet; and supportive of published APIs".

The above definition makes me believe we're on our way to having a private cloud and RedHat's cloud offerings include everything we've done above with a few additions which would allow our clusters to satisfy the cloud definition. The additions are specifically what RedHat calls cloud management services and RedHat implements them with the following three packages.

Satellite, which provides configuration management and network services via DHCP, DNS, and PXE. Since we use RHN and have a kickstart server this doesn't seem like too much of a reach for us. I can also imagine Satellite offering the functionality I experienced when I tried out EC2. We're going to look more into satellite, spacewalk, and possibly puppet.
RHEV-M, which offers a GUI for managing VMs similar to vmWare's virtual infrastructure client or EC2's VM manager. It implements the catalog aspect of the cloud definition and allows for easy deployment. Our current method of deploying VMs is to virt-clone a golden image and our catalog consists of Ubuntu Server, Fedora 13, Win200{3,8} (yuck), and RHEL5.5. We then console into the cloned system, update it's network configuration (our clusters have several networks trunked into them) and other settings. Does RHEV-M make this easier? RHEV-M also offers live migration, high availability, load balancing, and power saving, though we have all of this from vanilla clusvcadm and just haven't yet implemented power saving or ballooned dynamic memory. We're comfortable doing this via the CLI and are considering scripting it, though we're still considering trying RHEV-M to make sure we're not missing out.
MRG Grid, which implements the API aspect defined above for portability between clouds. This feature allows one to automatically spin a VM out to EC2 as per a certification between RedHat and Amazon. We have do not yet have this feature and move VMs using tricks like dd|nc, mounting snapshots and doing virt-clones, etc. I wonder how quickly and what with service interruption a VM can be moved to another cloud and back.

evercookie

2010-10-25T09:13:00.002-04:00

Beware the evercookie.

Disable Evoluent vertical mouse thumb and bottom buttons

2010-10-20T13:17:00.002-04:00

My Evoluent vertical mouse has a thumb and bottom button. I got the mouse for ergonomic reasons, not for these buttons, and I occasionally click them by accident which causes actions I didn't intend. The following xmodmap command disables them:

xmodmap -e "pointer = 1 2 3 4 5 15 14 13 12 10 11 9 8 7 6"

bash set theory

2010-10-10T21:14:00.002-04:00

Intersection of two files:

$ cat one
a
b
c
$ cat two
b
c
d
e
$ for x in `cat one`; do grep $x two; done
b
c
$

Complement of two files:

$ cat one
a
b
c
$ cat two
b
c
d
e
$ sort one one two | uniq -u
d
e
$ sort two two one | uniq -u
a
$

After playing around for a little I found Peteris Krumins' blog who provided the complement method I'm pasting above.

udev: renamed network interface eth0 to eth1

2010-09-30T16:02:00.004-04:00

I ran into this problem today after cloning a Fedora13 VM (running KVM):

 Device eth0 does not seem to be present, delaying initialization

dmesg revealed that it had renamed eth0 to eth1:

 udev: renamed network interface eth0 to eth1

I want eth0, not eth1. Virt-cloning changes the mac address so that probably confused udev. If this happens, then you can remove the a bad rule that it made for itself and ask it to try again. In other words:

 rm /etc/udev/rules.d/70-persistent-net.rules

then reboot.

tahoe-lafs

2010-09-30T10:09:00.002-04:00

Tahoe-LAFS offers a GPL'd file system which supports decentralized storage system with provider-independent security.

pymazon

2010-09-27T17:17:00.002-04:00

You can download your non-DRM'd music with a GPL3'd client pymanzon.

facebook postmortem

2010-09-27T12:15:00.001-04:00

The facebook postmortem is technically interesting.

OpenLDAP memberOf Overlay

2010-09-27T09:14:00.002-04:00

We use LDAP groups with the memberOf attribute in OpenLDAP. We're looking into the memberOf Overlay so that we can have atomicity during updates to this attribute.

Simple idea for multifactor authentication

2010-09-20T10:17:00.002-04:00

Google Apps will soon offer a user-friendly way to implement multifactor auth with a cell phone. I'm looking forward to when it's an FOSS module that we can plug into our web-based services so that we can have extra security.

SET: Social Engineering Toolkit

2010-09-16T14:17:00.001-04:00

The SET: Social Engineering Toolkit shows how easy it is to socially engineer people. Not that I intend to do this, I just find it concerning as a sysadmin.

Published with Blogger-droid v1.5.9

gpg2, gpg-agent, enigmail

2010-09-16T09:47:00.003-04:00

A 'yum upgrade' on Fedora 13 brought in gpg2 instead of the original gpg. It is "more suited for server and embedded platforms... and more targeted to the desktop", as per the man page. Fedora is now also providing Thunderbird 3.1.3 which works nicely with enigmail 1.2.2, provided that you invoke gpg-agent.

FSF responds to Oracle v. Google

2010-09-15T11:59:00.000-04:00

FSF responds to Oracle v. Google and the threat of software patents

Life, the Universe, Everything!

2010-09-14T09:22:00.002-04:00

pi % of 1337

The Elements of Style: 0

2010-09-05T08:44:00.003-04:00

This is not directly related to Unix, but makes me think of good code.

"The approach to style is by way of plainness, simplicity, orderliness, and sincerity."

"Rich, ornate prose, is hard to digest, generally unwholesome, and sometimes nauseating."

The Elements of Style, Strunk and White.

gns3

2010-09-05T07:59:00.003-04:00

"GNS3 is a graphical network simulator that allows simulation of complex networks. It can also be used to experiment features of Cisco IOS, Juniper JunOS or to check configurations that need to be deployed later on real routers."

Installing it looks very easy as per the installation video. It also has a tutorial.

XIV in New York Times

2010-09-03T14:20:00.002-04:00

XIV article in New York Times.

MTR

2010-08-31T15:41:00.002-04:00

"MTR combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool." Seems to have come with a stock Fedora 13:

sh-4.1# rpm -qa | grep mtr
mtr-0.75-6.fc13.x86_64
sh-4.1#

It's sort of like the top of traceroute.

tcp timeout in linux kernel

2010-08-19T15:44:00.002-04:00

Some processes on our systems leave idle connections which get cut off by our external firewall after a certain amount of time as per its intentional configuration. We're attempting to fix this by modifying the tcp timeout directly in the linux kernel. My colleague believes that echo'ing a new value into:

/proc/sys/net/ipv4/tcp_keepalive_time

that is lower than the default of 7200 seconds and which correlates better with the firewall timeout will fix our problem. I'll be curious if it works.

rootsh and sudosh

2010-08-19T15:40:00.001-04:00

I'm considering using rootsh and sudosh on my servers.

Password Strength

2010-08-12T17:05:00.001-04:00

I just referred someone to John P's How I'd Hack Your Weak Passwords.

Tunnel programs

2010-08-06T16:13:00.001-04:00

Todo: try ptunnel ozymandns proxytunnel and SSH port forwaring,
Note: I'm on a bus reading a magazine with cool tricks with the above but I'm about to throw the magazine away.

Published with Blogger-droid v1.4.9

6to4 outbound relay servers, web apps, and programmer archaeologists to dig through the layers

2010-08-05T11:07:00.007-04:00

At work yesterday we discussed how backwards compatibility over time leads to so many layers that there might actually be careers for programmer archaeologists.

Our conversation started with our organization's move towards IPv6 (start early, learn early, act as a consultant for those who waited, and profit!) and got into the details of a life of a packet for a person using IPv4 to access a resource we might host over IPv6 only. Someone pointed out how a 6to4 outbound relay server would help us as recommended by Geoff Huston.

We then tried to imagine the same transaction occurring for a web app where javascript requests XML (you do this every time you click a message in gmail) and just how much has to happen. The XML was generated on the server side by a script, which ran on top of a JVM (running on top of Xen or KVM (which was live migrating via a CLVM-based cluster)) which queried a database, which was using the ext3 file system, which retrieved blocks through a fibre switch to get it from a SAN, which distributed the data over 120 SATA disks using RAID-X.... all the way back and up down through OSI but cranked back through a 6to4 Relay Service and finally applied to the DOM in a browser with more lines of code than a Unix kernel (a relic left over from the browser wars for backward compatibility with invalid HTML)... and we had to laugh... Glad I learned to deal with abstraction early.

Someone mentioned the book A Deepness in the Sky to bring up the concept of a programmer archaeologists which I found rather funny. As per the wikipedia:

"An interesting feature of the Qeng Ho's computer and timekeeping systems is the advent of "programmer archaeologists"[1]: the Qeng Ho are packrats of computer programs and systems, retaining them over millennia, even as far back to the era of Unix programs (as implied by one passage mentioning that the fundamental time-keeping system is the Unix epoch, retained for backwards compatibility). This massive accumulation of data implies that almost any useful program one could want already exists in the Qeng Ho fleet library, hence the need for computer archaeologists to dig up needed programs, work around their peculiarities and bugs, and assemble them into useful constructs."

11th Annual System Administrator Appreciation Day

2010-07-30T11:42:00.001-04:00

Here I go again: sysadminday.com.

New Mouse

2010-07-30T09:08:00.003-04:00

I just got an Evoluent VM3R2-RSB Vertical Mouse 3. It does feel more natural and I guess you never know how much it hurts your wrists to keep them flat all day, until you try this mouse. I'm still getting used to it however. It's sensitive and I keep over moving it. So far I like it and am looking forward to getting used to it.

New Sorting World Record

2010-07-29T09:53:00.002-04:00

There is a new Data sorting world record: 1 terabyte, 1 minute using TritonSort. I think this is really cool! It nicely combines things I found interesting from my good old Algorithms Class with my recent interest in building powerful computers using commodity hardware.

DMCA Exceptions

2010-07-27T10:03:00.003-04:00

More often than not there is bad news in the area of copyright that involves people loosing freedom. Yesterday, as a friend of mine pointed out, there was good news.

The Library of Congress introduced new DMCA exceptions documented by the EFF and gizmodo. Also, GE played the role of the good guys and took a case to the 5th Circuit Appeals Court against MGE UPS Systems and won a ruling from a Federal judge that: It's OK to Break DRM for Fair Use.

Network Topology Icons

2010-07-25T09:21:00.002-04:00

Cisco hosts charts of their Network Topology Icons including a PDF.

yum confused?

2010-07-22T15:50:00.003-04:00

If you run a 'yum upgrade' and end up with a message like this:

Error: failed to retrieve repodata/filelists.xml.gz from rhel-x86_64-server-5 
error was [Errno -1] Metadata file does not match checksum 
 You could try using --skip-broken to work around the problem 
 You could try running: package-cleanup --problems 
                        package-cleanup --dupes 
                        rpm -Va --nofiles --nodigest

it's likely that yum got confused. This happened to me when I changed a subscription channel, re-registered a system and tried to yum upgrade. I have been able to consistently fix it by running:

 rm -rf /var/cache/yum/*
 yum clean all
 yum update

Qlogic retry interval

2010-07-20T08:23:00.006-04:00

I posted earlier about a test with the XIV with where the retry interval was too high and IO was queued longer than it had to be on the system buffer as opposed to being written to disk (see graph). We solved this a while ago, but I forgot to post an update.

Qlogic cards have an on-board BIOS setting, which sets the path failure delay to 45 seconds. In order for multipath to perform properly on these systems, this limit should be lowered. We're currently setting the limit to 0. The options to find in the BIOS are under the advanced settings for each host controller.

You should also set the following in in /etc/modprobe.conf (Fedora/RedHat/CentOS) or /etc/modprobe.d/aliases (Debian/Ubuntu):

options qla2xxx ql2xfailover=0 
options qla2xxx qlport_down_retry=1
options qla2xxx ql2xmaxqdepth=64

Note that the maxqdepth setting on the third line isn't about fail over but about getting more IOPS with respect to the queue depth. This is a standard XIV tuning practice. If you're using Emulex cards you can set the queue depth to 64 with the following line in your modprobe configuration file:

options lpfc lpfc_hba_queue_depth=64

but it is not necessary to change a retry or failover interval.

Debian, Dell m600, XIV SAN

2010-07-19T16:55:00.016-04:00

Installing Debian on a Dell m600 for use with an IBM XIV SAN (or any multipath'd SAN) requires a few tricks.

Modified Network Install ISO

The Broadcom NetExtreme II NICs that come in the Dell blades require non-free firmware not included in Lenny. I don't fault Debian for this. The same problem exists for the Qlogic HBA's. You can get around this by making your own custom Lenny install boot ISO. J Snell's howto describes how to add firmware for the NIC, but not the HBA. However, adding the HBA is just as easy. Note that the install will go more smoothly (no ambiguity as to which disk is /dev/sda) if no LUNs are mapped by the SAN. You'll see it question where it can load that firmware (bnx2-06-4.0.5.fw and ql2400_fw.bin) during the install.

Some details on how I created my ISO are:

 
cd /mnt/
mkdir cdrom
mount -o loop debian-testing-amd64-CD-1.iso cdrom
cd /home/$USER/debian
mkdir isocopy
cp -av cdrom isocopy
cd isocopy/
 
dpkg-deb -x firmware-bnx2_0.25_all.deb nic-firmware
cp nic-firmware/lib/firmware/* isocopy/cdrom
 
dpkg-deb -x firmware-qlogic_0.14+lenny2_all.deb hba-firmware
cp hba-firmware/lib/firmware/* isocopy/cdrom
 
cd isocopy/cdrom
mkisofs -o ../modified-debian.iso -b isolinux/isolinux.bin -c isolinux/boot.cat\
 -no-emul-boot -boot-load-size 4 -boot-info-table -J -R -V disks .

Burn the modified-debian.iso to a CD-ROM and boot the server from it. Once booted to the new ISO, you need to immediately switch to another console and copy the files over before the installer looks for them:

 
mkdir lib/firmware
cp /cdrom/*.fw lib/firmware

After that's done, the Debian installer should be able to autoload the bnx2 driver without any problems. Note that you can't do the above until after the system mounts the CD ROM. It will do this automatically. You'll want to do this before you scan for network devices but after the system mounts the CD ROM. I.e. if you try to do the above before the system automounts the CD you'll have trouble.

Mount by UUID

After assigning multiple paths from the SAN the Debian system became confused as to which block device was the local disk where / should be mounted. E.g.

server:~# ls /dev/sd<tab><tab>
sda    sdaa1  sdaf   sdaj   sdan1  sdd1   sdi    sdm1   sdr    sdv1   
sda1   sdab   sdaf1  sdak   sdao   sde    sdi1   sdn    sdr1   sdw    
sda2   sdab1  sdag   sdak1  sdap   sde1   sdj    sdn1   sds    sdw1   
sda5   sdac   sdag1  sdal   sdb    sdf    sdj1   sdo    sds1   sdx    
sda6   sdac1  sdah   sdal1  sdb1   sdg    sdk    sdo1   sdt    sdx1   
sda7   sdad   sdah1  sdam   sdc    sdg1   sdl    sdp    sdt1   sdy    
sda8   sdad1  sdai   sdam1  sdc1   sdh    sdl1   sdq    sdu    sdy1   
sdaa   sdae   sdai1  sdan   sdd    sdh1   sdm    sdq1   sdv    sdz    
server:~#

Not only that, but it changed after each reboot. So, /dev/sda{1,2,5,6,7,8} was the local disk in the above scenario, but after a reboot it was /dev/sdao{1,2,5,6,7,8} etc. If this happens you'll see a message like the following after waiting for the system to boot:

Target filesystem doesn't have /sbin/init
No init found. Try passing init= bootarg

You'll then be dropped into BusyBox. You should then determine which disk is your local disk and mount it. In my case I was able to know it by the amount of partitions that it had as per above (e.g. only sdX had six partitions). Then mount that block device:

cd /
mkdir /root
mount -t ext3 /dev/sdap1 /root

Then bind /dev into your new root directory:

mount --bind /dev/ /root/dev/

Then chroot into your new root directory:

chroot /root/

then modify your /etc/fstab so you can mount the new block device. BusyBox doesn't have vi so you can use use sed:

sed s/sda/sdap/g -i /etc/fstab

You should then be able to mount everything:

mount -a

I would normally mount by label at this point, but e2label does not work on Debian the way it works on RedHat/Fedora. However, you can mount by UUID with Debian and you can find the UUID with the vol_id command:

vol_id /dev/sdao7

If you grep UUID from the above you should be able to set $UUID to the output and then modify another sed script to bring the UUID into the /etc/fstab. Once you're able to mount a copule of partitions by UUID reboot to see if you can bring the system up with / mounting correctly.

Multipath

Multipath is able to resolve the confusion from the many block devices and allow you to mount by nice names like /dev/mpathX.

apt-get install multipath-tools dmsetup
modprobe dm_mod dm-multipath dm-round-robin

You can optimize the queue depth for the XIV on Debian with Qlogic HBAs by adding the following to /etc/modprobe.d/aliases:

options qla2xxx ql2xmaxqdepth=64
options qla2xxx ql2xfailover=0 
options qla2xxx qlport_down_retry=1

I was then able to test multipath successfully.

Configuration Management and Security: Bellovin & Bush

2010-07-19T15:08:00.002-04:00

I started reading a paper on Configuration Management and Security which is good. I should remember to finish it.

IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 27, NO. 3, APRIL 2009
Configuration Management and Security
Steven M. Bellovin and Randy Bush

Abstract:
Proper configuration management is vital for host and network security. We outline the problems, especially for large-scale environments, and discuss the security aspects of a number of different configuration scenarios, including security appliances (e.g., firewalls), desktop and server computers, and PDAs. We conclude by discussing research challenges.

Cisco 2501

2010-07-15T20:46:00.008-04:00

My friend let me borrow an old Cisco 2501 Router so I could play with IOS at home. I plugged a rj45 to rs232 adapter into the 2501 and my thinkpad running Xubuntu and set up my serial console with minicom as per the Ubuntu CiscoConsole howto.

After starting minicom back up I see:

Welcome to minicom 2.2

OPTIONS: I18n 
Compiled on Apr 27 2007, 15:50:20.
Port /dev/ttyS0

               Press CTRL-A Z for help on special keys
                                                     
                                                     
                                                     
First, would you like to see the current interface summary? [yes]: AT S7=45 S0=0
% Please answer 'yes' or 'no'.

First, would you like to see the current interface summary? [yes]: yes

Any interface listed with OK? value "NO" does not have a valid configuration

Interface        IP-Address      OK?  Method    Status                 Protocol
Ethernet0        unassigned      NO   not set   up                     down    
Serial0          unassigned      NO   not set   down                   down    
Serial1          unassigned      NO   not set   down                   down    

Configuring global parameters:

  Enter host name [Router]: 
...
Use this configuration? [yes/no]: yes
Building configuration...



Press RETURN to get started!


%LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0, changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0, changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1, changed state to down
%LINK-3-UPDOWN: Interface Ethernet0, changed state to up
%LINK-3-UPDOWN: Interface Serial0, changed state to down
%LINK-3-UPDOWN: Interface Serial1, changed state to down
%LINK-5-CHANGED: Interface Ethernet0, changed state to administratively down
%LINK-5-CHANGED: Interface Serial0, changed state to administratively down
%LINK-5-CHANGED: Interface Serial1, changed state to administratively down
%SYS-5-RESTART: System restarted --
Cisco Internetwork Operating System Software 
IOS (tm) 3000 Software (CPA25-Y-L), Version 11.0(4), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-1995 by cisco Systems, Inc.
Compiled Mon 18-Dec-95 19:21 by alanyu
%SNMP-3-SOCKET: can't open UDP socket
Router>
Router>
Router>sh version
Cisco Internetwork Operating System Software 
IOS (tm) 3000 Software (CPA25-Y-L), Version 11.0(4), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-1995 by cisco Systems, Inc.
Compiled Mon 18-Dec-95 19:21 by alanyu
...
cisco 2500 (68030) processor (revision L) with 2044K/2048K bytes of memory.
Processor board ID 02425768, with hardware revision 00000000
...
Router>
Router>
Router>

All set. Looks like I'll be playing with IOS 11.0.

Mail Clients: mutt + zimbra et al

2010-07-15T18:13:00.000-04:00

We run Zimbra where I work. I've been using Thunderbird (and webmail for Calendaring) but I used to use mutt when my mail was just on a server running qmail.

A friend of mine caught me saying I was tempted to get mutt working with zimbra via imap/smtp. He beat me to it. Here's a .muttrc:

# Automatically log in to this mailbox at startup
set spoolfile="imaps://netid@incoming.domain.tld/"
# Define the = shortcut, and the entry point for the folder browser
set folder="imaps://netid@incoming.domain.tld/"
# Use a well-defined SMTP server
set smtp_url="smtps://netid@outgoing.domain.tld:465/"
# Define Zimbra-default folders
set record="=Sent"
set postponed="=Drafts"
# activate TLS if available on the server
set ssl_starttls=yes
# always use SSL when connecting to a server
set ssl_force_tls=yes
# Don't wait to enter mailbox manually
unset imap_passive
# Automatically poll subscribed mailboxes for new mail
set imap_check_subscribed
# Reduce polling frequency to a sane level
set mail_check=60
# And poll the current mailbox more often (not needed with IDLE in post
1.5.11)
set timeout=10
# keep a cache of headers for faster loading (1.5.9+?)
set header_cache=~/.hcache
# Display download progress every 5K
set net_inc=5
# Use emacs (remember to server-start)
set editor="emacsclient -t"
# Set From: address
my_hdr From: First Last

Note that you must have emacs running in the background to use "emacsclient -t". If you do this add "(server-start)" to your .emacs or use "M-x server-start".

Our discussion on mail clients lead to others worth checking out:

Droid X bricks if you try to mod it

2010-07-15T11:07:00.002-04:00

Someone passed a long a mobilecrunch.com article: Droid X actually self-destructs if you try to mod it. If this is true I'm glad I didn't hold out for one.

Fully Free Android system-image?

2010-07-10T13:54:00.000-04:00

The Android Developer's blog recently posted that: "You can now build and boot a fully open-source system image out of the box, for the emulator, as well as for Dream (ADP1), Sapphire (ADP2), and Passion (Nexus One)."

iPhone to Nexus One

2010-07-10T13:33:00.008-04:00

I switched from an iPhone to a Nexus One this week. My AT&T contract does not expire for a while. Switching was easy; once I received the phone in the mail I removed the iPhone sim card with a paper clip and inserted it into the Nexus One. After charging the Nexus One everything worked flawlessly.

EFF 20th Anniversary Animation

2010-07-09T17:03:00.000-04:00

Amusing EFF 20th Anniversary Animation by Nina Paley.

openssl the command is amazing

2010-06-30T18:10:00.004-04:00

There are lots of things you can do with the openssl command that I didn't know about:

Verify if numbers are prime:

$ openssl prime 119054759245460753
1A6F7AC39A53511 is not prime
$

Encrypt a file with your favorite cipher:

openssl list-cipher-commands

base64 encode a file

openssl enc -base64 -in file.txt

Generate a shadow-style password hash:

$ openssl passwd -1 MySecret
$1$sXiKzkus$haDZ9JpVrRHBznY5OxB82.
$

Many others. I never knew it could do some much. Thanks madboa.com.

Also, I played around with openssl while updating certificates for about 30 web servers. I was able to check that the new cert was installed correctly on all of the hosts easily looking at that host's SSL finger prints as served from Apache:

echo EOF | openssl s_client -connect $host:443 -showcerts | openssl x509 -fingerprint -noout -md5

The above fits well into a bash loop which can be run before and after you replace the certs:

for x in `cat vhosts.txt`; do 
   echo "vhost: $x";echo EOF | openssl \
     s_client -connect $x:443 -showcerts \
     | openssl x509 -fingerprint -noout \
     -md5; 
done 
| egrep "vhost|Fingerprint" > finger_prints.txt

You can then diff the finger prints files to verify that they're what you're expecting them to be.

Android Development with Clojure

2010-06-27T09:43:00.001-04:00

Android Development is now looking even more attractive: Clojure and Android with Emacs on Ubuntu.

FLOSS Smart Phones

2010-06-26T12:42:00.014-04:00

Brad Kuhn wrote an article on the state of Smart Phones with respect to Free, Libre, and Open Source Software (FLOSS) and says:

HTC Dream currently gives the most software freedom among Android/Linux deployments. It is unlikely that Google wants anything besides their applications to be proprietary. While Google has been unresponsive when asked why these hardware interface libraries are proprietary, it is likely that HTC, the hardware maker with whom Google contracted, insisted that these components remain proprietary, and perhaps fear patent suits... no detailed analysis of the Nexus One is yet available, it's likely similar to the HTC Dream.

With regard to permission to run one's own software on the device -- something central to phone development -- Brad points out:

Google is our best ally in this regard. The HTC Dream developer models, called the Android Dev Phone 1 (aka ADP1). permit the user to install any operating system on the phone, and the purchase agreement extract no promises from the purchaser regarding what software runs on the device. Google has no interest in locking you to a single carrier, but only to a single Google experience application vendor. Offering a user “carrier freedom of choice”, while tying those users tighter to Google applications, is probably a central part of their marketing plans.

Other interesting quotes include:

Community-oriented forks... [must] begin in the Android/Linux space.
A traditional “get some volunteers together and write some code” approach can achieve great advancement toward community-oriented FLOSS systems on mobile devices. Developers could initially focus on applications for the existing “mostly FLOSS” platforms of MeeGo and Android/Linux.
We need to identify the proprietary software that is important, and write free software replacements. It's catch-up work, but our community is usually successful at such tasks. So, let's get coding on mobile! (see FSF Bulletin, Issue 16, May 2010)

Brad has some additional blog entires on android and is a co-founder of the Replicant project. I'm curious about switching providers so I'm curious about the distinction between the Hero and Dream.

PostgreSQL has an IP address data type

2010-06-24T20:01:00.002-04:00

Goodbye EMC

2010-06-23T16:30:00.003-04:00

I have been getting up around 2:30 every morning once a week for the last six weeks, to work during agreed upon maintenance windows, to migrate data from two EMC Clariions to two IBM XIVs. Today was the last migration. All of the production data on both EMCs has been moved. If it breaks tomorrow, it wouldn't be so bad, though we'd loose our new development storage system. I'm very happy to be rid of depending on EMC.

xwrits

2010-06-23T15:50:00.000-04:00

todo: try xwrits.

"Linux Magazine" article on SSD file systems

2010-06-23T10:57:00.016-04:00

"Linux Magazine" recently did an article (referenced but not included in link) on choosing a file system for your SSD. I am now running Fedora 13 on an Intel X25-M as mentioned earlier in SSD experiment. I'm using ext4 and mounting with the noatime,nodiratime options. Performance is snappy. The article had the following benchmark.

                                   MBps
test                     ext2  ext4  ext4 no jrnl   btrfs   btrfs(ssd_spread)
-----------------------------------------------------------------------------
dbench -D /test 10       520   407   428            347     347
bonnie++ /test -s 2048   38    58    72             64      67

The article also had some useful links.

Foot Pedals

2010-06-21T16:42:00.003-04:00

I am looking for a USB Foot Pedal to act as a control key and an alt/meta key. Bill Clemson did this. I'll be looking around.

Multicast and Red Hat Cluster Suite

2010-06-10T21:08:00.011-04:00

Getting members of a Red Hat Cluster Suite to communicate and get quorum can be difficult. This post documents how I verified that multicast was working as I step towards setting up a cluster.

After you start your cluster see which multicast address was assigned:

[root@vserver0 ~]# grep "default multicast" /var/log/messages
Dec 27 16:51:51 vserver0 openais[6953]: [MAIN ] Using default multicast address of 239.192.104.1 
Dec 27 18:26:33 vserver0 openais[3664]: [MAIN ] Using default multicast address of 239.192.104.1 
Dec 28 04:25:28 vserver0 openais[13028]: [MAIN ] Using default multicast address of 239.192.104.1 
Dec 30 14:02:35 vserver0 openais[9533]: [MAIN ] Using default multicast address of 239.192.104.1 
[root@vserver0 ~]#

I also see that each node is listening on the same address:

[root@vserver0 ~]# netstat -an | grep 239
udp        0      0 239.192.104.1:5405          0.0.0.0:*
[root@vserver0 ~]# 

[root@vserver1 ~]# netstat -an | grep 239
udp        0      0 239.192.104.1:5405          0.0.0.0:*
[root@vserver1 ~]# 

[root@vserver2 ~]# netstat -an | grep 239
udp        0      0 239.192.104.1:5405          0.0.0.0:*
[root@vserver2 ~]#

The cluster documentation talks about configuring multicast and implies that you might have to enable it on your router. You can verify if your router is passing multicast packets between nodes with tcpdump and iperf as described in taosecurity.blogspot.com. I can reproduce the results described in the taosecurity blog in my cluster as so:

wget ftp://ftp.pbone.net/mirror/ftp.freshrpms.net/pub/freshrpms/pub/dag/redhat/el5/en/x86_64/dries/RPMS/iperf-
2.0.4-1.el5.rf.x86_64.rpm
rpm -ivh iperf-2.0.4-1.el5.rf.x86_64.rpm

I set up vserver0 as an iperf server listening on multicast address 239.192.104.1:

[root@vserver0 ~]# iperf -s -u -B 239.192.104.1 -i 1
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 239.192.104.1
Joining multicast group  239.192.104.1
Receiving 1470 byte datagrams
UDP buffer size:   126 KByte (default)
------------------------------------------------------------

Now I generate multicast traffic from vserver1.

[root@vserver1 ~]# iperf -c 239.192.104.1 -u -T 32 -t 3 -i 1
------------------------------------------------------------
Client connecting to 239.192.104.1, UDP port 5001
Sending 1470 byte datagrams
Setting multicast TTL to 32
UDP buffer size:   126 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.129 port 48770 connected with 239.192.104.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec    129 KBytes  1.06 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec    128 KBytes  1.05 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec    128 KBytes  1.05 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 3.0 sec    386 KBytes  1.05 Mbits/sec
[  3] Sent 269 datagrams
[root@vserver1 ~]#

Here is what vserver0 sees:

------------------------------------------------------------
[  3] local 239.192.104.1 port 5001 connected with 192.168.1.129 port 48770
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0- 1.0 sec    134 KBytes  1.09 Mbits/sec  1.109 ms    0/   93 (0%)
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  1.0- 2.0 sec    128 KBytes  1.05 Mbits/sec  0.136 ms    0/   89 (0%)
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0- 3.0 sec    386 KBytes  1.07 Mbits/sec  0.957 ms    0/  269 (0%)

The traffic looks like this:

[root@vserver0 ~]# tcpdump -n -i eth0 -s 1515 udp | grep  239.192.104.1 > muticast.txt
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 1515 bytes
343 packets captured
343 packets received by filter
0 packets dropped by kernel

[root@vserver0 ~]# 

[root@vserver0 ~]# wc -l muticast.txt 
283 muticast.txt
[root@vserver0 ~]# head -5 muticast.txt 
15:09:23.364851 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:23.760086 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:24.156427 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:24.555871 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
15:09:24.956558 IP 192.168.1.129.5149 > 239.192.104.1.netsupport: UDP, length 118
[root@vserver0 ~]#

So, I seem to have generated multicast traffic and ensured that a member of the multicast group actually received it.

According to RedHat (http://sources.redhat.com/cluster/doc/usage.txt): "CMAN can be configured to use multicast instead of broadcast (broadcast is used by default if no multicast parameters are given)." They also go on to say that you must enable it in /etc/cluster/cluster.conf:

<cman>
    <multicast addr="224.0.0.1"/>
</cman>
<clusternode name="nd1">
    <multicast addr="224.0.0.1" interface="eth0"/>
</clusternode>

SELinux: the basic idea and how to debug

2010-06-10T09:02:00.001-04:00

SELinux stores file context as objects which you can see with the -Z flag to ls or ps. Try to cp '/etc/hosts /home/$USER' and then 'mv /home/$USE/hosts /var/www/html' after editing it. The home context will stop Apache from serving the file. You'll see an error in /var/log/messages with what sealert options to run to see exactly what was blocked and how to fix it with restorecon.

Update: 11/24/10

Correction: A lot of the time restorecon won't fix it. Instead you'll need to change it with chcon. A typical example is when SELinux and Apache have a disagreement because you want to serve from a directory besides /var/www/, say /mnt/webapps/subversion for example. Even if you configure your httpd.conf to tell it that /mnt/webapps/subversion is a valid web directory it doesn't seem to work for permissions related reasons. The way to check this is to run:

ls -Z /var/www/html/
ls -Z /mnt/webapps/subversion

and see if you notice a difference. One difference that might been seen is the following:

unconfined_u:object_r:httpd_sys_content_t:s0
unconfined_u:object_r:mnt_t:s0

An easy way to fix this with chcon is:

chcon -R -t httpd_sys_content_t /mnt/webapps/subversion/

yum local install

2010-05-27T10:20:00.001-04:00

Given a set of RPMs you can install them with 'yum localinstall' and yum will resolve the dependencies:

yum localinstall --nogpgcheck file1.rpm file2.rpm ...

XIV IOPS

2010-05-24T09:47:00.004-04:00

The following table shows how the IOPS increase as you add more modules. It is for an XIV with 1T disks:

Nodes	IOPs	TB
6	52,000	27
9	70,000	43
10	78,000	50
15	105,000	79

CREDO and Hero

2010-05-13T09:03:00.003-04:00

I have an AT&T plan. I'm considering switching to CREDO now that they have an Android-based Phone. I am a little concerned that the HTC Hero is a little dated however; e.g. a 528MHz vs 1GHz snapdragon that the newer phones have. I've asked CREDO what's to stop someone from getting an EVO and using it under their plan. I'll post an update if they write back.

diaspora

2010-05-13T09:00:00.002-04:00

A while ago I linked to Eben Moglen's talk on Freedom in the Cloud. Some NYU students have started working on such a project: diaspora.

cheat sheets

2010-05-07T15:40:00.001-04:00

Nice Cheat Sheets.

XIV resiliency under some load

2010-05-05T19:18:00.017-04:00

Today we beat up the XIV by pulling a disk, pulling a whole module of 12 disks and re-zoning its fabric switch so that one of the HBAs lost connectivity, all while under some IO load. It did well.

We used a development server called physics0, which is a Dell M610 with a QLogic HBA running RHEL5.5. It was connected to a six-node XIV 2814 with 1T disks.

We started running four dd's:

$ ps axu | grep dd
root     12994 23.9  0.0  63156   580 pts/0    R    11:13   4:22 dd if /dev/mapper/mpath1p1 of /dev/null
root     12995 23.8  0.0  63156   572 pts/0    D    11:13   4:20 dd if /dev/mapper/mpath1p1 of /dev/null
root     12996 24.0  0.0  63156   572 pts/0    R    11:13   4:23 dd if /dev/mapper/mpath1p1 of /dev/null
root     12997 23.7  0.0  63156   576 pts/0    R    11:13   4:19 dd if /dev/mapper/mpath1p1 of /dev/null
$

top looked like this:

top - 11:43:35 up 4 days, 18:45,  3 users,  load average: 4.00, 4.00, 3.40
Tasks: 214 total,   3 running, 211 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.4%us,  4.0%sy,  0.0%ni, 80.8%id, 14.8%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.0%us, 10.6%sy,  0.0%ni, 53.1%id, 36.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us, 12.0%sy,  0.0%ni, 49.1%id, 38.9%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  1.0%sy,  0.0%ni, 95.6%id,  3.2%wa,  0.2%hi,  0.0%si,  0.0%st
Cpu4  :  0.0%us,  1.0%sy,  0.0%ni, 95.6%id,  3.4%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.2%us, 26.4%sy,  0.0%ni,  7.8%id, 49.6%wa,  1.4%hi, 14.6%si,  0.0%st
Cpu6  :  0.0%us, 26.9%sy,  0.0%ni,  7.6%id, 50.3%wa,  1.0%hi, 14.2%si,  0.0%st
Cpu7  :  0.0%us,  3.2%sy,  0.0%ni, 95.6%id,  1.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8177436k total,  8130020k used,    47416k free,  7084504k buffers
Swap:  2096472k total,      220k used,  2096252k free,    36764k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
12996 root      18   0 63156  572  476 D 26.0  0.0   7:00.78 dd                 
12995 root      18   0 63156  572  476 D 22.8  0.0   6:57.09 dd                 
12997 root      18   0 63156  576  476 D 22.6  0.0   6:55.44 dd                 
12994 root      18   0 63156  580  476 R 22.2  0.0   6:58.79 dd                 
  363 root      10  -5     0    0    0 R  2.8  0.0   1:17.65 kswapd0            
 4134 root      18   0 10228  740  628 S  0.2  0.0   0:18.81 hald-addon-stor

First we removed a disk. You can see the dip in IO and quick recovery: The graph above was made with the xivtop program (control click IOPS, Latency, BW to see all three at the same time).

We saw a graphical representation of the disk turning red on the XIV GUI and were then able to test it, add it, watch it rebuild. As it rebuilt we could get an update on the status with the xcli:

> > monitor_redist 
Type             Initial Capacity to Copy (GB)   Capacity Remaining to Copy (GB)   -1210286092one   Time Started          Estimated Time to Finish   Time Elapsed   
Redistribution   26                              17                                33      2010-05-05 11:25:31   0:04:08                    0:02:02        
> >

We then had the Cisco MDS9124 disable access to Module4 port 2 on the XIV. There was 100% a drop in IO until the retry interval ran out and it switched to an alternate path. IOs were queued until it passed them down a different path. The queue held it and I expect that it will hold it but we want to lower the retry interval so that it doesn't queue it as long. You can see the dip in IO on the graph. It queued it for 45 seconds: We also observed that the event log saw the available paths lesson:

> > event_list
...
2010-05-05 12:09:27   Informational   HOST_MULTIPATH_OK                                  Host 'nfile0' has redundant connections to the system. #paths=3                                                                                                              
2010-05-05 12:12:15   Informational   HOST_NO_MULTIPATH_ONLY_ONE_PORT                    Host 'physics0' is connected to the system through only one of its ports. #paths=2                                                                                           
2010-05-05 12:13:13   Informational   HOST_MULTIPATH_OK
Host 'physics0' has redundant connections to the system. #paths=4     

> >

Update: see Qlogic retry interval for how to implement instantaneous fail over as opposed to queuing for 45 seconds.

Unplugging the module was the most satisfying. I walked right up behind the unit and pulled both power cords out. The interruption in IO dip was longer as a result of loosing the module. Also IOPs averaged out not at 14,500 IOPs but 10,000 IOPs: The event log showed the unplugged module:

> > event_list
...
2010-05-05 12:46:14   Critical        MODULE_FAILED                                      1:Module:4 failed.                                                                                                                                                           
2010-05-05 12:46:17   Informational   DATA_REBUILD_STARTED                               Rebuild process started because system data is not protected. 1726767077764f the data must be rebuilt.                                                                                                                                                                 
2010-05-05 12:46:20   Informational   HOST_NO_MULTIPATH_ONLY_ONE_MODULE                  Host 'physics0' is connected to the system through only one interface module. #paths=2                                                                                       
2010-05-05 12:49:49   Informational   DATA_REBUILD_COMPLETED                             Rebuild process completed. System data is now protected.     
2010-05-05 13:16:37   Informational   DISK_FINISHED_PHASEIN                 System finished phasing in 1:Disk:4:1.                                                                                                                                       
2010-05-05 13:16:37   Informational   DISK_FINISHED_PHASEIN                 System finished phasing in 1:Disk:4:2.                                                                                                                                       
...
2010-05-05 13:16:38   Informational   DISK_FINISHED_PHASEIN                 System finished phasing in 1:Disk:4:12.                                                                                                     

> >

We can also observe what's "not ok" while the module is unblocked:

> > component_list filter=notok
Component ID           Status         Currently Functioning   
1:Disk:4:1             Failed         no                      
1:Disk:4:2             Failed         no                      
1:Disk:4:3             Failed         no                      
1:Disk:4:4             Failed         no                      
1:Disk:4:5             Failed         no                      
1:Disk:4:6             Failed         no                      
1:Disk:4:8             Failed         no                      
1:Disk:4:9             Failed         no                      
1:Disk:4:10            Failed         no                      
1:Disk:4:11            Failed         no                      
1:Disk:4:12            Failed         no                      
1:Module:4             Initializing   yes                     
1:Data:4               Failed         no                      
1:Interface:4          Failed         no                      
1:Remote:4             Failed         no                      
1:Interface:6          Ready          yes                     
1:Remote:6             Ready          yes                     
1:Ethernet_Cable:4:1   Failed         yes                     
1:Ethernet_Cable:4:2   Failed         yes                     
1:Ethernet_Cable:4:7   Failed         yes                     
1:Ethernet_Cable:4:8   Failed         yes                     
1:Ethernet_Cable:4:9   Failed         yes                     
1:Ethernet_Cable:4:10  Failed         yes                     
1:Disk:4:7             Failed         no                      
> >

and being added back in:

> > component_list filter=notok
Component ID   Status       Currently Functioning   
1:Interface:6  Ready        yes                     
1:Remote:6     Ready        yes                     
1:Disk:4:1     Phasing In   yes                     
1:Disk:4:2     Phasing In   yes                     
1:Disk:4:3     Phasing In   yes                     
1:Disk:4:4     Phasing In   yes                     
1:Disk:4:5     Phasing In   yes                     
1:Disk:4:6     Phasing In   yes                     
1:Disk:4:7     Phasing In   yes                     
1:Disk:4:8     Phasing In   yes                     
1:Disk:4:9     Phasing In   yes                     
1:Disk:4:10    Phasing In   yes                     
1:Disk:4:11    Phasing In   yes                     
1:Disk:4:12    Phasing In   yes  
> >

It's good to know it can take a beating before being sent into production.

The XIV is here...

2010-05-02T11:04:00.010-04:00

My organization's XIV arrived recently: Each rack has several commodity 2U servers with 12 1T drives: Inside of each server is 8G of RAM and two 2.3 Ghz Intel Quad Cores: which are connected together with Ethernet in a pre-wired APC cabinet: Also, the two 60A, 200-240 VAC, single phase, two pole, line-line ground female receptacles are huge:

Zimbra and Exchange TCO Comparison from the University of Pennsylvania

2010-04-30T08:56:00.000-04:00

Zimbra and Exchange TCO Comparison from the University of Pennsylvania.

SSD Experiment

2010-04-28T14:56:00.008-04:00

I've been considering experimenting with an SSD drive. I already have a 1T Seagate Barracuda. My plan is to get a 160G Intel X25-M SSD. On a hardware layer, it looks like it would work with my Dell Precision 690 using the SATA-300 interface. I also see that it includes a 3.5-inch bay adapter (given that a standard SATA drive is 3.5" and this is a a 2.5" drive). On a software layer, I would configure the two drives as follows:

160G SSD:
* 96G system drive with the following partitions:
```
/                     10G
/home                 10G
/tmp                   5G
/usr                  10G
/var                  61G
```
* 64G of cache/swap: I would like to use the extra speed for cache, not storage. Ideas:
- Swap: The simplest thing is to swapon to it and then run a lot of VMs under KVM and see if performance for the VM is tolerable. If it is then its much cheaper than buying more RAM.
- Disk Cache: If I were running ZFS I'd add the SSDs to the L2ARC on-disk cache.
- Other types of Cache: Lots of ideas on this topic recently posted on slashdot.

1T SATA:
I would then use the remaining 1T for directories under /home and /var to store VMs (as raw LVs) as well media. I'd also keep an extra 20G of the SATA space free to play with new file systems.

postmortem of the apache.org crack

2010-04-28T12:34:00.002-04:00

apache.org has posted a good model for the self-reporting of incidents.

tiered ram

2010-04-11T23:23:00.000-04:00

A friend of mine suggested that I could get more memory related performance out of my servers by using fast disk for swap. Specifically, I could buy 1TB of SSD disk and have a dom0 Xen server swapon it. E.g. the largest VM server in my organization's data center has 128G of RAM. If I needed to run more VMs and some percentage of them could take a slight performance hit, then I could allocate 256G of SSD as swap space to up to four equally large servers to have RAM*2 of swap. I could then rely on Linux's paging algorithm to prioritize for those VMs that really need real RAM and run a lot more VMs for cheaper than if I had to buy 1TB of RAM.

Disclaimer: I have not tried this.

Gratuitous ARPs whenever you need them

2010-04-08T18:39:00.004-04:00

Today I cloned a VM which had several IPs bounded to eth0. When I brought up the clone eth0 answered pings from my workstation (which is on a different network), but eth0:1, eth0:2, .. eth0:N did not. virt-clone had changed the MAC address as expected but why would a mac address change cause this problem? Turns out that the router between my workstation and dom0 had the old MAC address in its ARP cache and the ARP cache timeout is set high enough that I'd have to wait for it in order to reach the other IPs. But why wouldn't eth0 also be affected? Apparently when you boot a server it sends a Gratuitous ARP for eth0 but not eth0:1 etc. So the ARP cache for eth0 was updated but not the other IPs bound to eth0. What I wanted was to send a Gratuitous ARP for every IP bound to eth0 so that the cache would be refreshed. This is where arping is useful. E.g. to send a gratuitous ARP for the IP 192.168.6.212 whose gateway is 192.168.6.1, run:

/sbin/arping -s 192.168.6.212 -I eth0 192.168.6.1

From there it was just a matter of using a bash loop:

for x in `ifconfig | grep 192 | awk {'print $2'} | sed s/addr://g`; 
  do /sbin/arping -c 1 -s $x -I eth0 192.168.6.1 ;
done

I'm lucky my co-workers were able to help me understand this problem.

shaping like behavior while doing an scp

2010-04-08T18:29:00.004-04:00

While scp'ing a large file between two systems on the same network I experienced something similar to shaping, though I was not shaped:

$ scp drupal1* user@host:/var/foo/
drupal1_data.img.gz                             2% 3122MB  33.6MB/s 
...
drupal1_data.img.gz                            20%  294MB   0.0KB/s -
stalled -Write failed: Broken pipe
lost connection
$

The ascii illustration above isn't a direct paste but should convey what happened. I watched 33.6MB/s show down to a complete stop. What's odd is that I was then able to do the scp again and maintain 33.6MB/s and get the scp done. However, I also saw the issue above now and then intermittently. I was going from Fedora 12 to RHEL5.5. I'm curious and want to look into this more. Someone suggested I would find relative information at High Performance SSH/SCP - HPN-SSH

Update Dec 2011::
A solution to the above is posted at linuxsecure.blogspot.com. It comes down to adding:

net.ipv4.tcp_sack = 0

to /etc/sysctl.conf and then enacting it with "sysctl -p".

Python SSH with Paramiko quickly

2010-04-05T17:53:00.003-04:00

A quick python script to SSH into a server and run a list of commands. Uses Paramiko (doc). Fedora has a package:

yum install python-paramiko

Assumes you are using RSA Public Key Authentication:

#!/usr/bin/python
import paramiko, os, getpass

# Variables
username = 'you'
host = 'your.server.com'
port = 22
key = '~/.ssh/id_rsa'
msg = "Enter passphrase for key '" + key + "': "
private_key_pass = getpass.getpass(prompt=msg)
private_key_file = os.path.expanduser(key)

# Connctions
pkey = paramiko.RSAKey.from_private_key_file(private_key_file,\
                                              private_key_pass)
transport = paramiko.Transport((host, port))
transport.connect(username = username, pkey = pkey)

# Commands
cmds = ['ls', 'ls /foo']
for cmd in cmds:
    channel = transport.open_session()
    channel.exec_command(cmd)
    output = channel.makefile('rb', -1).readlines()
    if output:
        print "Success:"
        print output
    else:
        print "Error:"
        print channel.makefile_stderr('rb', -1).readlines()

transport.close()