Video: Troubleshooting Neutron Virtual Networks

This video on Troubleshooting Neutron Virtual Networks is really helpful for debugging OpenStack network problems. Make sure you understand TUN/TAP devices before watching.

qpid101

#!/usr/bin/env python
# Filename:                qpid101.py
# Description:             Qpid 101
# Supported Langauge(s):   Python 2.7.x
# Time-stamp:              <2014-07-19 01:04:29 someone> 
# -------------------------------------------------------
# * What's qpid?
# It's like a queue from data structures 101. How do you share
# the queue between two or more separate programs on two more
# more separate computers? Forget about passing pointers and
# sharing namespaces. Instead use a network queue server with  
# its own protocol. Then imagine special queues like how in data
# structures class there were special trees and make this server
# support those special extensions to a traditional queue. 
#
# * Could I use something other than a queue?
# Sure, you could use a DB or a RESTful web-service but DBs are 
# better at storing data, not passing it. A web-service between
# two programs doesn't allow for the message to be queued for
# later if the other program is down; i.e. it is not asynchronous.
# Why not make something new and focus on what it should do well? 
# See http://qpid.apache.org/overview.html for more justification.
# 
# * How how do I use it? 
# In a 101-level nutshell follow the examples below from:
# https://qpid.apache.org/releases/qpid-trunk/programming/book
# 
# * How do I install the Python library? 
# On Fedroa: 'yum install python-qpid python-qpid_messaging' 
#
# * Need a qpid server? 
# The swat-a-fly-with-a-telephone-pole-method...
# If you have an openstack install around, allow the client (to be
# written below) to connect to your openstack server on port 5672.
# -------------------------------------------------------
import sys
from qpid.messaging import *
broker =  "192.168.122.28:5672"
address = "amq.topic"
connection = Connection(broker)
try:
    # establish connection to message broker
    connection.open()
    # create a session object on which messages can be sent/recv'd
    session = connection.session()
    # create a sender that sends to the given address
    sender = session.sender(address)
    # create a receiver receives messages from the given address
    receiver = session.receiver(address)
    # send the message
    sender.send(Message("Hello world!"));
    # receive the message (only wait 1 second, else wait forver)
    message = receiver.fetch(timeout=1)
    print message.content
    # ack receipt of message during session
    session.acknowledge()
except MessagingError,m:
    print m
finally:
    connection.close()
# -------------------------------------------------------
# Not too exciting on the client: 
#     sh-4.2$ ./qpid101.py 
#     Hello world!
#     sh-4.2$ 
#
# Let's look at the server:
# * 'yum install qpid-tools'
# 
# * Run 'qpid-printevents' and re-run qpid101.py 
# ** The following is the output; all lines from one run; fields trimmed
# 
# 1. clientConnect broker=localhost:5672 rhost=192.168.122.28:5672-192.168.122.1:33254 user=anonymous
# 2. queueDeclare broker=localhost:5672 rhost=192.168.122.28:5672-192.168.122.1:33254 user=anonymous
#      qName=41a24690-c85b-4d4b-97c4-cdfc4ae0664a:0.0 durable=F excl=T autoDel=T args={} disp=created
# 3. bind broker=localhost:5672 rhost=192.168.122.28:5672-192.168.122.1:33254 user=anonymous exName=amq.topic
#      qName=41a24690-c85b-4d4b-97c4-cdfc4ae0664a:0.0 key=# args={}
# 4. subscribe broker=localhost:5672 rhost=192.168.122.28:5672-192.168.122.1:33254 user=anonymous
#      qName=41a24690-c85b-4d4b-97c4-cdfc4ae0664a:0.0 dest=0 excl=F args={}
# 5. unsubscribe broker=localhost:5672 rhost=192.168.122.28:5672-192.168.122.1:33254 user=anonymous dest=0
# 6. queueDelete broker=localhost:5672 rhost=192.168.122.28:5672-192.168.122.1:33254 user=anonymous
#      qName=41a24690-c85b-4d4b-97c4-cdfc4ae0664a:0.0
# 7. clientDisconnect broker=localhost:5672 rhost=192.168.122.28:5672-192.168.122.1:33254 user=anonymous
#
# Try flooding it with 'while [ 1 ]; do ./qpid101.py > /dev/null ; done' and watch it run
# 
# * Run 'qpid-queue-stats' 
# ** It's a ~150 line Python program and prints the enq/deq rate every 10 seconds
#
# * Let's look at qpid-stat
# ** It's a ~500 line Python program that displays msgIn/MsgOut per queue and other stats
# ** Try 'watch "qpid-stat -q -S msgIn"' to see the queues sorted by messages in
# ** run '/etc/init.d/qpidd restart' as root in another termainal and watch the msgIn reset
# 
# queue      dur  autoDel  excl  msg   msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
# =========================================================================================
# q-plugin                          0  25.3k  25.3k      0   19.2m    19.2m        1     2
# 2dcb1680...              Y        0  15.3k  15.3k      0   5.70m    5.70m        1     2
# conductor                         0  7.66k  7.66k      0   8.68m    8.68m        1     2
# ...
#
# ** Try 'watch "qpid-stat -c -I"' to pick out who is connecting
# *** Use the first 10 lines of code via a Python REPL and wrap the send in loop
# *** Pick out your client address and note the change in idle, and msgIn
# 
# client-addr                 cproc          cpid   connected   idle    msgIn  msgOut
# ===================================================================================
# :5672-192.168.122.1:41724                  21772  8m 25s      3m 30s   120k  40.2k
# :5672-192.168.122.28:54254  cinder-volume  15803  9h 18m 17s  10s     4.72k  5.02k
# ...
# 

hello docker

I tried out Docker for the first time. I got the hello world node.js web app running as well as the RHEL7 Release Candidate Docker Image working. Fun.

subscription-manager cheat sheet

Red Hat is moving from RHN Classic to Red Hat Subscription Management so you may want to start using subscription-manager instead of rhn_register. Here's a cheat sheet on how to register your system to the CDN, attach to what should be the appropriate channel, see other repos and add an extra repo. For more info see the documentation.

subscription-manager register
subscription-manager attach --auto
subscription-manager list --available
subscription-manager repos --enable=rhel-6-server-optional-rpms

ksar wrapper

I have been using ksar to analyze the output of sar. I often use it one specific way so I have a wrapper script which assumes you've done the following:

1. Download kSar-X.y.z.zip from sourceforge
2. Unpack it in /usr/local/ksar/
3. ln -s /usr/local/ksar/kSar-X.y.z /usr/local/ksar/ksar
4. Add the following script somewhere in your $PATH

#!/bin/bash
# Filename:                ksar
# Description:             Wrapper to run ksar
# Supported Langauge(s):   GNU bash, version 4.2.x
# Time-stamp:              <2014-02-14 16:12:18 someguy> 
# -------------------------------------------------------
# http://sourceforge.net/projects/ksar/
# -------------------------------------------------------
if [ "$#" -eq 1 ]
then
    /bin/java -jar /usr/local/ksar/ksar/kSar.jar -input $1  -outputPDF $1.pdf
else
    echo "Usage: 'ksar foo.sar'";
    echo "where foo.sar is a sar file produced by sar";
    echo "Note that such a file can be found in an sosreport";
fi

The assumption is that you just want to generate a PDF for a particular sar report

$ ksar sar11 
time to parse: 560ms number of line: 2527 line/msec: 4.0
$ 
$ mupdf sar11.pdf & 

RDO quickstart

Tonight I installed OpenStack with RDO Quickstart on RHEL6. They've really made it easy.

rhel7 beta ifconfig is gone; now what?

My minimally installed rhel7 beta system didn't include ifconfig.

[root@rhel7 ~]# ifconfig 
-bash: ifconfig: command not found
[root@rhel7 ~]# 

You can get it with the net-tools package:

yum install net-tools

but the Networking Guide makes no mention of it and seems to encourage nmcli. If you need to get your system online quickly and are unable to get net-tools installed here's a nmcli crash course.

Edit /etc/sysconfig/network-scripts/ifcfg-ethX, like usual, and bring the device up with:

nmcli con up eth0

or bring it down with:

nmcli con down eth0

You can go as short as "nmcli c u eth0" or as long as "nmcli connection up eth0".

To see your active connections:

nmcli c s a

or do the long version:

[root@rhel7 ~]# nmcli connection show active
NAME  UUID                                  DEVICES  DEFAULT  VPN  MASTER-PATH 
eth0  5a26138b-99c5-4769-a036-7856598e696e  eth0     yes      no   --          
[root@rhel7 ~]# 

To then zoom in on the details of one of the devices see:

nmcli d sh eth0

or again do the long version and you'll get a lot more info:

[root@rhel7 ~]# nmcli device show eth0 | wc -l
55
[root@rhel7 ~]# 

So you might want to grep for just the IP or Mac:

[root@rhel7 ~]# nmcli d sh eth0 | grep ip
IP4.ADDRESS[1]:                         ip = 192.168.122.70/24, gw = 192.168.122.1
DHCP4.OPTION[6]:                        ip_address = 192.168.122.70
IP6.ADDRESS[1]:                         ip = fe80::5054:ff:fe5f:edfa/64, gw = ::
[root@rhel7 ~]# 

[root@rhel7 ~]# nmcli d sh eth0 | grep -i hw
GENERAL.HWADDR:                         52:54:00:5F:ED:FA
[root@rhel7 ~]# 

For bonus points, set your hostname using hostnamectl.

hostnamectl set-hostname $hostname

But don't forget to edit /etc/hosts.

The Slab

Today I learned about memory management via slab allocation and slabtop along with the move to slub after 2.6.23.