Interactions with the kvm monitor in OpenStack

For debug purposes, is is convenient to access the kvm monitor to gain information about the state of the instance:

# python qmp-shell /var/lib/libvirt/qemu/instance-00000043.monitor
Connected!
(QEMU) query-commands
[{u'name': u'qom-list-types'}, {u'name': u'change-vnc-password'}, ...
(QEMU) query-vnc
{u'family': u'ipv4', u'service': u'5900', u'clients': [],
  u'enabled': True, u'auth': u'none', u'host': u'192.168.100.5'}
...

The libvirt daemon and the nova-compute daemon prevent access to the monitor and must be stopped.

Figuring out which process uses the monitor

The qemu monitor shell cannot run if another process accesses the kvm monitor. For a given instance, the monitor socket can be retrieved from the command line on the node running the kvm process:

# COLUMNS=20000 ps -fA | grep instance-00000043
106      11787     1  6 00:57 ?        00:40:25 /usr/bin/kvm -S -M pc-1.1 -enable-kvm
 -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000043 \
  -uuid 50b8a9ac-58a5-4743-a763-47e8e1f4901a -nodefconfig -nodefaults \
  -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000043.monitor,server,nowait \
  -mon chardev=charmonitor,id=monitor,mode=control \
  ...

Using the lsof command, the processes using the monitor socket ( /var/lib/libvirt/qemu/instance-00000043 ) can be listed:

# lsof | grep instance-00000043
kvm     11787 libvirt-qemu   18u  unix 0xffff880fbba96900         0t0 24081442 /var/lib/libvirt/qemu/instance-00000043.monitor

The linux kernel does not provide enough information for lsof to figure out the other endpoint of the socket, which is what is needed to figure out which process is connected to the monitor. However, the device address ( 0xffff880fbba96900 ) can be used with gdb to figure it out. The first step is to load the kernel with debug symbols:

# uname -a
Linux bm0005.the.re 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64 GNU/Linux
# apt-get install linux-image-3.2.0-3-amd64-dbg
...
# dpkg -L linux-image-3.2.0-3-amd64-dbg
...
/usr/lib/debug/boot/vmlinux-3.2.0-3-amd64

It is then possible to print the other end device address with:

# gdb /usr/lib/debug/boot/vmlinux-3.2.0-3-amd64 /proc/kcore
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
(gdb) p ((struct unix_sock*)0xffff880fbba96900)->peer
$1 = (struct sock *) 0xffff880fbba96c40
(gdb) quit
# lsof -n | grep 0xffff880fbba96c40
libvirtd  11162  root   27u unix 0xffff880fbba96c40 0t0  24069813 socket

The libvirtd process is connected to the other end of the /var/lib/libvirt/qemu/instance-00000043.monitor socket.

Stopping daemons

On a given OpenStack compute node, two daemons need to be stopped to get access to the monitor. Stopping them will not stop the running kvm instances, it will only prevent commands to be run from the OpenStack controller.

/etc/init.d/libvirt-bin stop
/etc/init.d/nova-compute stop

Running the monitor shell

The monitor shell is not packaged for Debian GNU/Linux yet. It can be downloaded as follows:

wget http://git.savannah.gnu.org/cgit/qemu.git/plain/QMP/qmp.py
wget http://git.savannah.gnu.org/cgit/qemu.git/plain/QMP/qmp-shell

and run against a monitor socket as follows:

# python qmp-shell /var/lib/libvirt/qemu/instance-00000043.monitor
Connected!
(QEMU) query-commands
[{u'name': u'qom-list-types'}, {u'name': u'change-vnc-password'}, ...
(QEMU) query-vnc
{u'family': u'ipv4', u'service': u'5900', u'clients': [],
  u'enabled': True, u'auth': u'none', u'host': u'192.168.100.5'}
...

To exit the shell use the bye command and not quit or Control-D because it will kill the kvm process.