Interactions with the kvm monitor in OpenStack

For debug purposes, is is convenient to access the kvm monitor to gain information about the state of the instance:

# python qmp-shell /var/lib/libvirt/qemu/instance-00000043.monitor
Connected!
(QEMU) query-commands
[{u'name': u'qom-list-types'}, {u'name': u'change-vnc-password'}, ...
(QEMU) query-vnc
{u'family': u'ipv4', u'service': u'5900', u'clients': [],
  u'enabled': True, u'auth': u'none', u'host': u'192.168.100.5'}
...

The libvirt daemon and the nova-compute daemon prevent access to the monitor and must be stopped.
Continue reading “Interactions with the kvm monitor in OpenStack”

Disaster recovery with kvm in OpenStack

A kvm process running an OpenStack instance with a volume attached to it becomes unresponsive. After manually investigating the cause of the problem, the kvm process is killed. The iscsi sessions is acknowledged to be lost and the disk is re-attached. The instance fails to reboot and the root file system is repaired from the compute node:

# qemu-nbd --port 20000 /var/lib/nova/instances/instance-00000103/disk &
# nbd-client localhost 20000 /dev/nbd0
Negotiation: ..size = 10240MB
bs=1024, sz=10737418240 bytes
# mount /dev/nbd0p1 /mnt
... fix things ...
# umount /mnt
# nbd-client -d /dev/nbd0

Continue reading “Disaster recovery with kvm in OpenStack”

Debugging razor with OpenStack

razor can be used to deploy OpenStack. However, because it has been released mid 2012, debugging problems is often required. A razor virtual machine is created in an OpenStack cluster configured to allow network boot an OpenStack instance and nested virtual machines. When a new virtual machine instance is created in the same tenant as the razor machine, it gets a pixie boot from razor.
Testing and debugging razor can then be done within OpenStack and does not require dedicated hardware, as shown in the following screenshot of the VNC console of a virtual machine being deployed from razor ( it shows the razor microkernel booting ):

Continue reading “Debugging razor with OpenStack”

Network boot an OpenStack instance

When an OpenStack instance is defined as a raw disk, the default libvirt XML description is set to boot from it, assuming it contains a boot record. The libvirt.xml.template file is modified to add an attempt to boot from the network before booting from the disk:

<boot dev="network" />

The ebtables and iptables network filtering rules are relaxed by removing the filterref element from the libvirt.xml.template so that the DHCP request from a new instance can be answered by a DHCP server provided by another instance running on the same VLAN.
The dnsmasq run by OpenStack is configured so that it will not answer DHCP requests issued by iPXE

dhcp-userclass=set:ignore,iPXE
dhcp-ignore=tag:ignore

so that DHCP requests issued by DHCP clients such as ISC DHCP get an IP address from the OpenStack provided dnsmasq while instances trying to boot from the network will get their IP and kernel from the DHCP server running on an instance created by a user.
Continue reading “Network boot an OpenStack instance”

packaging-farm for OpenStack Essex

The Essex OpenStack release is being packaged for Debian GNU/Linux wheezy upcoming release. Each new release of a package is committed to a git repository ( the keystone is an example ) by a member of the OpenStack Debian GNU/Linux release team and uploaded to Debian GNU/Linux : a process that can require a few weeks before it is available in the Debian GNU/Linux worldwide mirror sites. In the meantime, a temporary repository is setup with packaging-farm to host the packages being processed so that they can be used immediately by adding the following line to the source list of packages.

deb http://wheezy.the.re/packaging-farm/openstack/gnulinux/debian wheezy main

Continue reading “packaging-farm for OpenStack Essex”

Migrating ganeti virtual machines to OpenStack (part 2 / 2)

A ganeti cluster hosts GNU/Linux based fully virtualized virtual machines, booting from a disk image with a boot loader and a partition table. The case of para-virtualized virtual machines has been covered in a previous post. An OpenStack LVM volume is created on a designated bare metal machine with euca-create-volume –zone ZONE –size INGB. The disk image is copied verbatim into the volume with a command such as dd if=/dev/drbd9 | ssh dd of=/dev/nova-volumes/vol-0001. The virtual machine is then defined on the same host to save bandwidth with nova boot … –block_device_mapping vda=39::0:0 … –availability_zone=ZONE … .
Continue reading “Migrating ganeti virtual machines to OpenStack (part 2 / 2)”

Migrating ganeti virtual machines to OpenStack (part 1 / 2)

A ganeti cluster hosts GNU/Linux based para-virtualized virtual machines. The disk is extracted with kpartx(8) and uploaded to the glance OpenStack image service with glance add … disk_format=ami…. It is associated with a kernel image compatible with both OpenStack and the existing file system with glance update … kernel_id=0dfff976-1f55-4184-954c-a111f4a28eef ramdisk_id=aa87c84c-d3be-41d0-a272-0b4a85801a34 ….
Continue reading “Migrating ganeti virtual machines to OpenStack (part 1 / 2)”

Minimal Debian GNU/Linux wheezy bootable image for OpenStack

A Debian GNU/Linux Wheezy Beta 2 AMI is uploaded to OpenStack. It really is an Ext4 file system that OpenStack will resize to use all the size of the primary disk of the instance. An AMI is made of three files, the kernel (the AKI), the ramdisk (the ARI) used at boot time and the file system to boot from (the AMI). In OpenStack, the AMI is specified to the nova boot … –image command line.

# glance index name='Debian GNU/Linux Wheezy Beta2'
ID
------------------------------------
a8f86804-3927-419e-9612-ad35cbafefa0
 Name                           Disk Format          Container Format
 ------------------------------ -------------------- --------------------
 Debian GNU/Linux Wheezy Beta2  ami                  ami

It contains references to the required AKI and ARI, as shown below:

# nova image-show a8f86804-3927-419e-9612-ad35cbafefa0
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
| created             | 2012-09-30T15:26:43Z                 |
| id                  | a8f86804-3927-419e-9612-ad35cbafefa0 |
| metadata  kernel_id | 2e714ea3-45e5-4bb8-ab5d-92bfff64ad28 |
| metadata ramdisk_id | 6458acca-24ef-4568-bb2b-e52322a5a11c |
| minDisk             | 0                                    |
| minRam              | 0                                    |
| name                | Debian GNU/Linux Wheezy Beta2        |
| progress            | 100                                  |
| status              | ACTIVE                               |
| updated             | 2012-09-30T15:26:45Z                 |
+---------------------+--------------------------------------+

Continue reading “Minimal Debian GNU/Linux wheezy bootable image for OpenStack”