OpenStack nested virtual machines

Running a virtual machine within a virtual machine (aka nested virtual machines) is supported by OpenStack by adding a stanza such as

    <cpu mode='custom' match='exact'>
      <feature policy='require' name='vmx'/>

in the libvirt.xml.template file. The nested feature must be enabled on the OpenStack node for intel processors by adding the nested=1 argument when loading the kvm_intel module so that:

# cat /sys/module/kvm_intel/parameters/nested

This feature can be used to exercise an automated Openstack deployement on Debian GNU/Linux wheezy with razor within OpenStack itself.

Activate nesting on the node

If the processor is Intel, the kvm_intel must loaded at boot time with the nested option. The list of options from the kvm_intel module can be displayed with:

#  modinfo kvm_intel
filename:       /lib/modules/3.2.0-3-amd64/kernel/arch/x86/kvm/kvm-intel.ko
license:        GPL
author:         Qumranet
depends:        kvm
intree:         Y
vermagic:       3.2.0-3-amd64 SMP mod_unload modversions
parm:           vpid:bool
parm:           flexpriority:bool
parm:           ept:bool
parm:           unrestricted_guest:bool
parm:           emulate_invalid_guest_state:bool
parm:           vmm_exclusive:bool
parm:           yield_on_hlt:bool
parm:           fasteoi:bool
parm:           nested:bool
parm:           ple_gap:int
parm:           ple_window:int

And the status of the nested boolean flag can be displayed with:

#  cat /sys/module/kvm_intel/parameters/nested

If the kvm_intel module is not in use because its reference count on the last column is 0:

# lsmod | grep kvm_intel
kvm_intel             121968  0
kvm                   287662  1 kvm_intel

it can be reloaded with the nested option set to 1:

# rmmod kvm_intel
# modprobe kvm_intel nested=1
#   cat /sys/module/kvm_intel/parameters/nested

For this option to be set at boot time, a modprobe.d(5) file can be created with the following puppet snippet:

  file { '/etc/modprobe.d/kvm_intel.conf':
    ensure  => present,
    owner   => "root",
    group   => "root",
    mode    => 0444,
    content  => "options kvm_intel nested=1\n",

Create cluster wide cpu description

By default OpenStack does not expose features of the host processor to the instance. To ensure that all processors in the OpenStack cluster can support nested virtual machines, the virsh cpu-baseline command can be used as follows:

# nova-manage host list 2>/dev/null | \
 sed -e '1d' | cut -f1 -d' ' | while read host ; do \
 ssh $host virsh  capabilities ; done | \
 virsh cpu-baseline /dev/stdin
<cpu mode='custom' match='exact'>
  <model fallback='allow'>Nehalem</model>
  <feature policy='require' name='rdtscp'/>
  <feature policy='require' name='dca'/>
  <feature policy='require' name='pdcm'/>
  <feature policy='require' name='xtpr'/>
  <feature policy='require' name='tm2'/>
  <feature policy='require' name='est'/>
  <feature policy='require' name='vmx'/>
  <feature policy='require' name='ds_cpl'/>
  <feature policy='require' name='monitor'/>
  <feature policy='require' name='dtes64'/>
  <feature policy='require' name='pbe'/>
  <feature policy='require' name='tm'/>
  <feature policy='require' name='ht'/>
  <feature policy='require' name='ss'/>
  <feature policy='require' name='acpi'/>
  <feature policy='require' name='ds'/>
  <feature policy='require' name='vme'/>

The virsh capabilities command is run on all hosts in the OpenStack cluster, as returned by nova-manage host list to get the cpu element that describes the processor. The result is streamed to the virsh cpu-baseline command which returns the common denominator. For instance, if one host does not support vmx, it won’t show in the output, even if all the others do. This common subset between all hosts ensures that an instance created on any given host can be migrated to another. The output of virsh cpu-baseline is edited to remove all but the vmx feature which is the only feature of interest:

    <cpu mode='custom' match='exact'>
      <model fallback='allow'>Nehalem</model>
      <feature policy='require' name='vmx'/>

Configure OpenStack to allow nested virtual machines

The libvirt.xml.template file on each OpenStack compute node is used as a template to launch the kvm process. The cpu element including <feature policy=’require’ name=’vmx’/> is added to it, manually. The addition is done in a copy of the file located in /etc/puppet/files/libvirt.xml.template on the puppetmaster. It is then broadcasted to all the node of the cluster using the following puppet manifest snippet:

  file { '/etc/nova/libvirt.xml.template':
    ensure  => present,
    owner   => "root",
    group   => "root",
    mode    => 0444,
    replace => true,
    source  => "puppet:///files/libvirt.xml.template",

  nova_config { 'libvirt_xml_template': value => '/etc/nova/libvirt.xml.template' }

The first stanza copies the modified libvirt.xml.template to /etc/nova and the second instruct nova-compute to use it instead of the default template provided by the nova-compute package.