Posted on

nova-network debugging tips

A single machine is installed with Debian GNU/Linux OpenStack Folsom. Four instances are created and it turns out that nova-network is configured with the wrong public interface. It can be fixed without shutting down the instance:

nova suspend target1

The instance is suspended to disk (as if it was a laptop) and the corresponding KVM process is killed. While the instance is suspended, nova-network can be stopped.

/etc/init.d/nova-network stop

The source of the problem was a typo in the public interface leading to an incorrect VLAN interface

13: vlan100@eth2:  mtu 1500 qdisc noqueue state DOWN mode DEFAULT
    link/ether fa:16:3e:54:5b:57 brd ff:ff:ff:ff:ff:ff

it can be fixed in the /etc/nova/nova.conf configuration file at the line:

public_interface = eth3

The incorrect VLAN interface is manually deleted and nova-network can be restarted. The instance is then resumed with

nova resume target1

and nova-network will automatically re-create the VLAN interface.
Continue reading “nova-network debugging tips”

Posted on

Upstream University at the OpenStack summit

Upstream University What if contributing to OpenStack was made a lot easier by a few days of training? You could get this training at Upstream University, which was created shortly after the OpenStack design summit, in April 2012, with this sole goal of improving developers’ contribution skills. Upstream University has since coached new OpenStack contributors, from eNovance and Cloudwatt, developers; for the kernel Linux and many others. OpenStack
To celebrate its first year, Upstream University is organizing a session in advance of the next OpenStack summit, in Portland. If you can fly in two days ahead of the event to spend the weekend improving your OpenStack contribution skills, please consider submitting an application to attend the workshop. This a one-time offer for free training.
Continue reading “Upstream University at the OpenStack summit”

Posted on

ceph code coverage (part 2/2)

WARNING: teuthology has changed significantly, the instructions won’t work anymore.

When running ceph integration tests with teuthology, code coverage reports shows which lines of code were involved. Adding coverage: true to the integration task and using code compiled for code coverage instrumentation with flavor: gcov collects coverage data. lcov is then used

./virtualenv/bin/teuthology-coverage -v --html-output /tmp/html ...

to create an HTML report. It shows that lines 217 and 218 of mon/Monitor.cc are not being used by the scenario.


Continue reading “ceph code coverage (part 2/2)”

Posted on

Installing OpenStack Folsom on Debian GNU/Linux wheezy

Installing and testing OpenStack Folsom on a virgin Debian GNU/Linux wheezy takes less than one hour. A set of packages is archived to make sure it keeps working. After checking the pre-requisites such as a public and private interface, the packages are installed and debconf questions answered as instructed.
The networks must then be created with

nova-manage network create private --fixed_range_v4=10.20.0.0/16 \
  --network_size=256 --num_networks=2 --vlan=100

/etc/nova/nova.conf is updated to set vlan_interface=dummy0, public_interface=eth0 and fixed_range=10.20.0.0/16. /etc/nova/nova-compute.conf is updated to use LibvirtBridgeDriver and an instance can be booted with:

nova boot --poll --flavor m1.tiny --image cirrOS-0.3.0-x86_64 \
  --key_name loic test

Continue reading “Installing OpenStack Folsom on Debian GNU/Linux wheezy”

Posted on

anatomy of an OpenStack based integration test for a backuppc puppet module

An integration test is run by jenkins within an OpenStack tenant. It checks that the backuppc puppet module is installed

ssh root@$instance test -f /etc/backuppc/hosts || return 3

A full backup is run

ssh root@$instance su -c '"/usr/share/backuppc/bin/BackupPC_serverMesg \
   backup nagios.novalocal nagios.novalocal backuppc 1"' \
                              backuppc || return 4
ssh root@$instance tail -f /var/lib/backuppc/pc/nagios.novalocal/LOG.* | \
    sed --unbuffered -e "s/^/$instance: /" -e '/full backup 0 complete/q'

and a nagios plugin asserts its status is monitored

    while ! ( echo "GET services"
        echo "Filter: host_alias = $instance.novalocal"
        echo "Filter: check_command = check_nrpe_1arg"'!'"check_backuppc" ) |
        ssh root@nagios unixcat /var/lib/nagios3/rw/live |
        grep "BACKUPPC OK - (0/" ; do
        sleep 1
    done

Continue reading “anatomy of an OpenStack based integration test for a backuppc puppet module”

Posted on

nagios puppet module for the April infrastructure

This document explains the nagios configuration for the infrastructure of the April non profit organisation.
It is used to configure the nagios server overseeing all the services. The nagios plugins that cannot be run from the server ( such as check_oom_killer ) are installed locally and connected to nagios with nrpe. All services are bound to private IPs within the 192.168.0.0/16 network and exposed to the nagios server ( using OpenVPN to connect bare metal machines together ) and the firewalls are set to allow TCP on the nrpe port ( 5666 ).
Continue reading “nagios puppet module for the April infrastructure”

Posted on

Migrating OpenVZ virtual machines to OpenStack

A OpenVZ cluster hosts GNU/Linux based virtual machines. The disk is extracted with rsync and uploaded to the glance OpenStack image service with glance add … disk_format=ami…. It is associated with a kernel image compatible with both OpenStack and the existing file system with glance update … kernel_id=0dfff976-1f55-4184-954c-a111f4a28eef ramdisk_id=aa87c84c-d3be-41d0-a272-0b4a85801a34 ….
Continue reading “Migrating OpenVZ virtual machines to OpenStack”

Posted on

realistic puppet tests with jenkins and OpenStack (part 2/2)

The April infrastructure uses puppet manifests stored in a git repository. On each commit, a jenkins job is run and it performs realistic tests in a dedicated OpenStack tenant.

If the test is successfull, jenkins pushes the commit to the production branch. The production machines can then pull from it:

root@puppet:/srv/admins# git pull
Updating 5efbe80..cf59d69
Fast-forward
 .gitmodules                      |    6 +++
 jenkins/openstack-test.sh        |   53 +++++++++++++++++++++++++++
  jenkins/run-test-in-openstack.sh |  215 +++++++++++++++++++++++++++
 puppetmaster/manifests/site.pp   |   43 ++++++++++++++++++++--
 puppetmaster/modules/apt         |    1 +
 6 files changed, 315 insertions(+), 165 deletions(-)
 create mode 100755 jenkins/openstack-test.sh
 create mode 100644 jenkins/run-test-in-openstack.sh
 create mode 160000 puppetmaster/modules/apt
root@puppet:/srv/admins# git branch -v
  master     5efbe80 [behind 19] ajout du support nagios, configuration .... refs #1053
* production cf59d69 Set the nagios password for debugging ...

Continue reading “realistic puppet tests with jenkins and OpenStack (part 2/2)”

Posted on

routing ipv6 from Hetzner to an OpenStack instance

The 2a01:4f8:162:12e3::2 IPv6 address is assigned to http://packaging-farm.dachary.org/ from the IPv6 subnet provided by Hetzner. The OpenStack host on which the instance running packaging-farm.dachary.org is running is configured as a proxy with

sysctl -w net.ipv6.conf.all.proxy_ndp=1
ip -6 neigh add proxy 2a01:4f8:162:12e3::2 dev eth0

and an OpenStack Essex bug is worked around by manually disabling hairpin_mode:

echo 0 > /sys/class/net/br2003/brif/vnet1/hairpin_mode

The page can then be retrieved with

$ curl --verbose -6 http://packaging-farm.dachary.org/
* About to connect() to packaging-farm.dachary.org port 80 (#0)
*   Trying 2a01:4f8:162:12e3::2...
* connected
* Connected to packaging-farm.dachary.org (2a01:4f8:162:12e3::2) port 80 (#0)
> Host: packaging-farm.dachary.org
> Accept: */*
>
...
<address>Apache/2.2.19 (Debian) Server at packaging-farm.dachary.org Port 80</address>
</body></html>
* Connection #0 to host packaging-farm.dachary.org left intact
* Closing connection #0

Continue reading “routing ipv6 from Hetzner to an OpenStack instance”

Posted on

realistic puppet tests with jenkins and OpenStack (part 1/2)

A git repository containing a puppet module is bound to a jenkins project. When the repository changes, jenkins boots a virgin puppetmaster OpenStack instance in a dedicated tenant. It runs the run-jenkins-test-in-openstack.sh script in the puppetmaster instance. In addition to the puppet unit tests, the script will launch realistic tests by launching OpenStack instances and checking their state. The checks are done with nagios which can also be used in a production environment to continuously monitor the deployment.
Continue reading “realistic puppet tests with jenkins and OpenStack (part 1/2)”