realistic puppet tests with jenkins and OpenStack (part 1/2)

A git repository containing a puppet module is bound to a jenkins project. When the repository changes, jenkins boots a virgin puppetmaster OpenStack instance in a dedicated tenant. It runs the run-jenkins-test-in-openstack.sh script in the puppetmaster instance. In addition to the puppet unit tests, the script will launch realistic tests by launching OpenStack instances and checking their state. The checks are done with nagios which can also be used in a production environment to continuously monitor the deployment.

The puppet unit tests are not enough to test a puppet module or manifest. Actually running the puppetmaster in a controlled environment is the only way to ensure a module or manifest performs as expected. It is also required to protect against regressions because the module changed or because it relies on an external resource (such as a Debian GNU/Linux repository) that has been modified.

dedicated OpenStack tenant

The run-jenkins-test-in-openstack.sh is expected to be found at the root of the git repository. Jenkins will provide it with:

  • A read only copy of the git repository
  • The OpenStack credentials to create instances using the nova command
  • The EC2 credentials to use the euca-* commands

The there-ci tenant is created and will be used exclusively by jenkins. Its quotas are adjusted to control the resource usage in case the test script has a bug leading it to use all the available resources.
A there-ci user is created and associated with the there-ci tenant.

The corresponding credentials are uploaded to the machine running jenkins, in the /etc/jenkins directory:

root@jenkins:~# ls -l /etc/jenkins/
total 20
-rw------- 1 tomcat6 tomcat6 1029 Nov 20 21:10 cacert.pem
-rw------- 1 tomcat6 tomcat6 2551 Nov 20 21:10 cert.pem
-rw------- 1 tomcat6 tomcat6  914 Nov 20 21:10 ec2rc.sh
-rw------- 1 tomcat6 tomcat6  955 Nov 21 21:07 openrc.sh
-rw------- 1 tomcat6 tomcat6  891 Nov 20 21:10 pk.pem

They are

chown tomcat6:tomcat6 /etc/jenkins/*

because the jenkins server runs under the tomcat6 identity and needs read permissions on those files. The openrc.sh script is edited to hardcode the there-ci user password.

# With Keystone you pass the keystone password.
#echo "Please enter your OpenStack Password: "
#read -s OS_PASSWORD_INPUT
export OS_PASSWORD=XXXXXXXXX

Since the jenkins virtual machine is not running in the there-ci tenant, a floating IP must be associated to the there-ci tenant.

The IP is bound to the openstack-there-ci.the.re hostname for convenience and will be used under this name by jenkins.

registering jenkins ssh key in OpenStack

The jenkins installation runs under the tomcat6 user and a ssh key is created for it.

root@jenkins /etc# git diff
root@jenkins /etc# diff --git a/passwd b/passwd
index 805b5ca..278e32e 100644
--- a/passwd
+++ b/passwd
@@ -21,4 +21,4 @@ ntp:x:101:103::/home/ntp:/bin/false
 sshd:x:102:65534::/var/run/sshd:/usr/sbin/nologin
 shellinabox:x:103:105:Shell In A Box,,,:/var/lib/shellinabox:/bin/false
 postfix:x:104:107::/var/spool/postfix:/bin/false
-tomcat6:x:105:109::/usr/share/tomcat6:/bin/false
+tomcat6:x:105:109::/usr/share/tomcat6:/bin/bash
root@jenkins /etc# chown tomcat6 /usr/share/tomcat6
root@jenkins /etc# su - tomcat6
tomcat6@jenkins:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/usr/share/tomcat6/.ssh/id_rsa):
Created directory '/usr/share/tomcat6/.ssh'.
Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /usr/share/tomcat6/.ssh/id_rsa.
Your public key has been saved in /usr/share/tomcat6/.ssh/id_rsa.pub.
The key fingerprint is:
a0:cc:bb:18:94:3f:de:3d:3a:56:59:b1:34:e4:82:97 tomcat6@jenkins
The key's randomart image is:
+--[ RSA 2048]----+
|         ..      |
|       . o+      |
|      o E..+     |
|   + . o .o      |
|  o +   So       |
| . . .  o        |
|  . +  .         |
|   + +o..        |
|  . o.oo..       |
+-----------------+

It is then registered with the there-ci user to allow jenkins to run ssh commands on the instance launched in the there-ci tenant.

tomcat6@jenkins:~$ nova keypair-add --pub_key ~/.ssh/id_rsa.pub jenkins
tomcat6@jenkins:~$ nova keypair-list
+---------+-------------------------------------------------+
|   Name  |                   Fingerprint                   |
+---------+-------------------------------------------------+
| jenkins | a0:cc:bb:18:94:3f:de:3d:3a:56:59:b1:34:e4:82:97 |
+---------+-------------------------------------------------+

The nova command line is installed from the wheezy repository:

echo deb http://ftp.debian.org/debian wheezy main > /etc/apt/sources.list.d/wheezy.list
apt-get update
apt-get install libc6-dev
apt-get install python-novaclient

The git id must also be set otherwise jenkins complains:

tomcat6@jenkins:~$ git config --global user.email "contact@the.re"
tomcat6@jenkins:~$ git config --global user.name "Loic Dachary"

The default security group is modified to allow ssh connections.

puppet test OpenStack image

Since the goal is to test puppet modules or manifests, the test script needs an environment able to

  • boot openstack instances in the there-ci tenant
  • run a puppetmaster
  • clone the git repository containing the module or manifest to be tested

The requirements are installed manually and snapshoted into the puppet.novalocal private image.
A Debian GNU/Linux wheezy instance is created and bound to openstack-there-ci.the.re for ssh access.

tomcat6@jenkins:~$ nova boot --image 'Debian GNU/Linux Wheezy Beta2' \
 --flavor e.1-cpu.10GB-disk.1GB-ram \
 --key_name jenkins --availability_zone=bm0007 --poll puppet
+------------------------+--------------------------------------+
|        Property        |                Value                 |
+------------------------+--------------------------------------+
|   OS-DCF:diskConfig    |                MANUAL                |
| OS-EXT-STS:power_state |                  0                   |
| OS-EXT-STS:task_state  |              scheduling              |
|  OS-EXT-STS:vm_state   |               building               |
|       accessIPv4       |                                      |
|       accessIPv6       |                                      |
|       adminPass        |             Vyv5HZ9cchAQ             |
|      config_drive      |                                      |
|        created         |         2012-11-22T20:23:51Z         |
|         flavor         |      e.1-cpu.10GB-disk.1GB-ram       |
|         hostId         |                                      |
|           id           | 9bdcb8f5-0a82-4082-8861-166d4eabfbec |
|         image          |    Debian GNU/Linux Wheezy Beta2     |
|        key_name        |               jenkins                |
|        metadata        |                  {}                  |
|          name          |                puppet                |
|        progress        |                  0                   |
|         status         |                BUILD                 |
|       tenant_id        |   b3d9240b60694c0382ce38ca192b5569   |
|        updated         |         2012-11-22T20:23:51Z         |
|        user_id         |   eda33dfaddac4acd9f26570089bc6e24   |
+------------------------+--------------------------------------+

Instance building... 100% complete
Finished
tomcat6@jenkins:~$ nova list
+--------------------------------------+--------+--------+-------------------------+
|                  ID                  |  Name  | Status |         Networks        |
+--------------------------------------+--------+--------+-------------------------+
| 9bdcb8f5-0a82-4082-8861-166d4eabfbec | puppet | ACTIVE | novanetwork=10.145.10.5 |
+--------------------------------------+--------+--------+-------------------------+
tomcat6@jenkins:~$ nova floating-ip-list
+------------+-------------+----------+--------+
|     Ip     | Instance Id | Fixed Ip |  Pool  |
+------------+-------------+----------+--------+
| 5.9.177.68 |     None    |   None   | flouzo |
+------------+-------------+----------+--------+
tomcat6@jenkins:~$ nova add-floating-ip puppet 5.9.177.68
tomcat6@jenkins:~$ host openstack-there-ci.the.re
openstack-there-ci.the.re has address 5.9.177.68
tomcat6@jenkins:~$ ssh root@openstack-there-ci.the.re ip a
The authenticity of host 'openstack-there-ci.the.re (5.9.177.68)' can't be established.
RSA key fingerprint is 76:d1:c4:94:2f:55:6d:f1:a8:ea:f1:8c:d3:78:08:27.
Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'openstack-there-ci.the.re,5.9.177.68' (RSA) to the list of known hosts.
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:6c:5b:99 brd ff:ff:ff:ff:ff:ff
    inet 10.145.10.5/24 brd 10.145.10.255 scope global eth0
    inet6 fe80::f816:3eff:fe6c:5b99/64 scope link tentative dadfailed
       valid_lft forever preferred_lft forever


The there-ci credentials are injected in the instance.

tomcat6@jenkins:~$ scp /etc/jenkins/* root@openstack-there-ci.the.re:
cacert.pem                                    100% 1029     1.0KB/s   00:00
cert.pem                                      100% 2551     2.5KB/s   00:00
ec2rc.sh                                      100%  914     0.9KB/s   00:00
openrc.sh                                     100%  955     0.9KB/s   00:00
pk.pem                                        100%  887     0.9KB/s   00:00

The packages required to run a puppetmaster are loaded

ssh root@openstack-there-ci.the.re apt-get install puppet \
 augeas-tools puppetmaster \
 sqlite3 libsqlite3-ruby libactiverecord-ruby git

The hostname is set to puppet.novalocal so that the puppet client instances find it automatically. Finaly, the instance is snapshoted into a private image with:

nova image-create puppet puppet.novalocal

running the test

The openstack-test.sh is uploaded to the jenkins instance and will be used by all jobs related to testing puppet modules or manifests.The l2mesh job is then configured to call it with the git location as an argument.

It runs a new instance from the puppet.novalocal image, clones the target git repository in it and runs the the test script it contains.

The test presented here does not actually do anything and always succeeds. The environment is ready to run the realistic puppet test and it will be the topic of the second part of this post.