The April infrastructure uses puppet manifests stored in a git repository. On each commit, a jenkins job is run and it performs realistic tests in a dedicated OpenStack tenant.
If the test is successfull, jenkins pushes the commit to the production branch. The production machines can then pull from it:
root@puppet:/srv/admins# git pull Updating 5efbe80..cf59d69 Fast-forward .gitmodules | 6 +++ jenkins/openstack-test.sh | 53 +++++++++++++++++++++++++++ jenkins/run-test-in-openstack.sh | 215 +++++++++++++++++++++++++++ puppetmaster/manifests/site.pp | 43 ++++++++++++++++++++-- puppetmaster/modules/apt | 1 + 6 files changed, 315 insertions(+), 165 deletions(-) create mode 100755 jenkins/openstack-test.sh create mode 100644 jenkins/run-test-in-openstack.sh create mode 160000 puppetmaster/modules/apt root@puppet:/srv/admins# git branch -v master 5efbe80 [behind 19] ajout du support nagios, configuration .... refs #1053 * production cf59d69 Set the nagios password for debugging ...
Public and private address spaces
Although the git repository and the jenkins test results are visibile from the internet, the tests are run using an address space and hostnames that are not publicly accessible. The .novalocal top level domain name is used in the tenant dedicated to tests (bound to 10.0.0.0/8) . The .vm.april-int domain name is used by jenkins to drive OpenStack (bound to 192.168.0.0/16). The .april.org domain name is what will be exposed when the puppet master goes in production. They are used as follows:
- jenkins.april.org is also known as jenkins.vm.april-int and launches tests via ssh in april-ci.vm.april-int. april-ci.vm.april-int has no name in the april.org domain and cannot be accessed from the internet.
- april-ci.vm.april-int is an instance in the april-ci OpenStack tenant and is also known as april-ci.novalocal. It will nova boot the puppet.novalocal and nagios.novalocal instances in the april-ci OpenStack tenant.
jenkins job definition
The machine running jenkins does not contain the necessary dependencies to run the tests : they are delegated to an instance containing the nova command which is necessary to boot an OpenStack instance:
: ${CI:=april-ci.vm.april-int}
...
scp ${PRIVKEY_PATH} ${PUBKEY_PATH} root@${CI}:.ssh/
rsync -av ${CONFIG_DIR}/ root@${CI}:${CONFIG_DIR}/
ssh root@${CI} rm -fr "$directory"
ssh root@${CI} git clone "$git" "$directory"
ssh root@${CI} cd "$directory" \; bash jenkins/run-test-in-openstack.sh ${CONFIG_DIR} ${AVAILABILITY_ZONE} "$git" "$directory"
It is called from jenkins as follows:
$HOME/openstack-test.sh https://agir.april.org/git/admins.git admins
When the test succeeds, it pushes the commit to the production branch.
testing the puppetmaster
The test script assumes it runs in the context of a dedicated OpenStack tenant and that a puppet client image exists under the name puppet. From this it will:
- create a puppetmaster image
- boot a puppet.novalocal instance from this puppetmaster image
- copy the puppet master configuration from the git repository to /etc/puppet
- boot a nagios.novalocal instance from the puppet client image
- wait for nagios.novalocal to be deployed using the puppet master manifest and run the nagios service
- check the nagios status to assert that the deployment was successfull
The ssh key of jenkins is registered to OpenStack and will be used to run ssh commands on the newly created instances.
function key_exists() {
local name="$1"
nova keypair-list | grep "$name"
}
function key_add() {
local name="$1"
local pubkey="$2"
local command="nova keypair-add --pub_key \"$pubkey\" \"$name\""
eval $command
if ! key_exists "$name" ; then
echo "The command '$command' was run but 'nova keypair-list' does not show $name"
return 1
else
return 0
fi
}
function key_exists_or_add() {
local name="$1"
local pubkey="$2"
key_exists "$name" || key_add "$name" "$pubkey"
}
create a puppetmaster image
Although it would be possible to re-install the puppet master packages every time, preparing a bootable image after installation will reduce the time required for the test to complete. When instance_run ( which calls nova boot ) returns, the machine is still in the process of booting and launching the ssh server. The nmap command is used to wait for for the ssh port to become active, waiting one second between each attempt. The same method is used when a command has to wait for an asynchronous process to complete.
function create_puppetmaster() {
instance_run puppetmaster puppet e.1-cpu.10GB-disk.512MB-ram || return 1
while ! nmap puppet -PN -p ssh | grep open ; do sleep 1 ; done
while ! ssh -o 'StrictHostKeyChecking=false' root@puppetmaster apt-get update -o Acquire::Pdiffs=false ; do sleep 1 ; done
ssh root@puppetmaster apt-get install -qy puppet augeas-tools puppetmaster sqlite3 libsqlite3-ruby libactiverecord-ruby git
ssh root@puppetmaster apt-get remove -y ruby1.9.1
nova image-create --poll puppet puppetmaster
nova delete puppetmaster
}
The image created can then reduced in since on the bare metal machine running the glance repository. If the instance uuid is 6465a3a1-bc1e-48db-84a7-b78a3976c609 the resize2fs command can be used to reduce it to its minimal size:
e2fsck -f /var/lib/glance/images/6465a3a1-bc1e-48db-84a7-b78a3976c609 resize2fs -M /var/lib/glance/images/6465a3a1-bc1e-48db-84a7-b78a3976c609
If the image has already been copied over to the /var/lib/nova/instances/_base directory of a compute node, it won’t be updated.
boot the puppet.novalocal instance
The puppetmaster image is booted to create a puppet.novalocal instance. A puppet client with a fully qualified domain name foo.novalcal will assume a puppet master is available from puppet.novalocal.
ssh root@${PUPPETMASTER[instance]} rm -fr /srv/"$directory"
ssh root@${PUPPETMASTER[instance]} git clone "$git" /srv/"$directory"
ssh root@${PUPPETMASTER[instance]} cd /srv/"$directory" \; git submodule init \; git submodule update
ssh root@${PUPPETMASTER[instance]} rm -fr /etc/puppet/\*.conf /etc/puppet/\{files,manifests,modules\}
ssh root@${PUPPETMASTER[instance]} ln -s /srv/admins/puppetmaster/\*.conf /srv/admins/puppetmaster/\{files,manifests,modules\} /etc/puppet
The $git repository is cloned and linked to /etc/puppet. If the puppet.novalocal instance already exists, it is reused and the git repository is pulled for updates.
ssh root@${PUPPETMASTER[instance]} cd /srv/"$directory" \; git pull
The puppet master is run to accept all puppet clients without confirmation.
ssh root@${PUPPETMASTER[instance]} \
echo "'DAEMON_OPTS=\"--autosign true\"'" \>\> \
/etc/default/puppetmaster
Although this is dangerous in a production environment, there is no risk in the test tenant since the puppet master has no IP address that can be reached from the internet.
booting a puppet deployable instance
An instance is created by the name nagios.novalocal and using a puppet client. It will be configured using the puppet manifest prepared for it.
node nagios_server inherits ssh_server {
######################################################
package { 'nagios3': ensure => present, }
file { '/etc/nagios3/conf.d/contacts_nagios2.cfg':
ensure => present,
owner => 'root',
group => 'root',
mode => 0444,
replace => true,
source => 'puppet:///files/nagios/contacts_nagios2.cfg',
require => Package['nagios3'],
}
package {
'check-mk-livestatus': ensure => present,
require => Package['nagios3'],
}
file { '/etc/nagios3/conf.d/hosts.cfg':
ensure => present,
owner => 'root',
group => 'root',
mode => 0444,
before => Service['nagios3'],
}
exec { 'load livestatus':
command => "bash -c 'echo broker_module=/usr/lib/check_mk/livestatus.o /var/lib/nagios3/rw/live >> /etc/nagios3/nagios.cfg'",
require => Package['check-mk-livestatus'],
notify => Service['nagios3'],
onlyif => "bash -c '! grep livestatus /etc/nagios3/nagios.cfg'",
}
service { 'nagios3':
ensure => running,
enable => true,
hasrestart => true,
}
Nagios_host <<||>>
######################################################
}
The livestatus package is installed and configured as a tool to query the nagios database from the command line.
node 'nagios-hetzner.vm.april-int','nagios.novalocal' inherits nagios_server {
}
The nagios.novalocal hostname is used as an alternative to nagios-hetzner.vm.april-int which will be used in the production environment. This should be the only difference between the production environment and the test environment.
Although augeas could be used to set the broker_module line, it fails to parse the line because of the white space.
check the nagios status
The nagios checks assert the deployment works as expected. The nagios database is queried and the test script will return true if it is all ok.
function check_test_results() {
while ! echo -e "GET hosts\nFilter: name = nagios.novalocal" | \
ssh root@nagios unixcat /var/lib/nagios3/rw/live | \
grep "PING OK" ; do
sleep 1
done
}
The result will show in the jenkins console output.
+ grep 'PING OK' + ssh root@nagios unixcat /var/lib/nagios3/rw/live + echo -e 'GET hosts\nFilter: name = nagios.novalocal' + sleep 1 + grep 'PING OK' + ssh root@nagios unixcat /var/lib/nagios3/rw/live + echo -e 'GET hosts\nFilter: name = nagios.novalocal' 1;0;0;;;1;10.145.9.5;nagios;check-host-alive;...;PING OK - Packet loss = 0%... + cleanup
production upgrade
The puppet.vm.april-int puppet master is upgraded manually as follows. Note that the puppet configuration directory is a symbolic link to a checkout of the production branch.
root@puppet:/srv/admins# ls -l /etc/puppet lrwxrwxrwx 1 root root 24 Nov 20 15:22 /etc/puppet -> /srv/admins/puppetmaster
root@puppet:/srv/admins# git remote -v origin https://agir.april.org/git/admins.git (fetch) origin https://agir.april.org/git/admins.git (push) root@puppet:/srv/admins# git pull Updating 5efbe80..cf59d69 Fast-forward .gitmodules | 6 +++ jenkins/openstack-test.sh | 53 +++++++++++++++++++++++++++ jenkins/openstack-zabbix-ci.sh | 162 --------------------------------------------------------------------------------- jenkins/run-test-in-openstack.sh | 215 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ puppetmaster/manifests/site.pp | 43 ++++++++++++++++++++-- puppetmaster/modules/apt | 1 + 6 files changed, 315 insertions(+), 165 deletions(-) create mode 100755 jenkins/openstack-test.sh delete mode 100644 jenkins/openstack-zabbix-ci.sh create mode 100644 jenkins/run-test-in-openstack.sh create mode 160000 puppetmaster/modules/apt root@puppet:/srv/admins# git branch -v master 5efbe80 [behind 19] ajout du support nagios, configuration screenrc et correction des noms de domaine. refs #1053 * production cf59d69 Set the nagios password for debugging purposes because debian non interactive installation does not create any user. During cleanup, remove the certificates for all instances used during the test otherwise they will block the puppetization of another machine with the same name. refs #1041. root@puppet:/srv/admins# git submodule init Submodule 'puppetmaster/modules/apt' (https://github.com/puppetlabs/puppetlabs-apt.git) registered for path 'puppetmaster/modules/apt' Submodule 'puppetmaster/modules/concat' (git://github.com/ripienaar/puppet-concat.git) registered for path 'puppetmaster/modules/concat' Submodule 'puppetmaster/modules/ssh' (git://github.com/saz/puppet-ssh.git) registered for path 'puppetmaster/modules/ssh' Submodule 'puppetmaster/modules/stdlib' (git://github.com/puppetlabs/puppetlabs-stdlib.git) registered for path 'puppetmaster/modules/stdlib' root@puppet:/srv/admins# git submodule update Cloning into 'puppetmaster/modules/apt'... remote: Counting objects: 891, done. remote: Compressing objects: 100% (398/398), done. remote: Total 891 (delta 527), reused 812 (delta 477) Receiving objects: 100% (891/891), 124.29 KiB, done. Resolving deltas: 100% (527/527), done. Submodule path 'puppetmaster/modules/apt': checked out '7798599937f57371ebae0960b9ff2c14ac39156d' Cloning into 'puppetmaster/modules/concat'... remote: Counting objects: 263, done. remote: Compressing objects: 100% (151/151), done. remote: Total 263 (delta 114), reused 242 (delta 101) Receiving objects: 100% (263/263), 44.98 KiB, done. Resolving deltas: 100% (114/114), done. Submodule path 'puppetmaster/modules/concat': checked out '483edb52dfb3f79a3bedb3179c824d27397ef067' Cloning into 'puppetmaster/modules/ssh'... remote: Counting objects: 211, done. remote: Compressing objects: 100% (136/136), done. remote: Total 211 (delta 105), reused 170 (delta 64) Receiving objects: 100% (211/211), 22.59 KiB, done. Resolving deltas: 100% (105/105), done. Submodule path 'puppetmaster/modules/ssh': checked out 'c6fe1c360374c8fc5b664091a828d9dab0a6a515' Cloning into 'puppetmaster/modules/stdlib'... remote: Counting objects: 3022, done. remote: Compressing objects: 100% (1236/1236), done. remote: Total 3022 (delta 1562), reused 2847 (delta 1413) Receiving objects: 100% (3022/3022), 417.99 KiB | 709 KiB/s, done. Resolving deltas: 100% (1562/1562), done. Submodule path 'puppetmaster/modules/stdlib': checked out '326a13908ba8a725a85330d040e3c5cc0b211524'
One Reply to “realistic puppet tests with jenkins and OpenStack (part 2/2)”
Comments are closed.