An integration test is run by jenkins within an OpenStack tenant. It checks that the backuppc puppet module is installed
ssh root@$instance test -f /etc/backuppc/hosts || return 3
A full backup is run
ssh root@$instance su -c '"/usr/share/backuppc/bin/BackupPC_serverMesg \
backup nagios.novalocal nagios.novalocal backuppc 1"' \
backuppc || return 4
ssh root@$instance tail -f /var/lib/backuppc/pc/nagios.novalocal/LOG.* | \
sed --unbuffered -e "s/^/$instance: /" -e '/full backup 0 complete/q'
and a nagios plugin asserts its status is monitored
while ! ( echo "GET services"
echo "Filter: host_alias = $instance.novalocal"
echo "Filter: check_command = check_nrpe_1arg"'!'"check_backuppc" ) |
ssh root@nagios unixcat /var/lib/nagios3/rw/live |
grep "BACKUPPC OK - (0/" ; do
sleep 1
done
The test environment
The test is run by the run-test-in-openstack.sh script which is called from jenkins in a dedicated OpenStack tenant.
function run_tests() {
local directory="$1"
for path in $(find * -name test-in-openstack.sh) ; do
source "$path" || return 1
done
}
A puppet master ( puppet.novalocal ) and a nagios server ( nagios.novalocal ) are run before calling the script. The puppet master is loaded with the master branch of the git repository and the nagios server is deployed from it using the april_nagios puppet module with:
node 'nagios-hetzner.vm.april-int',
'nagios.vm.april-int',
'nagios.novalocal' inherits openstack-instance {
include april_nagios::server
}
The search domain within the OpenStack environment is novalocal and the nagios server can be referred to as nagios instead of nagios.novalocal.
The puppet master is configured to accept all new puppet clients without human interaction to allow for a fully automated run:
echo "'DAEMON_OPTS=\"--autosign true\"'" \>\> /etc/default/puppetmaster
The ssh public key of the user under which the tests are running is installed in the OpenStack tenant and will be used when creating instances so that the tests can run ssh commands using ssh root@instancename
function key_add() {
local name="$1"
local pubkey="$2"
local command="nova keypair-add --pub_key \"$pubkey\" \"$name\""
eval $command
if ! key_exists "$name" ; then
echo "The command '$command' was run but 'nova keypair-list' does not show $name"
return 1
else
return 0
fi
}
Running puppet unit tests
The puppet.novalocal puppet master is setup to run puppet rspec unit tests. A specific set of ruby gems is installed and known to work with ruby-1.8 and puppet-2.7 or puppet-2.6.
gem install --ignore-dependencies --version 1.1.3 diff-lcs gem install --ignore-dependencies --version 1.6.14 facter gem install --ignore-dependencies --version 0.0.1 metaclass gem install --ignore-dependencies --version 0.13.0 mocha gem install --ignore-dependencies --version 2.7.18 puppet gem install --ignore-dependencies --version 0.1.13 puppet-lint gem install --ignore-dependencies --version 0.2.0 puppetlabs_spec_helper gem install --ignore-dependencies --version 10.0.2 rake gem install --ignore-dependencies --version 2.12.0 rspec gem install --ignore-dependencies --version 2.12.0 rspec-core gem install --ignore-dependencies --version 2.12.0 rspec-expectations gem install --ignore-dependencies --version 2.12.0 rspec-mocks gem install --ignore-dependencies --version 0.1.4 rspec-puppet
So that the tests can be run with:
'GEM_HOME=$HOME/.gem-installed' 'PATH=$HOME/.gem-installed/bin:$PATH' rake spec
The puppet client
The integration tests for the puppet modules make use of a puppet client image. It is booted in the OpenStack tenant with:
nova boot --image puppet ... backuppc
and it will ask for a configuration to the puppet master under the name backuppc.novalocal. The puppet master configuration must contain such host name for test purposes. For instance:
node 'harmine.pavot.vm.april-int',
'backuppc.novalocal' inherits vserver-pavot {
include backuppc::server
include april_nagios::nrpe_server
include april_nagios::check_backuppc
}
Helper functions
The instance_run function is a wrapper around nova boot that waits for the instance to show with nova list.
function instance_run() {
local instance="$1"
local image="$2"
local flavor="$3"
nova boot \
--image "$image" \
--flavor "$flavor" \
--key_name ${KEYPAIR_NAME} \
--availability_zone ${AVAILABILITY_ZONE} \
--poll \
"$instance"
while ! nova list --name="$instance" | grep "$instance" ; do sleep 1 ; done
}
The instance_delete function is a wrapper around nova delete that waits until is disapears from the output of nova list and cleanup the puppet certificate.
function instance_delete() {
local instance="$1"
nova delete "$instance" || return 0
while nova list --name="$instance" |
grep "$instance" > /dev/null ; do sleep 1 ; done
ssh root@${PUPPETMASTER[instance]} puppetca clean $instance.novalocal || true
}
Commented test script for backuppc
The test script starts by running the puppet rspec unit tests:
local instance=backuppc ssh root@puppet cd /etc/puppet/modules/$instance \; ' \ GEM_HOME=$HOME/.gem-installed' \ 'PATH=$HOME/.gem-installed/bin:$PATH' rake spec || return 1
If the test is interrupted, a previous instance named backuppc.novalocal remains and must be deleted:
instance_delete $instance
A puppet client with the hostname backuppc.novalocal is created
instance_run $instance puppet e.1-cpu.10GB-disk.512MB-ram || return 2
and the script pauses until an ssh server becomes avaiable:
while ! nmap $instance -PN -p ssh | grep open ; do sleep 1 ; done
The /var/log/daemon.log file is monitored to wait for the completion of the initial puppet agent run at boot time ( when Finished catalog run is found ).
ssh -o 'StrictHostKeyChecking=false' root@$instance \
tail -f /var/log/daemon.log | sed --unbuffered -e "s/^/$instance: /" \
-e '/Finished catalog run/q'
A minimal side effect of the backuppc puppet module
package { 'backuppc':
ensure => installed,
}
is checked : the presence of the /etc/backuppc/hosts file.
ssh root@$instance test -f /etc/backuppc/hosts || return 3
The puppet class configuring nagios makes use of exported resources, which requires an extra puppet pass:
- nagios is configured by the puppet master
- backuppc is configured by the puppet master and records ( in the puppet master ) the fact that it is a candidate to be monitored ( this is what exported resources basically do )
- nagios runs the puppet client again and is notified by the puppet master that backuppc needs to be monitored and modifies its configuration accordingly
ssh root@nagios puppet agent -vt
The next test will be to verify that the backuppc.novalocal performs a backup of nagios.novalocal because the openstack-instance.pp manifests contains:
node 'nagios-hetzner.vm.april-int',
'nagios.vm.april-int',
'nagios.novalocal' inherits openstack-instance {
include april_nagios::server
}
which inherits from openstack-instance
node openstack-instance inherits default {
april_nagios::host { $fqdn: address => $ipaddress }
include april_ssh
include backuppc::client
}
declaring the instance to be a backuppc::client. This class makes use of exported resources and requires an extra run of puppet on backuppc.novalocal to configure backuppc so that it schedules the backup of the nagios.novalocal instance.
ssh root@$instance puppet agent -vt
Instead of waiting for the backup to start, it is forced to run immediately using the backuppc command line. It will start the backup of the nagios.novalocal instance and the logs ( /var/lib/backuppc/pc/nagios.novalocal/LOG.* ) will be displayed until the full backup 0 complete string shows, proving the backup succeeded.
ssh root@$instance su -c '"/usr/share/backuppc/bin/BackupPC_serverMesg \
backup nagios.novalocal nagios.novalocal backuppc 1"' backuppc || return 4
ssh root@$instance tail -f /var/lib/backuppc/pc/nagios.novalocal/LOG.* |
sed --unbuffered -e "s/^/$instance: /" -e '/full backup 0 complete/q'
Once the backup is complete, nagios.novalocal will notice because a test specific to backuppc was installed ( it comes with the nagios-plugins-contribs package ).
while ! ( echo "GET services"
echo "Filter: host_alias = $instance.novalocal"
echo "Filter: check_command = check_nrpe_1arg"'!'"check_backuppc" ) |
ssh root@nagios unixcat /var/lib/nagios3/rw/live |
grep "BACKUPPC OK - (0/" ; do
sleep 1
done
The livestatus query language is used against the nagios server database : GET the services but Filter to get only those with the host_alias field matching backuppc.novalocal and the check_command matching check_nrpe_1arg!check_backuppc command. The line found will display the string BACKUPPC OK – (0/ on success.
Finally the backuppc.novalocal instance is destroyed.
instance_delete $instance