Posted on

flushing OpenVPN routes to prevent temporary incorrect routing

An OpenVPN client routes 192.168.2.0/24.

root@controller:~# ip route show 192.168.2.0/24
192.168.2.0/24 via 192.168.0.21 dev tun0

When the OpenVPN client is down, IP in the 192.168.2.0/24 network will be cached to go thru the default route instead:

root@controller:~# ip route show cache 192.168.2.42
192.168.2.42 via 10.145.4.4 dev eth0  src 10.145.4.5

When the OpenVPN client is back, the cache needs to be flushed to prevent temporary incorrect routing.

root@controller:~# ip route flush cache

Continue reading “flushing OpenVPN routes to prevent temporary incorrect routing”

Posted on

puppet master hierarchy organization and conventions

April hosts dozens of services on less than ten hardware machines and less than fifty virtual machines. Their configuration is centralized in a puppet master repository. The order of magnitude is not expected to change in the next few years. The hierarchy is organized in manifests and modules. The manifests directory contains the inventory of all virtual machines and bare metal associated with the inclusion of classes and the instantiation of types.

node 'harmine.pavot.vm.april-int',
     'backuppc.novalocal' inherits vserver-pavot {
  include backuppc::server
  include april_nagios::nrpe_server
  include april_nagios::check_backuppc
}

The modules contain the configuration logic for a sub system such as backuppc, screen or ssh.

class screen {
  package { 'screen': ensure => present, }

  file { '/root/.screenrc':
    ensure  => present,
    owner   => 'root',
    group   => 'root',
    mode    => 0400,
    replace => true,
    source  => 'puppet:///screen/.screenrc',
  }

}

Continue reading “puppet master hierarchy organization and conventions”

Posted on

anatomy of an OpenStack based integration test for a backuppc puppet module

An integration test is run by jenkins within an OpenStack tenant. It checks that the backuppc puppet module is installed

ssh root@$instance test -f /etc/backuppc/hosts || return 3

A full backup is run

ssh root@$instance su -c '"/usr/share/backuppc/bin/BackupPC_serverMesg \
   backup nagios.novalocal nagios.novalocal backuppc 1"' \
                              backuppc || return 4
ssh root@$instance tail -f /var/lib/backuppc/pc/nagios.novalocal/LOG.* | \
    sed --unbuffered -e "s/^/$instance: /" -e '/full backup 0 complete/q'

and a nagios plugin asserts its status is monitored

    while ! ( echo "GET services"
        echo "Filter: host_alias = $instance.novalocal"
        echo "Filter: check_command = check_nrpe_1arg"'!'"check_backuppc" ) |
        ssh root@nagios unixcat /var/lib/nagios3/rw/live |
        grep "BACKUPPC OK - (0/" ; do
        sleep 1
    done

Continue reading “anatomy of an OpenStack based integration test for a backuppc puppet module”

Posted on

nagios puppet module for the April infrastructure

This document explains the nagios configuration for the infrastructure of the April non profit organisation.
It is used to configure the nagios server overseeing all the services. The nagios plugins that cannot be run from the server ( such as check_oom_killer ) are installed locally and connected to nagios with nrpe. All services are bound to private IPs within the 192.168.0.0/16 network and exposed to the nagios server ( using OpenVPN to connect bare metal machines together ) and the firewalls are set to allow TCP on the nrpe port ( 5666 ).
Continue reading “nagios puppet module for the April infrastructure”

Posted on

Migrating OpenVZ virtual machines to OpenStack

A OpenVZ cluster hosts GNU/Linux based virtual machines. The disk is extracted with rsync and uploaded to the glance OpenStack image service with glance add … disk_format=ami…. It is associated with a kernel image compatible with both OpenStack and the existing file system with glance update … kernel_id=0dfff976-1f55-4184-954c-a111f4a28eef ramdisk_id=aa87c84c-d3be-41d0-a272-0b4a85801a34 ….
Continue reading “Migrating OpenVZ virtual machines to OpenStack”

Posted on

realistic puppet tests with jenkins and OpenStack (part 2/2)

The April infrastructure uses puppet manifests stored in a git repository. On each commit, a jenkins job is run and it performs realistic tests in a dedicated OpenStack tenant.

If the test is successfull, jenkins pushes the commit to the production branch. The production machines can then pull from it:

root@puppet:/srv/admins# git pull
Updating 5efbe80..cf59d69
Fast-forward
 .gitmodules                      |    6 +++
 jenkins/openstack-test.sh        |   53 +++++++++++++++++++++++++++
  jenkins/run-test-in-openstack.sh |  215 +++++++++++++++++++++++++++
 puppetmaster/manifests/site.pp   |   43 ++++++++++++++++++++--
 puppetmaster/modules/apt         |    1 +
 6 files changed, 315 insertions(+), 165 deletions(-)
 create mode 100755 jenkins/openstack-test.sh
 create mode 100644 jenkins/run-test-in-openstack.sh
 create mode 160000 puppetmaster/modules/apt
root@puppet:/srv/admins# git branch -v
  master     5efbe80 [behind 19] ajout du support nagios, configuration .... refs #1053
* production cf59d69 Set the nagios password for debugging ...

Continue reading “realistic puppet tests with jenkins and OpenStack (part 2/2)”