Posted on

gerrit with jenkins : installation and configuration

The review and project management for Git based projects is installed on a virgin Debian GNU/Linux wheezy. Developers of l2mesh must submit patchs to the git repository to gerrit:

$ git review
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: new: 1, done
To ssh://loic@gerrit.the.re:29418/l2mesh
 * [new branch]      HEAD -> refs/publish/master/master

Gerrit is bound to jenkins : it will run tests on the patch to be reviewed and let gerrit know if it succeeds. If a developer reviews the patch positively, it can be merged into the repository.


Continue reading “gerrit with jenkins : installation and configuration”

Posted on

flushing OpenVPN routes to prevent temporary incorrect routing

An OpenVPN client routes 192.168.2.0/24.

root@controller:~# ip route show 192.168.2.0/24
192.168.2.0/24 via 192.168.0.21 dev tun0

When the OpenVPN client is down, IP in the 192.168.2.0/24 network will be cached to go thru the default route instead:

root@controller:~# ip route show cache 192.168.2.42
192.168.2.42 via 10.145.4.4 dev eth0  src 10.145.4.5

When the OpenVPN client is back, the cache needs to be flushed to prevent temporary incorrect routing.

root@controller:~# ip route flush cache

Continue reading “flushing OpenVPN routes to prevent temporary incorrect routing”

Posted on

puppet master hierarchy organization and conventions

April hosts dozens of services on less than ten hardware machines and less than fifty virtual machines. Their configuration is centralized in a puppet master repository. The order of magnitude is not expected to change in the next few years. The hierarchy is organized in manifests and modules. The manifests directory contains the inventory of all virtual machines and bare metal associated with the inclusion of classes and the instantiation of types.

node 'harmine.pavot.vm.april-int',
     'backuppc.novalocal' inherits vserver-pavot {
  include backuppc::server
  include april_nagios::nrpe_server
  include april_nagios::check_backuppc
}

The modules contain the configuration logic for a sub system such as backuppc, screen or ssh.

class screen {
  package { 'screen': ensure => present, }

  file { '/root/.screenrc':
    ensure  => present,
    owner   => 'root',
    group   => 'root',
    mode    => 0400,
    replace => true,
    source  => 'puppet:///screen/.screenrc',
  }

}

Continue reading “puppet master hierarchy organization and conventions”

Posted on

anatomy of an OpenStack based integration test for a backuppc puppet module

An integration test is run by jenkins within an OpenStack tenant. It checks that the backuppc puppet module is installed

ssh root@$instance test -f /etc/backuppc/hosts || return 3

A full backup is run

ssh root@$instance su -c '"/usr/share/backuppc/bin/BackupPC_serverMesg \
   backup nagios.novalocal nagios.novalocal backuppc 1"' \
                              backuppc || return 4
ssh root@$instance tail -f /var/lib/backuppc/pc/nagios.novalocal/LOG.* | \
    sed --unbuffered -e "s/^/$instance: /" -e '/full backup 0 complete/q'

and a nagios plugin asserts its status is monitored

    while ! ( echo "GET services"
        echo "Filter: host_alias = $instance.novalocal"
        echo "Filter: check_command = check_nrpe_1arg"'!'"check_backuppc" ) |
        ssh root@nagios unixcat /var/lib/nagios3/rw/live |
        grep "BACKUPPC OK - (0/" ; do
        sleep 1
    done

Continue reading “anatomy of an OpenStack based integration test for a backuppc puppet module”

Posted on

nagios puppet module for the April infrastructure

This document explains the nagios configuration for the infrastructure of the April non profit organisation.
It is used to configure the nagios server overseeing all the services. The nagios plugins that cannot be run from the server ( such as check_oom_killer ) are installed locally and connected to nagios with nrpe. All services are bound to private IPs within the 192.168.0.0/16 network and exposed to the nagios server ( using OpenVPN to connect bare metal machines together ) and the firewalls are set to allow TCP on the nrpe port ( 5666 ).
Continue reading “nagios puppet module for the April infrastructure”

Posted on

Migrating OpenVZ virtual machines to OpenStack

A OpenVZ cluster hosts GNU/Linux based virtual machines. The disk is extracted with rsync and uploaded to the glance OpenStack image service with glance add … disk_format=ami…. It is associated with a kernel image compatible with both OpenStack and the existing file system with glance update … kernel_id=0dfff976-1f55-4184-954c-a111f4a28eef ramdisk_id=aa87c84c-d3be-41d0-a272-0b4a85801a34 ….
Continue reading “Migrating OpenVZ virtual machines to OpenStack”

Posted on

realistic puppet tests with jenkins and OpenStack (part 2/2)

The April infrastructure uses puppet manifests stored in a git repository. On each commit, a jenkins job is run and it performs realistic tests in a dedicated OpenStack tenant.

If the test is successfull, jenkins pushes the commit to the production branch. The production machines can then pull from it:

root@puppet:/srv/admins# git pull
Updating 5efbe80..cf59d69
Fast-forward
 .gitmodules                      |    6 +++
 jenkins/openstack-test.sh        |   53 +++++++++++++++++++++++++++
  jenkins/run-test-in-openstack.sh |  215 +++++++++++++++++++++++++++
 puppetmaster/manifests/site.pp   |   43 ++++++++++++++++++++--
 puppetmaster/modules/apt         |    1 +
 6 files changed, 315 insertions(+), 165 deletions(-)
 create mode 100755 jenkins/openstack-test.sh
 create mode 100644 jenkins/run-test-in-openstack.sh
 create mode 160000 puppetmaster/modules/apt
root@puppet:/srv/admins# git branch -v
  master     5efbe80 [behind 19] ajout du support nagios, configuration .... refs #1053
* production cf59d69 Set the nagios password for debugging ...

Continue reading “realistic puppet tests with jenkins and OpenStack (part 2/2)”

Posted on

shared git repository over ssh

A git repository is configured to be shared between users from the same group with:

# grep group /srv/repos/git/l2mesh.git/config
        shared = group

The repository directory is set with the group sticky bit so that files created by a user have the same group and permissions as the directory:

root@redmine ~# ls -ld /srv/repos/git/l2mesh.git/config
-rw-rw-r-- 1 root git 82 Nov 23 13:09 /srv/repos/git/l2mesh.git/config

The repository can then be cloned with

git clone ssh://redmine.the.re/srv/repos/git/l2mesh.git

Continue reading “shared git repository over ssh”

Posted on

routing ipv6 from Hetzner to an OpenStack instance

The 2a01:4f8:162:12e3::2 IPv6 address is assigned to http://packaging-farm.dachary.org/ from the IPv6 subnet provided by Hetzner. The OpenStack host on which the instance running packaging-farm.dachary.org is running is configured as a proxy with

sysctl -w net.ipv6.conf.all.proxy_ndp=1
ip -6 neigh add proxy 2a01:4f8:162:12e3::2 dev eth0

and an OpenStack Essex bug is worked around by manually disabling hairpin_mode:

echo 0 > /sys/class/net/br2003/brif/vnet1/hairpin_mode

The page can then be retrieved with

$ curl --verbose -6 http://packaging-farm.dachary.org/
* About to connect() to packaging-farm.dachary.org port 80 (#0)
*   Trying 2a01:4f8:162:12e3::2...
* connected
* Connected to packaging-farm.dachary.org (2a01:4f8:162:12e3::2) port 80 (#0)
> Host: packaging-farm.dachary.org
> Accept: */*
>
...
<address>Apache/2.2.19 (Debian) Server at packaging-farm.dachary.org Port 80</address>
</body></html>
* Connection #0 to host packaging-farm.dachary.org left intact
* Closing connection #0

Continue reading “routing ipv6 from Hetzner to an OpenStack instance”