Building Ceph Debian GNU/Linux packages

The following script explains how to create Debian GNU/Linux packages for Ceph from a clone of the sources.

rm -fr releasedir
mkdir -p $releasedir
# remove all files not under git so they are not
# included in the distribution.
git clean -dxf
# git describe provides a version that is
# a) human readable
# b) is unique for each commit
# c) compares higher than any previous commit
# d) contains the short hash of the commit
vers=`git describe --match "v*" | sed s/^v//`
# creating the distribution tarbal requires some configure
# options (otherwise parts of the source tree will be left out).
./configure --with-rocksdb --with-ocf --with-rest-bench \
    --with-nss --with-debug --enable-cephfs-java \
    --with-lttng --with-babeltrace
# use distdir= to set the name of the top level directory of the
# tarbal to match the desired version
make distdir=ceph-$vers dist
# rename the tarbal to match debian conventions and extract it
mv ceph-$vers.tar.gz $releasedir/ceph_$vers.orig.tar.gz
tar -C $releasedir zxf ceph_$vers.orig.tar.gz
# copy the debian directory over and remove -dbg packages
# because they are large and take time to build
cp -a debian $releasedir/ceph-$vers/debian
cd $releasedir
perl -ni -e 'print if(!(/^Package: .*-dbg$/../^$/))' ceph-$vers/debian/control
perl -pi -e 's/--dbg-package.*//' ceph-$vers/debian/rules
# always set the debian version to 1 which is ok because the debian
# directory is included in the sources and the upstream version will
# change each time it is modified.
# update the changelog to match the desired version
cd ceph-$vers
chvers=`head -1 debian/changelog | perl -ne 's/.*\(//; s/\).*//; print'`
if [ "$chvers" != "$dvers" ]; then
   DEBEMAIL="" dch -b -v "$dvers" "new version"
# create the packages
# a) with ccache to speed things up when building repeatedly
# b) do not sign the packages
# c) use half of the available processors
PATH=/usr/lib/ccache:$PATH dpkg-buildpackage -j$(($(nproc) / 2)) -uc -us

Continue reading “Building Ceph Debian GNU/Linux packages”

Ceph early adopter : Université de Nantes

Loire Chantrerie Lombarderie
The Université de Nantes started using Ceph for backups early 2012, before the Bobtail was released or Inktank founded. The IRTS department, under the lead of Yann Dupont, created a twelve nodes Ceph cluster to store backups. It contains the data generated by 35,000 students and 4,500 employees totaling 100 millions inodes and 25TB of data (out of 40TB). The hardware is spread accross three geographical locations ( Loire, Chantrerie and Lombarderie ) and Ceph is configured to keep working transparently even when one of them is down. The backup pool has two replicas and the crushmap states that each must be stored in a different geographical location. For instance, when Lombarderie is unreachable, which happened this week because of a planned power outage combined with an unplanned UPS failure, Ceph keeps serving the objects from the replicas located in Loire and Chantrerie.
Continue reading “Ceph early adopter : Université de Nantes”

Bio++: efficient, extensible libraries and tools for computational molecular evolution

Efficient algorithms and programs for the analysis of the ever-growing amount of biological sequence data are strongly needed in the genomics era. The pace at which new data and methodologies are generated calls for the use of pre-existing, optimized – yet extensible – code, typically distributed as libraries or packages. This motivated the Bio++ project, aiming at developing a set of C++ libraries for sequence analysis, phylogenetics, population genetics and molecular evolution. The main attractiveness of Bio++ is the extensibility and reusability of its components through its object-oriented design, without compromising on the computer-efficiency of the underlying methods. We present here the second major release of the libraries, which provides an extended set of classes and methods. These extensions notably provide built-in access to sequence databases and new data structures for handling and manipulating sequences from the omics era, such as multiple genome alignments and sequencing reads libraries. More complex models of sequence evolution, such as mixture models and generic n-tuples alphabets, are also included.
The article was published May 21st, 2013. Read the full article : Bio++: efficient, extensible libraries and tools for computational molecular evolution

Virtualizing legacy hardware in OpenStack

A five years old hardware is being decommissioned and hosts fourteen vservers on a Debian GNU/Linux lenny running a 2.6.26-2-vserver-686-bigmem linux kernel. The April non profit relies on these services (mediawiki, pad, mumble, etc. ) for the benefit of its 5,000 members and many working groups. Instead of migrating each vserver individually to an OpenStack instance, it was decided that the vserver host would be copied over to an OpenStack instance.
The old hardware has 8GB of RAM, 150GB disk and a dual Xeon totaling 8 cores. The munin statistics show that no additional memory is needed, the disk is half full and an average of one core is used at all times. A 8GB RAM, 150GB disk and dual core openstack instance is prepared. The instance will be booted from a 150GB volume placed on the same hardware to get maximum disk I/O speed.
After the volume is created, it is mounted from the OpenStack node and the disk of the old machine is rsync’ed to it. It is then booted after modifying a few files such as fstab. The OpenStack node is in the same rack and the same switch as the old hardware. The IP is removed from the interface of the old hardware and it is bound to the OpenStack instance. Because it is running on nova-network with multi-host activated, it is bound to the interface of the OpenStack node which can take over immediately. The public interface of the node is set as an ARP proxy to advertise the bridge where the instance is connected. The security group of the instance are disabled ( by opening all protocols and ports ) because a firewall is running in the instance.
Continue reading “Virtualizing legacy hardware in OpenStack”

Disaster recovery on host failure in OpenStack

The host becomes unavailable because of a partial disk failure on an Essex based OpenStack cluster using LVM based volumes and multi-host nova-network. The host had daily backups using rsync / and each LV was copied and compressed. Although the disk is failing badly, the host is not down and some reads can still be done. The nova services are shutdown, the host disabled using nova-manage and an attempt is made to recover from partially damaged disks and LV, when it leads to better results than reverting to yesterday’s backup.
Continue reading “Disaster recovery on host failure in OpenStack”

nova-network debugging tips

A single machine is installed with Debian GNU/Linux OpenStack Folsom. Four instances are created and it turns out that nova-network is configured with the wrong public interface. It can be fixed without shutting down the instance:

nova suspend target1

The instance is suspended to disk (as if it was a laptop) and the corresponding KVM process is killed. While the instance is suspended, nova-network can be stopped.

/etc/init.d/nova-network stop

The source of the problem was a typo in the public interface leading to an incorrect VLAN interface

13: vlan100@eth2:  mtu 1500 qdisc noqueue state DOWN mode DEFAULT
    link/ether fa:16:3e:54:5b:57 brd ff:ff:ff:ff:ff:ff

it can be fixed in the /etc/nova/nova.conf configuration file at the line:

public_interface = eth3

The incorrect VLAN interface is manually deleted and nova-network can be restarted. The instance is then resumed with

nova resume target1

and nova-network will automatically re-create the VLAN interface.
Continue reading “nova-network debugging tips”

Installing OpenStack Folsom on Debian GNU/Linux wheezy

Installing and testing OpenStack Folsom on a virgin Debian GNU/Linux wheezy takes less than one hour. A set of packages is archived to make sure it keeps working. After checking the pre-requisites such as a public and private interface, the packages are installed and debconf questions answered as instructed.
The networks must then be created with

nova-manage network create private --fixed_range_v4= \
  --network_size=256 --num_networks=2 --vlan=100

/etc/nova/nova.conf is updated to set vlan_interface=dummy0, public_interface=eth0 and fixed_range= /etc/nova/nova-compute.conf is updated to use LibvirtBridgeDriver and an instance can be booted with:

nova boot --poll --flavor m1.tiny --image cirrOS-0.3.0-x86_64 \
  --key_name loic test

Continue reading “Installing OpenStack Folsom on Debian GNU/Linux wheezy”

flushing OpenVPN routes to prevent temporary incorrect routing

An OpenVPN client routes

root@controller:~# ip route show via dev tun0

When the OpenVPN client is down, IP in the network will be cached to go thru the default route instead:

root@controller:~# ip route show cache via dev eth0  src

When the OpenVPN client is back, the cache needs to be flushed to prevent temporary incorrect routing.

root@controller:~# ip route flush cache

Continue reading “flushing OpenVPN routes to prevent temporary incorrect routing”

anatomy of an OpenStack based integration test for a backuppc puppet module

An integration test is run by jenkins within an OpenStack tenant. It checks that the backuppc puppet module is installed

ssh root@$instance test -f /etc/backuppc/hosts || return 3

A full backup is run

ssh root@$instance su -c '"/usr/share/backuppc/bin/BackupPC_serverMesg \
   backup nagios.novalocal nagios.novalocal backuppc 1"' \
                              backuppc || return 4
ssh root@$instance tail -f /var/lib/backuppc/pc/nagios.novalocal/LOG.* | \
    sed --unbuffered -e "s/^/$instance: /" -e '/full backup 0 complete/q'

and a nagios plugin asserts its status is monitored

    while ! ( echo "GET services"
        echo "Filter: host_alias = $instance.novalocal"
        echo "Filter: check_command = check_nrpe_1arg"'!'"check_backuppc" ) |
        ssh root@nagios unixcat /var/lib/nagios3/rw/live |
        grep "BACKUPPC OK - (0/" ; do
        sleep 1

Continue reading “anatomy of an OpenStack based integration test for a backuppc puppet module”

realistic puppet tests with jenkins and OpenStack (part 2/2)

The April infrastructure uses puppet manifests stored in a git repository. On each commit, a jenkins job is run and it performs realistic tests in a dedicated OpenStack tenant.

If the test is successfull, jenkins pushes the commit to the production branch. The production machines can then pull from it:

root@puppet:/srv/admins# git pull
Updating 5efbe80..cf59d69
 .gitmodules                      |    6 +++
 jenkins/        |   53 +++++++++++++++++++++++++++
  jenkins/ |  215 +++++++++++++++++++++++++++
 puppetmaster/manifests/site.pp   |   43 ++++++++++++++++++++--
 puppetmaster/modules/apt         |    1 +
 6 files changed, 315 insertions(+), 165 deletions(-)
 create mode 100755 jenkins/
 create mode 100644 jenkins/
 create mode 160000 puppetmaster/modules/apt
root@puppet:/srv/admins# git branch -v
  master     5efbe80 [behind 19] ajout du support nagios, configuration .... refs #1053
* production cf59d69 Set the nagios password for debugging ...

Continue reading “realistic puppet tests with jenkins and OpenStack (part 2/2)”