ganeti for the gcc compile farm and OpenStreetMap

A ganeti cluster was setup on a GCC Compile Farm machine. A virtual machine was created for the benefit of OpenStreetMap. It will replace the current virtual machine with better I/O.

Non intrusive ganeti installation

The machines on the GCC Compile Farm are setup to host user accounts on various architectures. It is sometime convenient to run a virtual machine instead of developing in a user account. For instance one can test a system wide installation without disrupting the other users. The idea was to install a ganeti cluster in the least intrusive way. The installed machine was modified for :

  • NAT the virtual machines:
          up echo 1 > /proc/sys/net/ipv4/ip_forward
          up iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
  • Use a local DNS.

The installation followed the step of a more ambitious installation and stripped the OSPF part which is not necessary as a first step.


In squeeze all the software required to run ganeti2 are standard. A year ago it was significantly more difficult as it required the compilation of the DRBD from sources, among other things.
The second disk of the machine was used to create a single LVM partition

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      121601   976760001   8e  Linux LVM

that was bound to the all volume group:

root@gcc20:~# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  all    1   2   0 wz--n- 931.51g 421.51g

The default DRBD configuration was removed so that ganeti can handle it on its own.

mv /etc/drbd.conf /etc/drbd.conf.old
touch /etc/drbd.conf
/etc/init.d/drbd reload


A bridge and a fake lo:1 local interface were created in /etc/network/interfaces:

auto lo:1
iface lo:1 inet static
	up ip route add blackhole

auto br0
iface br0 inet static
        bridge_ports none
        bridge_stp off
	bridge_maxwait 5
        up ip route add dev br0
        up ip route add dev br0


Ganeti hosts and virtual machines names are DNS entries. It would be possible to use a public
DNS to create them. A local DNS configuration is created localy instead, to
reduce the propagation delays, ensure it can always be reached and isolate the maintainance
of the name space into a private network. The top level domain farm was created
and divided into ( bound to ) for all the ganeti hosts
and ( bound to ) for all the virtual machines.

Do not listen or answer on public IPs.

        allow-recursion {;; };
        listen-on {;; };
zone "" {
    type master;
    file "/etc/bind/db.10.10";

zone "farm" {
        type master;
        file "/etc/bind/";
; -*- mode: zone; -*-
; BIND reverse data file for broadcast zone
$TTL    604800
@       IN      SOA     localhost. root.localhost. (
        2011031200      ; serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
@       IN      NS      localhost.

;; perl -n -e 'print if(s/(\S+)\s+IN\s+A\s+10\.10\.(\d+)\.(\d+)/\3.\2\tIN\tPTR\t\' <
1.50    IN      PTR
; -*- mode: zone; -*-
$TTL 1h ; default Time-to-Live. defines the duration that the record may be cached (24h)

$ORIGIN farm.
@                       IN      SOA             ns hostmaster (
                        2011031200      ; serial
                                1h              ; refresh - time when the slave will try to refresh the zone from the master (8h)
                                30m             ; update retry - time between retries if the slave (secondary) (2h)
                                                ; fails to contact the master when refresh (above) has expired.
                                2w              ; expiry - time slave will continue to supply authoritative data for the zone (1w)
                                                ; after the master died
                                30m)            ; minimum - time a NAME ERROR = NXDOMAIN record is cached (24h)

                IN      NS      ns

ns              IN      A

rodo1.osm.vm    IN      A

A DHCP server is setup for each VM to use.

A MAC address is generated with the following oneliner:

MACADDR="52:54:$(dd if=/dev/urandom count=1 2>/dev/null |
  md5sum |
  sed 's/^\(..\)\(..\)\(..\)\(..\).*$/\1:\2:\3:\4/')"; echo $MACADDR

and used to define the entry:

group {
  option domain-name "farm";
  option routers;
  option domain-name-servers;
  subnet netmask {
    host {
        hardware ethernet 52:54:5a:8d:77:8e;
        option subnet-mask;

For both DHCP and the DNS, the configuration directories ( /etc/dhcp and /etc/bind respectively ) a git repository was created to keep an history of the changes.

Cluster and instance creation

The hostname of the machine is gcc20 which is bound to This is not acceptable for ganeti2. When creating a cluster, the hostname is used to create the first node of the cluster and make it the master. For this operation to succeed, the hostname was temporarily changed to and it's IP set in /etc/hosts to This was permanently set on eth0 in the /etc/network/interface as follows:

	up ip addr add broadcast dev eth0

The corresponding entries were added to the bind configuration:        IN      A              IN      A

The prerquisite to run gnt-cluster init were met:

  • IP is not 127.*
  • IP is not bound to any interface
  • The cluster was created with:

    gnt-cluster init --no-etc-hosts \
                     --enabled-hypervisors=kvm \
                     --nic-parameters link=br0 \
                     --master-netdev br0 \
                     --vg-name all

    After downloading the installation CD of Debian GNU/Linux squeeze, the OpenStreetMap instance was created:

    gnt-instance add -d -t plain -s 10G -B memory=512M,vcpus=1 \
      -H kvm:boot_order=cdrom,cdrom_image_path=/srv/ganeti/debian-6.0.0-amd64-CD-1.iso,vnc_bind_address= \
      -n -o debootstrap+default --net 0:mac=52:54:5a:8d:77:8e

    The installation process was done using the VNC console


    Because the DHCP server delivers a IP and a default route that is not on the same subnet ( ) the DHCP client fails to setup the default route during the installation process. When the network initialization asks if the installation should proceed without a default route, the network was configured manually with the following:

    • IP
    • Netmask
    • DNS
    • Gateway

    After the installation completed, the disk was made the primary boot device:

    gnt-instance modify -H boot_order=disk

    and the VM rebooted:

    gnt-instance restart --shutdown-timeout=1

    OpenStreeMap needs a lot of disk space and an additional 500GB disk was created for the virtual machine.

    gnt-instance modify --disk add:size=500G

    From within the virtual machine it shows as /dev/vdb.