A ganeti cluster was setup on a GCC Compile Farm machine. A virtual machine was created for the benefit of OpenStreetMap. It will replace the current http://osm.fsffrance.org/ virtual machine with better I/O.
Non intrusive ganeti installation
The machines on the GCC Compile Farm are setup to host user accounts on various architectures. It is sometime convenient to run a virtual machine instead of developing in a user account. For instance one can test a system wide installation without disrupting the other users. The idea was to install a ganeti cluster in the least intrusive way. The installed machine was modified for :
- NAT the virtual machines:
up echo 1 > /proc/sys/net/ipv4/ip_forward up iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE - Use a local DNS.
The installation followed the step of a more ambitious installation and stripped the OSPF part which is not necessary as a first step.
LVM and DRBD
In squeeze all the software required to run ganeti2 are standard. A year ago it was significantly more difficult as it required the compilation of the DRBD from sources, among other things.
The second disk of the machine was used to create a single LVM partition
Device Boot Start End Blocks Id System /dev/sdb1 1 121601 976760001 8e Linux LVM
that was bound to the all volume group:
root@gcc20:~# vgs VG #PV #LV #SN Attr VSize VFree all 1 2 0 wz--n- 931.51g 421.51g
The default DRBD configuration was removed so that ganeti can handle it on its own.
mv /etc/drbd.conf /etc/drbd.conf.old touch /etc/drbd.conf /etc/init.d/drbd reload
Network
A bridge and a fake lo:1 local interface were created in /etc/network/interfaces:
auto lo:1
iface lo:1 inet static
address 10.10.0.20
netmask 255.255.255.255
up ip route add blackhole 10.0.0.0/8
auto br0
iface br0 inet static
address 10.10.0.254
netmask 255.255.255.255
bridge_ports none
bridge_stp off
bridge_maxwait 5
up ip route add 10.10.0.254 dev br0
up ip route add 10.10.50.0/24 dev br0
BIND and DHCP
Ganeti hosts and virtual machines names are DNS entries. It would be possible to use a public
DNS to create them. A local DNS configuration is created localy instead, to
reduce the propagation delays, ensure it can always be reached and isolate the maintainance
of the name space into a private network. The top level domain farm was created
and divided into host.farm ( bound to 10.10.49.0/24 ) for all the ganeti hosts
and vm.farm ( bound to 10.10.50.0/24 ) for all the virtual machines.
/etc/bind/named.conf.options-
Do not listen or answer on public IPs.
allow-recursion { 127.0.0.1; 10.0.0.0/8; }; listen-on { 127.0.0.1; 10.10.0.254; }; /etc/bind/named.conf.local-
zone "10.10.in-addr.arpa" { type master; file "/etc/bind/db.10.10"; }; zone "farm" { type master; file "/etc/bind/db.farm"; }; /etc/bind/db.10.10-
; -*- mode: zone; -*- ; ; BIND reverse data file for broadcast zone ; $TTL 604800 @ IN SOA localhost. root.localhost. ( 2011031200 ; serial 604800 ; Refresh 86400 ; Retry 2419200 ; Expire 604800 ) ; Negative Cache TTL @ IN NS localhost. ;; ;; perl -n -e 'print if(s/(\S+)\s+IN\s+A\s+10\.10\.(\d+)\.(\d+)/\3.\2\tIN\tPTR\t\1.farm./)' < db.farm ;; 1.50 IN PTR rodo1.osm.vm.farm. /etc/bind/db.farm-
; -*- mode: zone; -*- ; $TTL 1h ; default Time-to-Live. defines the duration that the record may be cached (24h) $ORIGIN farm. @ IN SOA ns hostmaster ( 2011031200 ; serial 1h ; refresh - time when the slave will try to refresh the zone from the master (8h) 30m ; update retry - time between retries if the slave (secondary) (2h) ; fails to contact the master when refresh (above) has expired. 2w ; expiry - time slave will continue to supply authoritative data for the zone (1w) ; after the master died 30m) ; minimum - time a NAME ERROR = NXDOMAIN record is cached (24h) IN NS ns ns IN A 10.10.0.254 rodo1.osm.vm IN A 10.10.50.1 /etc/default/isc-dhcp-server-
INTERFACES="br0" /etc/dhcp/dhcpd.conf-
A MAC address is generated with the following oneliner:
MACADDR="52:54:$(dd if=/dev/urandom count=1 2>/dev/null | md5sum | sed 's/^\(..\)\(..\)\(..\)\(..\).*$/\1:\2:\3:\4/')"; echo $MACADDRand used to define the
rodo1.osm.vm.farmentry:group { option domain-name "farm"; option routers 10.10.0.254; option domain-name-servers 10.10.0.254; subnet 10.10.0.0 netmask 255.255.0.0 { host rodo1.osm.vm.farm { hardware ethernet 52:54:5a:8d:77:8e; fixed-address rodo1.osm.vm.farm; option subnet-mask 255.255.255.255; } } } - gcc20.host.farm IP is not 127.*
- gnt-cluster.host.farm IP is not bound to any interface
- IP 10.10.50.1
- Netmask 10.10.0.0
- DNS 10.10.0.254
- Gateway 10.10.0.254
A DHCP server is setup for each VM to use.
For both DHCP and the DNS, the configuration directories ( /etc/dhcp and /etc/bind respectively ) a git repository was created to keep an history of the changes.
Cluster and instance creation
The hostname of the machine is gcc20 which is bound to 127.0.1.1. This is not acceptable for ganeti2. When creating a cluster, the hostname is used to create the first node of the cluster and make it the master. For this operation to succeed, the hostname was temporarily changed to gcc20.host.farm and it's IP set in /etc/hosts to 10.10.49.20. This was permanently set on eth0 in the /etc/network/interface as follows:
up ip addr add 10.10.49.20/32 broadcast 10.10.49.255 dev eth0
The corresponding entries were added to the bind configuration:
gnt-cluster.host IN A 10.10.49.1 gcc20.host IN A 10.10.49.20
The prerquisite to run gnt-cluster init were met:
The cluster was created with:
gnt-cluster init --no-etc-hosts \
--enabled-hypervisors=kvm \
--nic-parameters link=br0 \
--master-netdev br0 \
--vg-name all gnt-cluster.host.farm
After downloading the installation CD of Debian GNU/Linux squeeze, the OpenStreetMap instance was created:
gnt-instance add -d -t plain -s 10G -B memory=512M,vcpus=1 \ -H kvm:boot_order=cdrom,cdrom_image_path=/srv/ganeti/debian-6.0.0-amd64-CD-1.iso,vnc_bind_address=0.0.0.0 \ -n gcc20.host.farm -o debootstrap+default --net 0:mac=52:54:5a:8d:77:8e rodo1.osm.vm.farm
The installation process was done using the VNC console
vncviewer gcc20.fsffrance.org:14050
Because the DHCP server delivers a 10.10.50.1/32 IP and a default route that is not on the same subnet ( 10.10.0.254 ) the DHCP client fails to setup the default route during the installation process. When the network initialization asks if the installation should proceed without a default route, the network was configured manually with the following:
After the installation completed, the disk was made the primary boot device:
gnt-instance modify -H boot_order=disk rodo1.osm.vm.farm
and the VM rebooted:
gnt-instance restart --shutdown-timeout=1 rodo1.osm.vm.farm
OpenStreeMap needs a lot of disk space and an additional 500GB disk was created for the virtual machine.
gnt-instance modify --disk add:size=500G rodo1.osm.vm.farm
From within the virtual machine it shows as /dev/vdb.