Debian GNU/Linux wheezy mirror for OpenStack

Deploying OpenStack on Debian GNU/Linux requires the wheezy repository, which not stable yet at this date. When the HOWTO is tested to work against the current state of the wheezy repository, it is worth keeping a copy to ensure it can be re-run reliably. A virtual machine is dedicated to store a wheezy mirror, updated with the ftpsync shell script:

# ssh mirror@debian.the.re
$ bin/ftpsync sync:all

It can then be used as follows:

# cat > /etc/apt/sources.list.d/debian.the.re.list <<EOF
deb http://debian.the.re/debian wheezy main
deb-src http://debian.the.re/debian wheezy main
EOF
# apt-get update

preparing the Debian GNU/Linux virtual machine

The machine to host the mirror is based on turnkey core.

# nova flavor-list
| 6  | e.1-cpu.10GB-disk.1GB-ram   | 1024      | 10   | 0         |      | 1     | 1.0         |
# nova boot --image c620dd0c-2313-43ae-8e87-47d1457a3cec   --flavor 6 \
  --key_name loic  --availability_zone=bm0001 debian --poll

Based on the size estimate updated daily, a 300GB volume is created

euca-create-volume --zone bm0001 --size 300

to accomodate the following mirrors:

• source 52GB
• all 60GB
• amd64 54GB

The volume is attached to the virtual machine

# nova volume-attach d02487ad-da2d-442d-a02f-10fbea06c4ce 34 /dev/vdb
# nova volume-list
+----+--------+--------------+------+-------------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type |             Attached to              |
+----+--------+--------------+------+-------------+--------------------------------------+
| 34 | in-use | None         | 300  | None        | d02487ad-da2d-442d-a02f-10fbea06c4ce |
+----+--------+--------------+------+-------------+--------------------------------------+

The line

10.145.0.8      debian.the

is added in the OpenStack puppet master file /etc/puppet/files/hosts and broadcasted to all bare metal. A reverse proxy configuration is also broadcasted to all bare metal machines. The IP of debian.the.re will always be routed to a machine with a reverse proxy. The bare metal machine hosting the virtual machine must also be the one to which the IP is routed.
The disk is formatted, within the virtual machine with:

mkfs.ext4 /dev/vdb

and declared in /etc/fstab

# mkdir /media/debian
# echo '/dev/vdb  /media/debian auto defaults  0  0' >> /etc/fstab
# mount /media/debian
# df -h
/dev/vdb              296G  191M  281G   1% /media/debian

mirroring Debian GNU/Linux

The instructions to mirror a Debian GNU/Linux are run. The mirror user is created and is made the owner of the /media/debian file system. The mirror scripts are extracted in its home directory as follows:

git clone https://ftp-master.debian.org/git/archvsync.git /home/mirror

The ~/etc/ftpsync.conf file is copied from the ~/etc/ftpsync.conf.sample template and adapted with the following values:

TO="/media/debian/"
RSYNC_HOST=ftp.fr.debian.org

The mirror closest to the virtual machine is selected ( ftp.fr.debian.org ).

France
Site: ftp.fr.debian.org
Type: Push-Primary

ARCH_EXCLUDE="alpha arm armel armhf hppa hurd-i386 i386 ia64 \
 kfreebsd-amd64 kfreebsd-i386 m68k mipsel mips powerpc s390 s390x sparc"

Only three architecture are mirrored : source, all and amd64 : this is done by excluding all other architectures with the ARCH_EXCLUDE variable.

EXCLUDE="--exclude=pool/contrib --exclude=pool/non-free"

The contrib and non-free pools are also excluded with the EXCLUDE variable.

The mirror is run with

 bin/ftpsync sync:all

and its progress can be followed with:

 tail -f ~/log/*.log

exposing the repository

To reduce the memory footprint and because only static files are being served, nginx is used.

# apt-get install nginx
# ln -s /media /var/www
# edit /etc/nginx/nginx.conf to add an option for human browsing
autoindex on

updating from the repository

The repository Release file contains an expiration date, for instance:

Valid-Until: Thu, 07 Oct 2010 08:17:56 UTC

Because of that, an apt-get upate will fail after the expiration date and the following should be used to bypass this check instead:

apt-get -o Acquire::Check-Valid-Until=false update

Alternatively, the sources.list(5) file can be modified to add the trusted flag that bypasses the expiration date.

deb [trusted=1] http://debian.the.re/debian wheezy main