HOWTO debug a teuthology task

To debug a modification to a ceph-qa-suite task ( for instance, a teuthology target is locked with:

$ ./virtualenv/bin/teuthology-lock --lock-many 1 --owner
$ ./virtualenv/bin/teuthology-lock --list-targets --owner > targets.yaml

and used to run the test with:

./virtualenv/bin/teuthology \
  --suite-path $HOME/software/ceph/ceph-qa-suite \
  --owner \
  $HOME/software/ceph/ceph-qa-suite/suites/rados/basic/tasks/repair_test.yaml \

where roles.yaml sets all roles to one target:

- [mon.0, osd.0, osd.1, osd.2, osd.3, osd.4, client.0]

Each run requires the installation and deinstallation of all Ceph packages and takes minutes. The installation part of repair_test.yaml can be commented out and the packages installed manually.

$ cat repair.yaml
#- install:
- ceph:
- repair_test:

Continue reading “HOWTO debug a teuthology task”

Gitlab workbench

Gitlab is installed on using docker images. redis is installed first, as an independant container:

docker pull sameersbn/redis:latest
docker run --name=redis -d sameersbn/redis:latest

then MySQL

docker pull sameersbn/mysql:latest
docker run --name=mysql -d \
  -e 'DB_NAME=gitlabhq_production' \
  -e 'DB_USER=gitlab' \
  -v /opt/mysql/data:/var/lib/mysql \

and finally gitlab

docker pull sameersbn/gitlab:latest
docker run --name='gitlab' -it -d  \
  --link mysql:mysql --link redis:redisio \
  -e ''  \
  -e 'SMTP_ENABLED=true' \
  -e '' \
  -e 'SMTP_USER=' \
  -e 'SMTP_HOST=' \
  -e 'SMTP_PORT=25' \
  -e 'SMTP_STARTTLS=false' \
  -e 'GITLAB_SIGNUP=true' \
  -e 'GITLAB_PORT=80' \
  -e '' \
  -e 'OAUTH_ALLOW_SSO=true' \
  -e 'OAUTH_GITHUB_API_KEY=github Client ID'  \
  -e 'OAUTH_GITHUB_APP_SECRET=github Client Secret' \
  -e 'GITLAB_SSH_PORT=22' \
  -p 22:22 -p 80:80 \
  -v /var/run/docker.sock:/run/docker.sock \
  -v /opt/gitlab/data:/home/git/data \
  -v $(which docker):/bin/docker \

The ssh server of the server will need to bind another port by editing /etc/ssh/sshd_config, changing the Port value and restarting the server with stop ssh ; start ssh.
The OmniAuth single sign on is configured following gitlab instructions, except for editing the config.yml file: the OAUTH_GITHUB_* are set instead, using information found in the applications settings github page.
It uses the automagic dockerlinks to connect it to the MySQL and redis servers (–link mysql:mysql –link redis:redisio). The SMTP server is configured using variables from the documentation to point to the server running on the host ( is the IP of the docker0 bridge on which all containers are connected and in the same IP range as the dynamic IP they are given). A postfix server is installed on the host:

$ sudo apt-get install postfix
... chose internet server ...

and it is configured to accept to relay mails from any docker contain in the IP range:

$ cat /etc/postfix/
myhostname =
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination =, localhost, localhost.localdomain, localhost
relayhost =
mynetworks = [::ffff:]/104 [::1]/128

A working SMTP server is required to allow sign up as required with GITLAB_SIGNUP=true. The gitlab persistent data is in /opt/mysql/data (bind mounted with -v /opt/mysql/data:/var/lib/mysql) for the MySQL database and /opt/gitlab/data (bind mounted with -v /opt/gitlab/data:/home/git/data) for repositories, gitlab assets etc. When the host reboots, the containers can be restarted as above, they only contains non persistent information.

Teuthology docker targets hack (2/5)

The teuthology container hack is improved to snapshot the container after Ceph and its dependencies have been installed. It helps quickly testing ceph-qa-suite tasks. A job doing nothing but install the Firefly version of Ceph takes 14 seconds after the initial installation (which can take between 5 to 15 minutes depending on how fast is the machine and how much bandwidth is available).

2014-11-17 01:21:00,067.067 INFO:teuthology.worker:Reserved job 42
2014-11-17 01:21:00,067.067 INFO:teuthology.worker:Config is:
machine_type: container
name: foo
os_type: ubuntu
os_version: '14.04'
    ceph: {branch: firefly}
priority: 1000
- [mon.a, osd.0, osd.1, client.0]
- {install: null}
tube: container
verbose: false

Fetching from upstream into /home/loic/src/ceph-qa-suite_master
completed on container001: sudo lsb_release '-is':  Ubuntu
reusing existing image ceph-base-ubuntu-14.04-firefly
running 'docker' 'stop' 'container001'
completed ('docker', 'stop', u'container001') on container001:  container001
2014-11-17 01:21:31,677.677 data:
{duration: 14, flavor: basic, success: true}
2014-11-17 01:21:31,677.677

Continue reading “Teuthology docker targets hack (2/5)”

Running make check on Ceph pull requests

Each Ceph contribution is expected to successfully run make check and pass all the unit tests it contains. The developer runs make check locally before submitting his changes but the result may be influenced by the development environment. A draft bot is proposed to watch the list of pull requests on a github repository and run a script based on each time a new patch is uploaded. --user loic-bot --password XXXXX \
   --owner ceph --repository ceph \
   --script $HOME/makecheck/

If the script fails, it adds a comment with the output of the run to the pull request. Otherwise it reports success in the same way.
Continue reading “Running make check on Ceph pull requests”