HOWTO debug a teuthology task

To debug a modification to a ceph-qa-suite task ( for instance, a teuthology target is locked with:

$ ./virtualenv/bin/teuthology-lock --lock-many 1 --owner
$ ./virtualenv/bin/teuthology-lock --list-targets --owner > targets.yaml

and used to run the test with:

./virtualenv/bin/teuthology \
  --suite-path $HOME/software/ceph/ceph-qa-suite \
  --owner \
  $HOME/software/ceph/ceph-qa-suite/suites/rados/basic/tasks/repair_test.yaml \

where roles.yaml sets all roles to one target:

- [mon.0, osd.0, osd.1, osd.2, osd.3, osd.4, client.0]

Each run requires the installation and deinstallation of all Ceph packages and takes minutes. The installation part of repair_test.yaml can be commented out and the packages installed manually.

$ cat repair.yaml
#- install:
- ceph:
- repair_test:

The existing packages are removed:

sudo apt-get remove --purge python-ceph ceph-common librados2 librbd1 ceph-fuse libcephfs1-dbg libcephfs-java libcephfs1 libcephfs-jni ceph-fs-common

The repository is set to the desired branch (for instance wip-10018-primary-erasure-code-hinfo), as compiled by gitbuilder.cgi with:

$ echo deb trusty main | sudo tee /etc/apt/sources.list.d/ceph.list

The packages are installed:

$ sudo apt-get update
$ sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph ceph-dbg ceph-mds ceph-mds-dbg ceph-common ceph-common-dbg ceph-fuse ceph-fuse-dbg ceph-test ceph-test-dbg radosgw radosgw-dbg python-ceph libcephfs1 libcephfs1-dbg libcephfs-java librados2 librados2-dbg librbd1 librbd1-dbg

and the repositories cleaned (this must be done again after each test completes):

$ rm -fr cephtest/ && sudo rm -fr /var/lib/ceph && sudo rm -fr /var/log/ceph/*

Utilities must be copied from the teuthology sources with (assuming vpm178 is the target):

$ scp teuthology/task/daemon-helper teuthology/task/adjust-ulimits ubuntu@vpm178:/tmp
$ ssh ubuntu@vpm178 sudo cp -a /tmp/{daemon-helper,adjust-ulimits} /usr/bin