Category: ceph

The Ceph command line and ceph-disk helper are python scripts for which there are integration tests (ceph-disk.sh and test.sh). It would be useful to add unit tests and pep8 checks.
It can be done by creating a python module instead of an isolated file (see for instance ceph-detect-init) with a tox.ini file including pep8, python2 and python3 test environments.
Since Ceph relies on autotools, the setup.py can be used with -local targets. For instance:

all-local::
        python setup.py build
clean-local::
	python setup.py clean
install-data-local::
	python setup.py install --root=$(DESTDIR) --install-layout=deb

Note the double : meaning it appends to an existing rule instead of overriding it. The –root=$(DESTDIR) will install the module files in the appropriate directory when building packages.
tox uses pip to fetch dependencies required to run tests from PyPI, but tests sometime run without network access. The depedencies can be collected by wheel with something like:

pip wheel -r requirements.txt

It will create a wheelhouse directory which can later be used with

pip install --no-index --use-wheel --find-links=wheelhouse \
  -r requirements.txt

Continue reading “Improving Ceph python scripts tests”

When running tests from the Ceph sources, the disk is used intensively and a ram disk can be used to reduce the latency. The kernel must be rebooted to set the ramdisk maximum size to 16GB. For instance on Ubuntu 14.04 in /etc/default/grub (the module name which could be rb or brd depending).

GRUB_CMDLINE_LINUX="brd.rd_size=16777216" # 16GB in KB

the grub configuration must then be updated with

sudo update-grub

After reboot the ram disk is formatted as an ext4 file system and mounted:

$ cat /sys/module/brd/parameters/rd_size
16777216
$ sudo mkfs -t ext4 /dev/ram1
$ sudo mount /dev/ram1 /srv
$ df -h /srv
Filesystem      Size  Used Avail Use% Mounted on
/dev/ram1        16G   44M   15G   1% /srv
$ free -g
             total       used       free     shared    buffers     cached
Mem:            31          0         31          0          0          0
-/+ buffers/cache:          0         31

Cloning ceph, compiling and running tests should now take less than 15 minutes with

$ git clone https://github.com/ceph/ceph
$ cd ceph
$ ./run-make-check.sh

When the ram disk is umounted, some of the memory used by the ram disk is still in use

$ free -g
             total       used       free     shared    buffers     cached
Mem:            31         27          4          0          0         17
-/+ buffers/cache:          9         22
$ sudo umount /srv
$ free -g
             total       used       free     shared    buffers     cached
Mem:            31         18         13          0          0          8
-/+ buffers/cache:          9         22

It can be flushed with

$ sudo blockdev --flushbufs /dev/ram1
$ free -g
             total       used       free     shared    buffers     cached
Mem:            31          9         22          0          0          8
-/+ buffers/cache:          0         31

Continue reading “Ceph make check in a ram disk”

When a teuthology target (i.e. machine) is provisioned with teuthology-lock for the purpose of testing Ceph, there is no way to choose the kernel. But it can be installed afterwards using the following:

cat > kernel.yaml <<EOF
interactive-on-error: true
roles:
- - mon.a
  - client.0
kernel:
   branch: testing
tasks:
- interactive:
EOF

Assuming the target on which the new kernel is to be installed is vpm083, running

$ teuthology  --owner loic@dachary.org \
  kernel.yaml <(teuthology-lock --list-targets vpm083)
...
2015-03-09 17:47 INFO:teuthology.task.internal:Starting timer...
2015-03-09 17:47 INFO:teuthology.run_tasks:Running task interactive...
Ceph test interactive mode, use ctx to interact with the cluster
>>>

will install an alternate kernel and reboot the machine:

[ubuntu@vpm083 ~]$ uname -a
Linux vpm083 3.19.0-ceph-00029-gaf5b96e #1 SMP Thu Mar 5 01:04:25 GNU/Linux
[ubuntu@vpm083 ~]$ lsb_release -a
LSB Version:	:base-4.0-amd64:base-4.0-noarch:
Distributor ID:	RedHatEnterpriseServer
Description:  release 6.5 (Santiago)
Release:	6.5
Codename:	Santiago

Command line arguments to the kernel may be added to /boot/grub/grub.conf. For instance loop.max_part=16 to allow partition creation on /dev/loop devices:

default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title rhel-6.5-cloudinit (3.19.0-ceph-00029-gaf5b96e)
        root (hd0,0)
        kernel /boot/vmlinuz-3.19.0 ro root=LABEL=79d3d2d4  loop.max_part=16
        initrd /boot/initramfs-3.19.0.img

When handling a Ceph OSD, it is convenient to assign it a symbolic name that can be chosen even before it is created. That’s what the uuid argument for ceph osd create is for. Without a uuid argument, a random uuid will be assigned to the OSD and can be used later. Since the ceph osd create uuid is idempotent, it can also be used to lookup the id of a given OSD.

$ osd_uuid=b2e780fc-ec82-4a91-a29d-20cd9159e5f6
# convert the OSD uuid into an OSD id
$ ceph osd create $osd_uuid
0
# convert the OSD id into an OSD uuid
$ ./ceph --format json osd dump | jq '.osds[] | select(.osd==0) | .uuid'
"b2e780fc-ec82-4a91-a29d-20cd9159e5f6"

The Ceph integration tests may fail because of environmental problems (network not available, packages not built, etc.). If six jobs failed out of seventy, these failed test can be re-run instead of re-scheduling the whole suite. It can be done using the **–filter** option of teuthology-suite with a comma separated list of the job description that failed.
The job description can either be copy/pasted from the web interface or extracted from the paddles json output with:

$ run=loic-2015-03-03_12:46:38-rgw-firefly-backports---basic-multi
$ paddles=http://paddles.front.sepia.ceph.com
$ eval filter=$(curl --silent $paddles/runs/$run/jobs/?status=fail |
  jq '.[].description' | \
  while read description ; do echo -n $description, ; done | \
  sed -e 's/,$//')

Where the paddles URL outputs a json description of each job of the form:

[
  {
    "os_type": "ubuntu",
    "nuke_on_error": true,
    "status": "pass",
    "failure_reason": null,
    "success": true,
...
    "description": "rgw/multifs/{clusters/fixed-2.yaml}"
  },
  {
    "os_type": "ubuntu",
...

The jobs/?status=fail part of the URL selects the jobs with “success”:false. The jq expression displays the description field (.[].description), one by line. These lines are aggregated into a comma separated list (while read description ; do echo -n $description, ; done) and the trailing comma is stripped (sed -e ‘s/,$//’). The filter variable is set to the resulting line and evaled to get rid of the quotes (eval filter=$(..)).
The command used to schedule the entire suite can be re-used by adding the –filter=”$filter” argument and will only run the failed jobs.

$ ./virtualenv/bin/teuthology-suite --filter="$filter" \
  --priority 101 --suite rgw --suite-branch firefly \
  --machine-type plana,burnupi,mira \
  --distro ubuntu --email loic@dachary.org \
  --owner loic@dachary.org  \
  --ceph firefly-backports
...
Suite rgw in suites/rgw scheduled 6 jobs.
Suite rgw in suites/rgw -- 56 jobs were filtered out.

When a Ceph teuthology integration test fails (for instance a rados jobs), it will collect core dumps which can be downloaded from the same directory where the logs and config.yaml files can be found, under the remote/mira076/coredump directory.
The binary from which the core dump comes from can be displayed with:

$ file 1425077911.7304.core
ELF 64-bit LSB  core file x86-64, version 1, from 'ceph-osd -f -i 3'

The teuthology logs contains command lines that can be used to install the corresponding binaries:

$ echo deb http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic\
/sha1/e54834bfac3c38562987730b317cb1944a96005b trusty main | \
  sudo tee /etc/apt/sources.list.d/ceph.list
$ sudo apt-get update
$ sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes \
  -o Dpkg::Options::="--force-confdef" \
  -o Dpkg::Options::="--force-confold" install \
  ceph=0.80.8-75-ge54834b-1trusty \
  ceph-dbg=0.80.8-75-ge54834b-1trusty

The ceph-dbg package contains debug symbols that will automatically be used by gdb(1):

$ gdb /usr/bin/ceph-osd 1425077911.7304.core
...
Reading symbols from /usr/bin/ceph-osd...
Reading symbols from /usr/lib/debug//usr/bin/ceph-osd...done.
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `ceph-osd -f -i 3'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f59d6e9af07 in _dl_map_object_deps at dl-deps.c:528
(gdb) bt
#0  0x00007f59d6e9af07 in _dl_map_object_deps at dl-deps.c:528
#1  0x00007f59d6ea1aab in dl_open_worker at dl-open.c:272
#2  0x00007f59d6e9cff4 in _dl_catch_error  at dl-error.c:187
#3  0x00007f59d6ea13bb in _dl_open    at dl-addr.c:61
#5  __GI__dl_addr at dl-addr.c:137
#6  0x00007f59c06dcbc0 in ?? ()
#7  0x00007f59d70b11c8 in _r_debug ()
#8  0x00007f59c06dcba0 in ?? ()
#9  0x00007f59c06dcbb0 in ?? ()
#10 0x00007f59c06dcb90 in ?? ()
#11 0x00007f59c06dca94 in ?? ()
#12 0x0000000000000000 in ?? ()