Testing Ceph with ARMv8 OpenStack instances

The Ceph integration tests can be run on ARMv8 (aka arm64 or aarch64) OpenStack instances on CloudLab or Runabove.

When logged in CloudLab an OpenStack cluster suitable for teuthology must be created. To start an experiment

click Change Profile

to select the OpenStackTeuthology profile

the description of the profile contains an example credential file (i.e. openrc.sh) that can be copy/pasted on the local machine

the m400 default machine type will select ARMv8 hardware

in the last step, choose a name for the experiment. The openrc.sh file must be modified to reflect the chosen name because it shows in the URL of the authentication service. If a new experiment by the same name is run a month later, the same openrc.sh file can be used.

the page is then updated to show the progress of the provisionning. Note that it takes about 15 minutes for it to complete: even when the page says the experiment is up, the OpenStack setup is still going on and need a few more minutes.

Finally click on Profile instructions to display the link to the horizon dashboard and the password of the admin user (i.e. configuring OpenStack inside your experiment, you’ll be able to visit the OpenStack Dashboard WWW interface (approx. 5-15 minutes). Your OpenStack admin and instance VM password is randomly-generated by Cloudlab, and it is: 0905d783e7e7 .).

When the cluster is created, running the smoke integration tests for rados on jewel can be done with ceph-workbench

ceph-workbench --verbose ceph-qa-suite --ceph jewel --suite smoke --filter rados

assuming the openrc.sh file has been set in ~/.ceph-workbench/openrc.sh. When the command returns, it displays the URL of the web interface

...
2016-04-07 11:25:57,625.625 DEBUG:paramiko.transport:EOF in transport thread
2016-04-07 11:25:57,628.628 INFO:teuthology.openstack:
pulpito web interface: http://128.110.155.162:8081/
ssh access           : ssh ubuntu@128.110.155.162 # logs in /usr/share/nginx/html

And when the test completes successfully it will show in green. Otherwise the logs of the failed tests can be downloaded and analyzed.

CloudLab Eligibility

CloudLab is for academics and short term experiments for research purposes. Ceph developers are granted access and help maintain the profile that allows other CloudLab users to create and test Ceph clusters. If you are a Ceph developer, create an account on cloudlab and apply for membership in the Ceph project : the project name is Ceph.

Runabove Eligibility

Runabove is in Beta testing and only allows running two instances at this time.

CloudLab improvements

The default OpenStack profile for CloudLab was used as a base and modified to give more space to instances, use a 4.2 kernel instead of the default 3.13, create a larger flavor for the compilation of Ceph packages and pre-create an image with a name suitable for teuthology. The details have been discussed April 6th, 2016 during the Ceph Developers Monthly meeting (see the pad and the planning). The contributions that are not Teuthology specific were contributed via the mailing list.

Future work include being able to re-use standard images instead of using AMI/ARI/AKI that are custom made, using a 4.3 kernel which includes virtio-pci instead of virtio-mmio and allows to dynamically attach disks, create an experiment from the command line instead of the web interface (possibly via web scrapping).

For debug purposes, the following command line was used to chose a specific branch of ceph-qa-suite, teuthology, ceph and ceph-cm-ansible

teuthology-openstack --verbose \
    --teuthology-git-url http://github.com/dachary/teuthology \
    --teuthology-branch openstack \
    --ceph-qa-suite-git-url http://github.com/dachary/ceph-qa-suite \
    --suite-branch wip-archs \
    --key-filename ~/.ceph-workbench/teuthology-myself.pem \
    --key-name teuthology-myself \
    --ceph-git-url http://github.com/dachary/ceph \
    --ceph wip-arm64-jewel \
    --suite smoke \
    --filter rados ansible.yaml