Lowering Ceph scrub I/O priority

Note: the following does not currently work in Firefly because of http://tracker.ceph.com/issues/9677 . It has been backported to Firefly and will likely be in 0.80.8.

The disk I/O of a Ceph OSD thread scrubbing is the same as all other threads by default. It can be lowered with ioprio options for all OSDs with:

ceph tell osd.* injectargs '--osd_disk_thread_ioprio_priority 7'
ceph tell osd.* injectargs '--osd_disk_thread_ioprio_class idle'

All other threads in the OSD will be be (best effort) with priority 4 which is the default for daemons. The disk thread will show as idle:

$ sudo iotop --batch --iter 1 | grep 'ceph-osd -i 0' | grep -v be/4
 4156 idle loic        0.00 B/s    0.00 B/s  0.00 %  0.00 % ./ceph-osd -i 0 ..

The change will only be effective if the scheduler is cfq (it can safely be modified at runtime)

# cat /sys/block/sda/queue/scheduler
noop [deadline] cfq
# echo cfq > /sys/block/sda/queue/scheduler

To display the current value for a given OSD the following can be run from the host on which it is running (because it uses the asok file found in /var/run/ceph):

$ ceph daemon osd.0 config get osd_disk_thread_ioprio_class
{ "osd_disk_thread_ioprio_class": "idle"}
$ ceph daemon osd.0 config get osd_disk_thread_ioprio_priority
{ "osd_disk_thread_ioprio_priority": "7"}