A kvm process running an OpenStack instance with a volume attached to it becomes unresponsive. After manually investigating the cause of the problem, the kvm process is killed. The iscsi sessions is acknowledged to be lost and the disk is re-attached. The instance fails to reboot and the root file system is repaired from the compute node:
# qemu-nbd --port 20000 /var/lib/nova/instances/instance-00000103/disk & # nbd-client localhost 20000 /dev/nbd0 Negotiation: ..size = 10240MB bs=1024, sz=10737418240 bytes # mount /dev/nbd0p1 /mnt ... fix things ... # umount /mnt # nbd-client -d /dev/nbd0
Recovering attached volumes
If the kvm process is killed, the nova database is in an inconsistent state and needs to be repaired. The volumes show as being in-use:
# nova volume-list +----+-----------+--------------+------+-------------+--------------------------------------+ | ID | Status | Display Name | Size | Volume Type | Attached to | +----+-----------+--------------+------+-------------+--------------------------------------+ | 38 | in-use | None | 500 | None | 83dc1b22-e0fd-4fc3-836b-68490b2d7a91 | +----+-----------+--------------+------+-------------+--------------------------------------+
The corresponding database record shows the machine on which it is located:
# mysql -e "select host,provider_location from volumes where id = 38" nova +---------------+------------------------------------------------------------------+ | host | provider_location | +---------------+------------------------------------------------------------------+ | bm0005.the.re | 192.168.100.5:3260,1 iqn.2010-10.org.openstack:volume-00000026 0 | +---------------+------------------------------------------------------------------+
The provider_location field shows the name of the volume which can be used to check that no such iscsi session exists on the bm0005.the.re host:
# iscsiadm -m session | grep volume-00000026 #
The volume state is reset in the database with:
mysql -e "update volumes set status = 'available', \ attach_status = 'detached', \ mountpoint = NULL where id = 38" nova
The volume can now be attached with:
# nova volume-attach 83dc1b22-e0fd-4fc3-836b-68490b2d7a91 38 /dev/vdc
Should this operation fail, the source of the problem can be found on the controller node providing API in the file /var/log/nova/nova-compute.log
Fixing a boot failure
If the instance fails to boot and does not display anything on the VNC console, it may be possible to fix the problem by editing the file system. Assuming the name of the failed instance is razor, the OpenStack node on which it runs is found with:
mysql -e "select id,host,hostname from instances where deleted = 0 and hostname ='razor'" nova +-----+---------------+----------+ | id | host | hostname | +-----+---------------+----------+ | 103 | bm0005.the.re | razor | +-----+---------------+----------+
On the host bm0005.the.re the disk can be found with the id field (i.e. 103 in the example above) as follows:
# ls -hl /var/lib/nova/instances/instance-000000103/disk -rw-r--r-- 1 libvirt-qemu kvm 1.2G Oct 26 16:11 /var/lib/nova/instances/instance-000000103/disk
The disk can be exposed thru a network block device as follows:
# qemu-nbd --port 20000 /var/lib/nova/instances/instance-00000103/disk &
It can then be bound to the /dev/nbd0 block device as follows:
# apt-get install nbd-client # nbd-client localhost 20000 /dev/nbd0 Negotiation: ..size = 10240MB bs=1024, sz=10737418240 bytes
Mounting the root partition to fix it, either with chroot or by modifying a file directly can be done with:
# mount /dev/nbd0p1 /mnt ... fix things ...
When all is as expected, the file system must be unmounted and the network block device de-associated with
# umount /mnt # nbd-client -d /dev/nbd0