Disaster recovery with kvm in OpenStack

A kvm process running an OpenStack instance with a volume attached to it becomes unresponsive. After manually investigating the cause of the problem, the kvm process is killed. The iscsi sessions is acknowledged to be lost and the disk is re-attached. The instance fails to reboot and the root file system is repaired from the compute node:

# qemu-nbd --port 20000 /var/lib/nova/instances/instance-00000103/disk &
# nbd-client localhost 20000 /dev/nbd0
Negotiation: ..size = 10240MB
bs=1024, sz=10737418240 bytes
# mount /dev/nbd0p1 /mnt
... fix things ...
# umount /mnt
# nbd-client -d /dev/nbd0

Continue reading “Disaster recovery with kvm in OpenStack”