Posted on

Two minor pitfalls when upgrading Havana stable

When upgrading an OpenStack compute or l3 agent node from 1:2013.2 to 1:2013.2.3 on Ubuntu precise 12.04.4:

  • The nova-compute version 1:2013.2 is expected to fail with 
    /var/log/nova/nova-compute.log
IncompatibleObjectVersion: Version 1.9 of Instance is not supported

    when interfaced with a 1:2013.2.3. It will not disrupt the running instances but will prevent operations on them until the upgrade is complete.

  • neutron-l3-agent will fail with:

    /var/log/neutron/metadata-agent.log
AttributeError: 'HTTPClient' object has no attribute 'auth_tenant_id'

    because the python-neutronclient package must also be upgraded . It only happens if upgrading with apt-get install neutron-common but will be fine if upgrading with apt-get dist-upgrade.

Posted on

Reseting an instance {power,vm,task}_state in Havana

Sometime, after an hypervisor crash or nova-compute error, an OpenStack instance can be left in a state that cannot be conveniently fixed with nova reset-state.

$ nova list
+--------------------------------------+----------------+---------+...
| ID                                   | Name           | Status  |
+--------------------------------------+----------------+---------+...
| ca9496e9-0bd2-4734-9cf9-eb4e264628f7 | www            | SHUTOFF |
+--------------------------------------+----------------+---------+...
...  -------------+-------------+----------------------------------+
      Task State  | Power State | Networks                         |
...  -------------+-------------+----------------------------------+
      powering-on | Shutdown    | fsf-lan=10.0.3.18, 93.20.168.177 |
...  -------------+-------------+----------------------------------+

Setting the fields for the instance directly in the database will allow operations on the instance (nova start or nova volume-detach for instance):

$ mysql -e "update instances set task_state = NULL, \
           vm_state = 'stopped', \
           power_state = 4 \
           where deleted = 0 and hostname = 'www' and \
           uuid = 'ca9496e9-0bd2-4734-9cf9-eb4e264628f7'" nova

Using the uuid is necessary to avoid modifying an unrelated instance with the same name. This should be done only after verifying that the instance does not exist on the hypervisor with:

$ ps -fauwwwx | grep ca9496e9-0bd2-4734-9cf9-eb4e264628f7