InstanceNotFound error after nova delete despite instance appearing in server list
0 reputation · 05 Jan 2022, 03:09 UTC
0 reputation · 05 Jan 2022, 03:09 UTC
After issuing a nova delete command, the instance remains visible in openstack server list output for a period governed by the oslo.cache expiration time. Subsequent attempts to show the instance with openstack server show return an InstanceNotFound error, creating a mismatch between the cached list view and the actual state.
The goal is to determine whether the observed stale‑cache behavior is an expected trade‑off for reduced database load or a configuration gap that should be addressed by tighter cache invalidation on delete operations.
What is the recommended cache expiration setting that balances consistency and performance? Should operators rely on manual cache flushes, or is automatic invalidation feasible without impacting Nova API latency?
35275 reputation · 05 Jan 2022, 08:14 UTC
The InstanceNotFound error occurring while an instance still appears in openstack server list is typically an expected behavior of the OpenStack asynchronous deletion lifecycle. When a delete command is issued, Nova marks the instance for deletion in the database and triggers a series of teardown tasks across the Nova Compute and Conductor services. The instance remains in the list because its database record exists until the final cleanup task completes.
The error occurs because openstack server show attempts to retrieve a full object detail that may no longer be available to the API's active lookup logic, or the record has transitioned to a state where it is logically "gone" but not yet physically purged from the database.
The oslo.cache expiration is a trade-off designed to protect the Nova database from excessive read pressure in large-scale deployments. In most production environments, relying on manual cache flushes is not recommended as it introduces operational overhead and does not address the underlying asynchronous nature of the Nova state machine.
Automatic invalidation is feasible, but the "stale" view is usually a symptom of the asynchronous deletion process rather than just a cache timeout. Even with a zero-second cache, the instance may appear in the list until the Compute node confirms the VM is destroyed and the Conductor removes the DB entry.
DELETING status as the source of truth. If the instance is visible but returns InstanceNotFound on a detail query, the deletion is already in progress.oslo.cache at default values unless you observe significant API latency. Reducing this too far can lead to database contention during peak load.openstack server show <instance_id>
If the command fails with InstanceNotFound but the list persists, monitor the nova-compute logs on the host where the VM resided to ensure the teardown is not hanging.To determine if this is a cache issue or a stuck deletion process, please provide the vm_state and vm_status from the Nova database for the affected UUID. If the state is deleted but the record persists, it is a database cleanup failure; if it is deleting, it is standard asynchronous behavior.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 05 Jan 2022, 05:23 UTC
When Nova deletes an instance, it immediately invalidates the per‑instance detail cache used by openstack server show, but the aggregated list cache (keyed by project, limit, marker, sort) is only refreshed when its TTL expires. This creates the window where the instance appears in openstack server list yet a detail request returns InstanceNotFound. Lowering the expiration_time for the [cache] region reduces that window, but note that the list cache is shared across all list queries, so a very low TTL increases database load proportionally to the list request rate.