Diagnosing OpenStack Nova Instance Launch Failures: Spawning/Error States

Recognizable condition

After issuing openstack server create (or legacy nova boot), the instance remains stuck in spawning or transitions to error. Console logs often show a timeout while waiting for VIF plugging or messages that the compute node hypervisor is unavailable.

Cause/diagnostic table

Symptom area	Likely cause	What to look for
Network config timeout	Neutron DHCP or metadata agent down	Agent status shows "xxx" (down) or logs report AMQP disconnect
Libvirt/Qemu limits	Process limits hit or permission errors	Libvirtd logs contain "operation not permitted" or "qemu-kvm: failed to allocate memory"
Placement service reports no resources	Missing or stale resource provider inventories	openstack resource provider list returns empty or shows VCPU = 0
Scheduler finds no valid hosts	Overly restrictive nova.conf filters or disabled compute service	Nova‑scheduler log lines like "No valid host found" and filters such as "RetryFilter" or "AggregateInstanceExtraSpecsFilter" returning false

Ordered checks

1. Verify nova‑compute service on the affected host
   • Run on the controller or any node with openstack CLI: openstack compute service list --host compute-host
   • Expected: State = "up", Status = "enabled". If "down" or "disabled", proceed to step 1‑fix.
   • Risk: Restarting nova‑compute may briefly affect instances already running on that host.
2. Inspect libvirtd logs for permission or device errors
   • On the compute host: journalctl -u libvirtd -f or tail -f /var/log/libvirt/libvirtd.log
   • Look for lines containing "permission denied", "failed to start domain", or "qemu-kvm: cannot set up guest memory".
   • If found, check /etc/libvirt/qemu.conf for correct user and group (usually qemu) and ensure the cgroup_device_acl includes needed devices.
   • Risk: Restarting libvirtd stops all VMs on the host; schedule during a maintenance window or evacuate instances first.
3. Check Neutron agent status and connectivity
   • Run: openstack network agent list
   • Expected: DHCP and metadata agents show " :-)" (alive) state; L3 agent (if used) also alive.
   • If any agent is down, examine its log: /var/log/neutron/dhcp-agent.log, /var/log/neutron/metadata-agent.log.
   • Verify AMQP connectivity: sudo rabbitmqctl list_connections (replace with your MQ broker).
   • Risk: Restarting agents in a production controller can cause brief network disruption; ensure HA setup or perform during low traffic.
4. Query Placement API for resource provider inventories
   • Run: openstack resource provider list --sort-column name
   • For each provider, check: openstack resource provider show uuid and look at inventories (VCPU, MEMORY_MB, DISK_GB).
   • If inventories are zero or missing, the provider may not have been synced.
   • Risk: Do not edit the placement database directly; use CLI commands only.
5. Review nova‑scheduler logs for filter outcomes
   • On the controller: grep -i "No valid host" /var/log/nova/nova-scheduler.log
   • Look for lines indicating which filter returned False (e.g., "RamFilter", "AggregateInstanceExtraSpecsFilter").
   • If a filter is overly restrictive, note its name for adjustment.
   • Risk: Changing scheduler filters can affect placement decisions across the cloud; test in a non‑production cell first.
6. Confirm compute node hypervisor is enabled and not disabled
   • Run: openstack compute service list --host compute-host --service nova-compute
   • Ensure the service is not marked "disabled" (or manually disabled via nova service-disable).
   • If disabled, re‑enable with nova service-enable host nova-compute.

Fixes tied to findings

• Nova‑compute down: sudo systemctl restart openstack-nova-compute (or the appropriate service name). After restart, re‑run openstack compute service list to confirm "up".
• Libvirt permission/device errors:
  • Correct /etc/libvirt/qemu.conf: set user = "qemu", group = "qemu", and ensure needed devices are in cgroup_device_acl.
  • Restart libvirtd: sudo systemctl restart libvirtd.
  • Verify with systemctl status libvirtd (should be active).
• Neutron agents down:
  • Restart the specific agent: sudo systemctl restart neutron-dhcp-agent (or metadata/l3).
  • Check MQ connectivity: sudo rabbitmqctl check_permissions -p / openstack (adjust vhost as needed).
  • Re‑run openstack network agent list to confirm " :-)".
• Placement shows no resources:
  • Restart placement service: sudo systemctl restart devstack@placement-api (or the service name used in your deployment).
  • Sync the database: openstack-status placement sync or placement-manage db sync depending on version.
  • Re‑query providers to confirm inventories are populated.
• Scheduler filters yield no hosts:
  • Identify the restrictive filter from scheduler logs.
  • Edit /etc/nova/nova.conf on the controller: remove or adjust the filter in the [filter_scheduler] section (e.g., comment out AggregateInstanceExtraSpecsFilter if not needed).
  • Restart nova‑scheduler: sudo systemctl restart openstack-nova-scheduler.
  • Retry instance launch.

Escalation criteria

• If after completing the full check‑and‑fix cycle the instance remains in error or spawning, repeat the cycle once more.
• Escalate to the next support tier (or open a bug) when:
  • The instance fails after two complete cycles, or
  • Multiple compute nodes exhibit the same failure pattern simultaneously.

Verification

After applying a fix:

1. Run openstack server list and confirm the target instance shows ACTIVE.
2. Ping the instance’s assigned IP (retrieved via openstack server show server-id -c addresses -f value) to verify connectivity.
3. Re‑check the specific component that was fixed (e.g., openstack compute service list for nova‑compute, openstack network agent list for Neutron agents, openstack resource provider list for Placement).
4. Review the relevant log files for absence of the earlier error messages.

Limitations and practical checks

This guide covers the most common failure modes observed in typical OpenStack deployments using the Nova, Neutron, and Placement services. It does not address:

• Storage backend issues (Cinder volume attachment failures).
• Hypervisor‑specific problems beyond libvirt/qemu (e.g., KVM module missing).
• Cell‑v2 routing failures or database replication lag.

To ensure the guide remains applicable, verify the versions of the core services (e.g., Nova ≥ 23.0.0, Neutron ≥ 20.0.0, Placement ≥ 23.0.0) and adjust log paths or service names if your distribution uses different naming conventions.