Diagnosing OpenStack Nova Live Migration Failures: A Step-by-Step Guide for Cloud Operators

Recognizable Failure Symptoms

Live migration in OpenStack Nova fails in predictable ways. The instance may hang at "preparing" or "migrating" state, the migration record shows error status with a pre-connection or cutover failure message, or the instance pauses indefinitely during the final cutover phase. In Horizon or openstack server migration list, you'll see the migration stuck with status: error and dest_compute: populated but the instance still running on the source.

These symptoms typically appear when migrating instances with local ephemeral disks (block migration) or when source and destination compute hosts have configuration drift. The root cause is almost always one of: CPU model mismatch, missing SSH trust between nova users, incompatible libvirt/QEMU versions, or storage/network misconfiguration.

Quick Cause-to-Symptom Reference

Ordered Diagnostic Checks

1. Verify Compute Host Readiness

Run these on the controller or any host with admin credentials and openstack client configured:

openstack compute service list --service nova-compute
openstack hypervisor list --long
openstack hypervisor stats show

Confirm both source and destination hosts show enabled: true, state: up, and identical cpu_info architecture (e.g., both x86_64 with same vendor). Check free_ram_mb and free_disk_gb on destination exceed instance requirements plus 10% overhead.

2. Inspect Nova-Compute Logs Immediately After Trigger

On both source and destination compute nodes, tail the logs while initiating migration:

# On source compute host
tail -f /var/log/nova/nova-compute.log | grep -i migration

# On destination compute host
tail -f /var/log/nova/nova-compute.log | grep -i migration

Look for MigrationPreCopyError, LiveMigrationFailure, or libvirt connection errors. Note the timestamp and correlate with libvirt logs.

3. Check Libvirt Daemon and SSH Trust

On each compute host, verify libvirtd is running and listening on the migration port (default 16514 for TLS, 16509 for TCP):

systemctl status libvirtd
ss -ltnp | grep -E '16509|16514'
cat /etc/libvirt/libvirtd.conf | grep -E 'listen_tls|listen_tcp|auth_tls|auth_tcp'

Test passwordless SSH for the nova user between hosts:

# Run as root on source, then destination
su -s /bin/bash -c "ssh -o BatchMode=yes -o ConnectTimeout=5 nova@ true" nova
su -s /bin/bash -c "ssh -o BatchMode=yes -o ConnectTimeout=5 nova@ true" nova

Both must succeed without password prompt. If not, regenerate and distribute /var/lib/nova/.ssh/id_rsa.pub to authorized_keys on the peer host.

4. Validate CPU Model and Libvirt/QEMU Compatibility

Compare CPU models exposed to guests:

# On both hosts
virsh capabilities | grep -A 20 ''
qemu-system-x86_64 --version
libvirtd --version

In /etc/nova/nova.conf on both hosts, ensure [libvirt] cpu_mode is set consistently. For heterogeneous clusters, use cpu_mode = custom with a common baseline model in cpu_models. Mismatched major versions (e.g., QEMU 6.x vs 7.x) often break live migration; align versions or restrict migration to same-version hosts via host aggregates.

5. Confirm Storage and Block Migration Settings

For instances on local ephemeral disks (non-shared storage), block migration must be enabled. Check /etc/nova/nova.conf on both hosts:

[libvirt]
live_migration_permit_post_copy = True
live_migration_permit_auto_converge = True
live_migration_with_native_tls = True  # or False if using TCP
live_migration_uri = qemu+tcp://%s/system  # adjust for TLS
block_migration = auto  # or 'True' to force

If using shared storage (Ceph RBD, NFS, iSCSI), ensure block_migration = False (default) and the instance's root disk is on the shared backend. Verify with openstack server show -f value -c volumes_attached and openstack volume show .

6. Check Migration Network MTU and Firewall

Live migration traffic uses the management or dedicated migration network. Verify MTU consistency (typically 1500 or 9000 for jumbo frames) and that libvirt migration ports are open:

# On both hosts
ip link show 
iptables -L -n -v | grep -E '16509|16514'
# Or for firewalld
firewall-cmd --list-all | grep -E '16509|16514'

Packet loss >0.1% on this network will cause cutover timeouts. Run ping -c 100 -s 1472 (adjust for MTU) to test.

Fixes Mapped to Findings

SSH Trust Missing

Regenerate keys on one host and copy to all compute nodes:

# On first compute host
su -s /bin/bash -c "ssh-keygen -t ed25519 -f /var/lib/nova/.ssh/id_ed25519 -N ''" nova
cat /var/lib/nova/.ssh/id_ed25519.pub
# Append this key to /var/lib/nova/.ssh/authorized_keys on ALL compute hosts
chown nova:nova /var/lib/nova/.ssh/authorized_keys
chmod 600 /var/lib/nova/.ssh/authorized_keys

Restart nova-compute on all hosts after updating.

CPU Model Mismatch

Define a common baseline in /etc/nova/nova.conf on all compute hosts:

[libvirt]
cpu_mode = custom
cpu_models = "SandyBridge,IvyBridge,Haswell,Broadwell,Skylake"
# Choose the lowest common denominator across your fleet

Restart nova-compute and libvirtd on all hosts. Verify with virsh domcapabilities --cpu-model SandyBridge on each host.

Libvirt Migration URI or TLS Misconfiguration

If using TLS (recommended), ensure certificates exist in /etc/pki/libvirt/ or /etc/libvirt/qemu/ and live_migration_with_native_tls = True. For TCP without TLS:

[libvirt]
live_migration_with_native_tls = False
live_migration_uri = qemu+tcp://%s/system
# In /etc/libvirt/libvirtd.conf:
listen_tls = 0
listen_tcp = 1
auth_tcp = "none"  # or "sasl" if configured
tcp_port = "16509"

Restart libvirtd and nova-compute. Confirm listening with ss -ltnp | grep 16509.

Block Migration Not Enabled for Local Disks

For instances with ephemeral disks on non-shared storage, either enable block migration globally or per-migration:

# Global in nova.conf (requires restart)
[libvirt]
block_migration = auto

# Or per-migration CLI (no restart)
openstack server migrate --block-migration 
openstack server migration confirm  

Warning: Block migration doubles network traffic during disk copy. Test during maintenance windows and ensure migration network has sufficient bandwidth (10 Gbps+ recommended).

Insufficient Destination Resources

If placement shows no allocation candidates, either evacuate other instances from the destination host or add capacity. Check for stuck allocations:

openstack resource provider allocation show 
# If stale, delete with caution
openstack resource provider allocation delete  --consumer 

Escalation Criteria

Escalate to infrastructure, storage, or network teams when:

• Shared storage backend (Ceph, NFS, NetApp) reports I/O errors, degraded OSDs, or latency spikes during migration. Check ceph -s, ceph health detail, or storage array logs.
• Network fabric shows packet loss, CRC errors, or congestion on the migration VLAN. Engage network team for switch port diagnostics and QoS verification.
• Repeated failures occur across multiple availability zones with different compute hosts, indicating a systemic issue (e.g., broken image, kernel regression, or control-plane outage).
• Libvirt reports internal error: unable to execute QEMU command 'migrate' with hardware virtualization errors (VMX/SVM) — may require host firmware update or BIOS configuration review.

Verification After Fixes

After applying a fix, run a test migration on a non-production instance:

openstack server migrate --wait 
openstack server migration list --server 
# Confirm status: completed
openstack server show  -f value -c OS-EXT-SRV-ATTR:host
# Should now show destination host

Monitor /var/log/nova/nova-compute.log and /var/log/libvirt/libvirtd.log on both hosts for clean completion. Verify instance reachability and application health post-migration.

Version-Specific Notes

Live migration behavior and required nova.conf options differ between OpenStack releases. In Yoga and Zed, live_migration_with_native_tls defaults to True; in 2025.x (Caracal+), the default may change and live_migration_uri syntax can vary. Do not copy configuration blindly across versions. Always consult the release notes for your specific OpenStack version before changing migration settings.

Limitations

This guide covers the most common failure modes for KVM/libvirt-based compute. It does not address: migrations involving ironic baremetal, vGPU-mediated devices (requires vendor-specific mediation), SR-IOV VF migration (requires PF driver support), or cross-cell migrations. For those scenarios, consult the respective driver documentation and test in a staging environment first.