Live Migration in OpenStack Nova with Shared Storage: A Practical Guide

Problem: Moving VMs without downtime

Operators often need to shift workloads for hardware maintenance, firmware updates, or load balancing. Stopping virtual machines is not acceptable for many services, so a technique that keeps the instance running while its compute host changes is required.

Thesis

OpenStack Nova live migration, when backed by shared storage (NFS, Ceph, etc.), can relocate a running VM with sub‑second downtime, provided the hypervisor, libvirt, CPU features, and network/storage meet specific prerequisites.

Prerequisites and configuration

• Hypervisor and libvirt: KVM/QEMU with libvirt version ≥1.2.2 on both source and destination computes.
• Shared storage: The directory /var/lib/nova/instances (or the configured instances_path) must be exported via NFS or mounted from a CephFS/RBD pool and accessible with identical permissions on both hosts.
• CPU compatibility: Either the hosts share the same CPU model, or Nova is configured with cpu_mode=custom and a matching cpu_model (or cpu_mode=host-passthrough with identical CPUs).
• Nova configuration (on each compute node, typically in /etc/nova/nova.conf):

  [libvirt]
  live_migration_flag=VIR_MIGRATE_LIVE
  live_migration_uri=qemu+ssh://%s/system
  [workarounds]
  disable_libvirt_liveshrink=true
  

  On the controller, ensure allow_migrate_to_same_host=False in [DEFAULT].

Migration workflow overview

1. Pre‑phase: Nova validates destination suitability, prepares the libvirt domain, and starts the migration thread.
2. Execution phase: Memory pages are copied iteratively while tracking dirty pages; the instance disk remains on shared storage, so no disk copy is needed.
3. Post‑phase: A final sync of remaining dirty pages occurs, the VM is paused briefly (<200 ms on a 1 GbE link), resumed on the destination host, and the source resources are released.

Worked example: DevStack with two computes and NFS

Where to run: Commands prefixed with [controller] are executed on the DevStack controller (or any node with nova client access). Compute‑specific commands run on the respective compute hosts with root or nova user privileges.

1. Prepare shared NFS

On a storage server, export /srv/nova_instances with rw,no_root_squash,sync. On each compute host, mount it at /var/lib/nova/instances and verify:

mount | grep /var/lib/nova/instances

2. Launch a test instance

On the controller:

nova boot --flavor m1.tiny --image cirros-0.4.0 --nic net-id=$(neutron net-list | awk '/ private / {print $2}') test-vm

Wait until nova show test-vm shows status=ACTIVE.

3. Perform live migration

Identify the destination host (e.g., compute2) and run:

nova live-migration test-vm compute2

The command returns immediately; monitor progress with:

nova show test-vm

Look for the field OS-EXT-SRV-ATTR:host changing from the source to compute2. While the migration proceeds, keep a ping running inside the VM (via nova get-vnc-console test-vm novnc or SSH) to confirm continuous connectivity.

4. Verify post‑migration

After the host field updates:

• Run nova show test-vm and confirm OS-EXT-SRV-ATTR:host matches the destination.
• Check that the instance’s VNC console remains responsive.
• Log into the VM and verify that application data (e.g., a file written before migration) is still present.

Trade‑offs and limitations

• CPU feature mismatches trigger a libvirt error "CPU compatibility" and abort the migration. Mitigate by standardizing CPU models or using cpu_mode=custom with a common baseline.
• Network and storage bandwidth directly affect total migration time. If the link is saturated or the storage latency is high, the dirty‑page sync may exceed the allowed downtime threshold, causing Nova to abort. Monitor with iptables -L -v -n on the migration network and iostat on the storage backend.
• Post‑copy migration (available since Newton) can reduce total time under heavy load but is not enabled by default; it requires live_migration_flag=VIR_MIGRATE_LIVE | VIR_MIGRATE_POSTCOPY and compatible libvirt/QEMU versions.

Actionable closing checklist

1. Confirm shared storage is mounted with identical UUIDs on both computes.
2. Verify libvirt version: virsh --version ≥1.2.2.
3. Ensure CPU flags match (cat /proc/cpuinfo) or set a compatible cpu_mode in nova.conf.
4. Run a low‑risk test migration (e.g., a cirros instance) and watch the host field change.
5. After migration, ping the instance from an external host to validate uninterrupted connectivity.
6. If migration fails, consult /var/log/nova/nova-compute.log for libvirt errors and address CPU or network issues before retrying.