Enabling and Troubleshooting Live Migration in Harvester HCI

Useful answer

To move a running virtual machine (VM) between Harvester hosts with minimal downtime, you need three things: shared storage that supports ReadWriteMany (RWX) access, a VM configured with spec.liveMigration.enabled: true, and compatible CPU/libvirt/QEMU versions across all nodes. When these conditions are met, Harvester can initiate a live migration that copies the VM’s memory while the VM continues to run, then switches the workload to the destination host.

How live migration works – a worked example

The following steps show how to enable live migration for an existing VM using the Harvester API (kubectl). Adjust the placeholders to match your environment.

1. Verify the storage class is RWX (run as a cluster‑admin or with admin role on the Harvester cluster):

   kubectl get sc longhorn -o yaml
   

   Look for volumeBindingMode: Immediate and allowVolumeExpansion: true; the important flag is accessModes: containing ReadWriteMany. If the storage class shows only ReadWriteOnce (e.g., a local hostPath class), live migration will not work.
2. Enable live migration on the VM. Suppose the VM is named web-server in the default namespace:

   kubectl patch vm web-server -n default --type merge -p '{"spec":{"liveMigration":{"enabled":true}}}'
   

   This patches the VirtualMachine custom resource (CR). You can also achieve the same via the Harvester UI: VM → Settings → Live Migration → toggle enabled.
3. Start the migration. In the UI, select the VM and choose “Migrate”, then pick a target host. Harvester will create a migration job and update the VM’s status to Migrating.
4. Verify the move. After a few seconds, the status should change to Running and the host field will show the destination node. You can also check from either host:

   virsh list --all | grep web-server
   

   The VM should appear only on the destination host after migration completes.

Mechanism in brief

Harvester uses libvirt’s virsh migrate command under the hood. The process requires:

• Shared storage: the VM’s disks must be accessible from both source and destination hosts at the same path. Longhorn, NFS, or any CSI driver that provides RWX satisfies this.
• CPU compatibility: all nodes must expose the same CPU model or use a compatibility mode (e.g., host-passthrough with cpu mode='custom' and a matched model). Mismatched CPUs trigger “CPU incompatibility” errors.
• Network: libvirt sends the VM’s memory pages over the management network. Latency under ~10 ms and at least 1 GbE bandwidth reduce migration time and the risk of timeouts.
• Matching libvirt/QEMU versions: the version shipped with Harvester must be identical on every node; otherwise the migration protocol may fail silently.
• Fencing: a functional STONITH mechanism prevents split‑brain if the source host fails while the VM’s memory is still being copied.

Limits and common mistakes

Storage‑related limits

• Local storage classes (hostPath, local PV) cannot be used because the disk is not present on the destination host. Migration may appear to succeed but the VM will lose access to its disk, leading to data corruption.
• Only virtio‑blk or virtio‑scsi disk types are supported. IDE or SCSI emulation disks will cause the migration to abort with an “unsupported device” error.
• If the storage class is RWX but the underlying storage experiences high latency or throttling, the migration can stall; monitor storage IOPS and latency during the operation.

CPU and version limits

• Harvester ≥ 1.1.0 is required for the spec.liveMigration field. Older versions lack the API and must rely on manual virsh commands.
• CPU model mismatches are a frequent source of failure. Use kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.architecture}' to confirm uniformity, or set a common baseline in the Harvester cluster settings (CPU → Compatibility Mode).
• Libvirt/QEMU version drift can happen after a node upgrade. Check with ssh node-xxx 'virsh version' on each host and ensure they match.

Network and fencing limits

• Multicast is not strictly required, but the Harvester network plugin must allow the TCP ports used for libvirt migration (default 49152‑49215). If a firewall blocks these ports, the migration will hang.
• Without fencing, a failed source host during migration can leave the VM running on both hosts (split‑brain). Verify your STONITH device is healthy: pcs status for a Pacemaker cluster or the equivalent for your fencing solution.

Verification steps

1. Confirm shared storage: kubectl get sc <storage-class-name> -o jsonpath='{.accessModes}' should output ["ReadWriteMany"].
2. After initiating migration, watch the VM’s status: kubectl get vm <vm-name> -n <namespace> -w. Look for the transition Running → Migrating → Running.
3. Check the destination host with virsh list --all; the VM should appear only there once migration finishes.
4. Inspect Harvester events for errors: kubectl get events -n harvester-system --field-selector involvedObject.name=<vm-name>. Look for messages like “CPU incompatibility” or “storage not shared”.

Practical way to check the result

After migration, run a simple workload inside the VM (e.g., a ping loop or a CPU stress test) and confirm it continues without interruption. If the VM pauses for more than a few seconds, revisit the storage latency, network bandwidth, or CPU compatibility checks above.