Live VM Migration in Harvester: Constraints, Workflow, and Verification

The Problem: Moving Running Workloads Without Downtime

You need to drain a node for kernel updates or hardware maintenance, but the VMs on it cannot tolerate a reboot. Harvester’s live migration promises to move running VMs between nodes with zero downtime. Understanding when it works and when it does not helps you avoid unexpected failures during maintenance.

How Harvester Live Migration Works

Harvester uses a shared‑nothing architecture. VM disk images reside on Longhorn replicated volumes, so storage does not move—only the VM’s memory state and CPU execution context are transferred over the cluster network. The control plane (harvester‑api) coordinates the migration via KubeVirt’s VirtualMachineInstanceMigration (VMI Migration) CRD, monitoring progress and rolling back on failure.

Network continuity is provided by Calico’s endpoint‑independent IP addressing. The VM retains its IP and MAC address; Calico updates routing tables to point to the new node, so the guest sees no network change and no ARP storms occur.

When Migration Fails: Hard Constraints

Three categories of VMs cannot be live‑migrated in Harvester v1.3.x:

• Direct device passthrough: PCI/USB devices bound to a specific node (GPUs, NICs with SR‑IOV, USB dongles).
• HostPath volumes: Any volume backed by a local node path that is not also replicated via Longhorn. If the HostPath is mirrored by Longhorn, migration may succeed, but a pure HostPath blocks migration.
• Non‑standard CPU models: Custom CPU pinning or model definitions that do not exist on the target node.

Attempting migration on these VMs results in a MigrationImpossible condition on the VMI Migration object within seconds; the VM remains running on the source node.

Worked Example: Migrating a Standard Web Tier VM

Assume a three‑node cluster (node-1, node-2, node-3) running Harvester v1.3.2. A VM named web-frontend in namespace production runs on node-1 with a Longhorn volume and no passthrough devices.

Step 1: Verify Migration Eligibility

Run from any control‑plane node with kubectl access (requires harvesterhci.io/virtualmachineinstances read permission):

kubectl get vmi -n production web-frontend -o jsonpath='{.spec.domain.devices}'

Look for hostDevices, gpus, or a custom cpu.model. Empty or standard cpu: {} indicates eligibility.

Step 2: Trigger Migration

Create a VirtualMachineInstanceMigration manifest (requires create permission on virtualmachineinstancemigrations):

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
  name: web-frontend-mig
  namespace: production
spec:
  vmiName: web-frontend

Apply it:

kubectl create -f web-frontend-mig.yaml

The migration controller selects a target node with sufficient CPU/memory and Longhorn replica access.

Step 3: Monitor Progress

Watch the VMI and the migration object (use the full kind; the short name vmim may not be available in all kubectl contexts):

kubectl get vmi -n production web-frontend -w
kubectl get virtualmachineinstancemigration -n production -w

The VMI’s status.migrationState progresses: Scheduled → Preparing → Running → Completed. The migration object shows phase: Succeeded on completion. For a 4 GB RAM VM on a 10 GbE cluster network, migration typically takes 30–60 seconds, but actual time depends on memory dirty‑page rate and concurrent workloads.

Step 4: Verify Post‑Migration Health

Run from the control‑plane node:

# Confirm the VM landed on the new node
kubectl get vmi -n production web-frontend -o jsonpath='{.status.nodeName}'

# Check network connectivity (requires QEMU guest agent inside the VM and
harvesterhci.io/virtualmachineinstances/exec permission)
kubectl exec -n production vmi/web-frontend -- ping -c 3 10.96.0.1

# Verify application responsiveness
curl -s -o /dev/null -w "%{http_code}" http://web-frontend.production.svc.cluster.local/healthz

The kubectl exec step only works if the QEMU guest agent is installed and running inside the VM; otherwise the command will fail. An HTTP 200 response indicates the application continued to serve requests throughout the migration.

Performance Impact and Trade‑offs

Live migration consumes network bandwidth (memory dirty‑page transfer) and storage I/O (Longhorn replica synchronization). On a 10 GbE cluster, expect 2–4 Gbps sustained during migration. Concurrent migrations share the same fabric; Harvester does not throttle migration traffic by default.

To limit bandwidth, edit the KubeVirt custom resource (requires cluster-admin and a KubeVirt operator restart):

spec:
  configuration:
    developerConfiguration:
      migrationBandwidthPerMigration: 100MiB

Ensure Longhorn replica rebuilds are not running concurrently; check kubectl get engines -n longhorn-system before starting large batches.

Limitations to Plan For

• No live migration for GPU/passthrough VMs: Must shut down, move, and restart. Reserve capacity for cold‑migration windows.
• One migration at a time per VMI: You cannot queue multiple migrations for the same VM.
• Rollback is automatic but not instant: On failure, the VMI reverts to the source node. Downtime during rollback is typically under 5 seconds, but this is not guaranteed zero.
• Version skew: Source and target nodes should run identical Harvester/KubeVirt versions. Mixed‑version clusters may succeed if the KubeVirt API versions are compatible, but this is not officially supported and should be avoided for production workloads.

Actionable Closing: Pre‑Flight Checklist

Before draining a node, list non‑migratable VMs (requires jq; install it if missing):

kubectl get vmi --all-namespaces -o json | jq -r '.items[] | select(.spec.domain.devices.hostDevices or .spec.domain.devices.gpus or .spec.domain.cpu.model) | "\(.metadata.namespace)/\(.metadata.name) on \(.status.nodeName)"'

Any output indicates VMs that need manual shutdown. For the remaining VMs, trigger migrations in small batches (e.g., 2–3 at a time), monitor with watch kubectl get virtualmachineinstancemigration -A, and verify each with the health checks above. This approach turns a potentially disruptive maintenance window into a routine operation.