Putting a Harvester Node into Maintenance Mode: Automatic VM Evacuation and Safe Hardware Service

Desired Outcome

Place a Harvester node into maintenance mode so its running virtual machines are automatically live-migrated to other nodes. The cluster stays available while you safely service or replace hardware on the evacuated node.

Prerequisites

• A Harvester cluster (v1.2 or later; behavior described here assumes a current 1.x release) with at least three healthy nodes, so quorum and capacity survive losing one node.
• VMs using shared, network-attached storage (Longhorn volumes, the default in Harvester) rather than node-local disks.
• Sufficient CPU and memory headroom on the remaining nodes to absorb the evacuated VMs.
• No VMs on the target node using PCI/GPU passthrough, USB host devices, or other host-pinned resources – these cannot be live-migrated.
• Access to the Harvester UI with permissions to manage nodes, or kubectl access to the cluster.

Procedure

1. Check cluster health first. In the Harvester UI, confirm all other nodes show Ready and that storage health is green. Do not start maintenance while another node is degraded.
2. Enable maintenance mode. In the UI go to Hosts, select the target node, and choose Enable Maintenance Mode from the node actions menu. Harvester cordons the node and begins evicting workloads; KubeVirt live-migrates each migratable VirtualMachineInstance (VMI) to another node.
3. Monitor evacuation. On the node detail page, watch the VM count drop toward zero. From a management workstation with cluster access you can also watch VMIs:

   kubectl get vmis -A -o wide
   

   Each migrated VMI should show a new node in the NODE column and remain in the Running phase. Non-migratable VMs (passthrough devices, local disks) will block the drain and must be shut down manually before the node finishes entering maintenance.

4. Confirm the node is in maintenance. The node status in the UI changes to Maintenance, and only system pods remain. Verify no user VMs are left before touching hardware.
5. Perform the hardware work. Shut down or power off the node via its BMC/IPMI once evacuation is complete.
6. Return the node to service. Power the node back on and let it rejoin the cluster. Then in the UI select the node and choose Disable Maintenance Mode (this uncordons it). VMs are not automatically moved back; they rebalance only when restarted or migrated, which is normal.

Expected Checks

• Node shows Maintenance status and zero running VMs before power-off.
• Every migrated VM shows Running with a new node assignment; spot-check guest connectivity (ping, SSH, or application health check) from inside or outside the guest.
• Longhorn volume health stays Healthy or Degraded-but-rebuilding during the operation; investigate any volume stuck in fault before proceeding.
• After re-enabling, the node returns to Ready and passes the UI health checks.

Recovery Options

• Stalled migration: If a VMI will not migrate (commonly due to passthrough devices or a migration timeout), shut that VM down gracefully from the guest or UI, let the drain finish, and start it again on another node after maintenance – or before, if capacity allows.
• Abort maintenance: Select Disable Maintenance Mode to uncordon the node. Already-migrated VMs stay where they are; remaining VMs continue running on the node.
• Insufficient capacity: If target nodes lack headroom, evacuations pend. Free resources (stop non-critical VMs) or add a node, then retry.

Limitations

• Live migration requires shared storage and no host-pinned devices; plan manual shutdowns for exceptions.
• Maintenance-mode behavior, migration timeouts, and UI options vary between Harvester releases – validate the exact steps against your deployed version's documentation before relying on them in production.

Practical Verification

After the full cycle, confirm the node is schedulable again:

kubectl get nodes

The serviced node should show Ready with no SchedulingDisabled marker. Migrate or restart one test VM onto it to prove it can host workloads normally.