Configuring and Testing Proxmox VE High Availability for Automatic VM Failover

Problem: Unexpected host failure causes immediate downtime

When a Proxmox VE node loses power or network connectivity, any virtual machines or containers running on that node stop instantly. For services that require continuous availability, this interruption can affect users and business processes until an administrator manually restarts the workloads on another node.

Thesis: Proxmox VE HA can automatically relocate VMs/CTs to healthy nodes when shared storage and proper fencing are in place

The built‑in High Availability (HA) stack uses corosync for cluster messaging and pacemaker for resource management. If a node becomes unresponsive, the HA framework fences the failed node (if configured) and restarts the affected VMs/CTs on another node that has access to the same shared storage, typically within 30 seconds.

How the HA stack works

Corosync provides a quorum‑based communication layer so all nodes agree on the cluster state. Pacemaker runs as a resource manager; it monitors HA‑managed VMs/CTs and initiates failover when the resource agent reports a node failure. Fencing agents (e.g., IPMI, iLO, or custom scripts) ensure the failed node is powered off or isolated to prevent split‑brain scenarios.

Worked example: three‑node cluster with NFS storage

1. Cluster nodes: pve1, pve2, pve3.
2. Shared storage: an NFS export mounted at /mnt/pve/nfs-shared on all nodes.
3. A web server VM (ID 101) runs on pve2 and is marked HA‑managed.
4. Simulate a power loss on pve2 by disconnecting its power or shutting it down.
5. Corosync detects loss of quorum messages from pve2; pacemaker triggers the fencing agent (if configured) to ensure the node is offline.
6. The HA resource agent for VM 101 is restarted on either pve1 or pve3, whichever has access to the NFS share.
7. Service interruption lasts roughly the time needed for the VM to shut down on the failed node and start on the target node—typically 20‑35 seconds in a LAN environment.

This example shows that, with shared storage and functional fencing, the workload continues without manual intervention.

Configuration considerations

• Enable HA for a VM/CT: ha-manager add vm:101 --state started.
• Configure a fencing agent appropriate for your hardware (e.g., IPMI). Test the agent in isolation before relying on it for HA.
• Ensure the shared storage is accessible from all nodes and that the storage lock mechanism (e.g., NFS lock, LVM‑lock) is compatible with HA.
• Set the node‑dead interval in /etc/pve/ha/resources.cfg if you need to adjust sensitivity; the default is 20 seconds.

Trade‑offs and limitations

HA adds operational complexity:

• Incorrect fencing can lead to split‑brain, where multiple nodes believe they are active and may write to the same disk image, causing corruption.
• Excessive sensitivity (short node‑dead timer) may trigger unnecessary migrations during transient network glitches, increasing overhead and potentially affecting performance.
• HA does not protect against storage failures; the shared storage itself must be reliable or replicated.

To mitigate these risks, validate HA in an isolated test lab, monitor the logs in /var/log/pve-ha*/logs and /var/log/corosync.log, and schedule regular failover drills.

Verification steps

After configuring HA, you can check the cluster state:

# Check corosync quorum and cluster membership
pvecm status

# Show pacemaker resource status
crm status

# List HA‑managed resources and their current node
pveha show

After simulating a node failure, verify that:

• The failed node is reported as offline or fenced in crm status.
• The VM/CT appears as started on a different node in pveha show.
• Logs contain a fencing action followed by a resource restart on the target node.

Closing: Test, monitor, and refine

Proxmox VE HA can turn an unexpected host outage into a seamless failover, but only when the underlying components—shared storage, corosync/pacemaker, and fencing—are correctly configured and validated. Begin with a small test cluster, simulate failures, review logs, and adjust the node‑dead timer or fencing settings as needed. Regular drills and log monitoring will help ensure the HA behaves as expected when a real incident occurs.