Proxmox VE High Availability: How Pacemaker Moves VMs When a Node Fails

The Problem: A Node Goes Down

When a Proxmox node loses power, network connectivity, or crashes, every VM or container that lives on it stops abruptly. In a single‑node setup, that means a service is unavailable until the node is back online. For workloads that must stay online, a manual restart is unacceptable.

What HA actually does

Proxmox VE’s High‑Availability (HA) feature uses Corosync for cluster communication and Pacemaker as the resource manager. Pacemaker watches each node’s health and the state of every HA‑managed VM or container. If it detects that a node is unreachable or that a VM has stopped, it can:

• Fence the failed node (if fencing is configured) to isolate it and prevent data corruption.
• Start the affected VM or container on another node that has access to the shared storage where its disk image resides.

The result is an automatic failover that usually happens within a few seconds, but the exact time depends on the cluster’s hardware, network latency, and storage performance.

Prerequisites for HA

1. Cluster creation: On the first node run pvecm create mycluster. Subsequent nodes join with pvecm add IP_OF_FIRST_NODE. All nodes must be running the same Proxmox VE version (e.g., 8.x).
2. Shared storage: Every HA‑managed VM must store its disk image on a datastore that all nodes can mount. Common options are NFS, Ceph RBD, or a ZFS pool exported over iSCSI. The storage itself should be highly available—e.g., a Ceph cluster with quorum, or an NFS server behind a load balancer with redundant backends.
3. Quorum: Verify that the cluster can reach quorum with pvecm status. In a three‑node cluster you should see “Quorum: 2 of 3” or “Quorum: 3 of 3” depending on your quorum policy.
4. Fencing: A fencing agent (IPMI, iLO, DRBD, or a custom script) must be configured for each node. Fencing guarantees that a failed node cannot continue to serve data after it has been isolated. Test the agent with pve-ha-manager fence nodeX; if the agent is not configured, the command will return an error and Pacemaker will not be able to fence the node.

Marking a VM or Container as HA‑Managed

From any node, add the resource to Pacemaker:

# Replace 101 with the VM or CT ID
pve-ha-manager add vm:101

Pacemaker will create a resource definition and attempt to start the VM immediately. Verify that the resource is active with pve-ha-manager status, which shows the current state and the node on which it is running.

A Practical Failover Test

To confirm that HA works, simulate a node outage. From a healthy node, bring down the network interface of node2:

ssh node2 "ip link set dev eth0 down"

Pacemaker will detect that node2 is no longer reachable. If fencing is configured, the node will be fenced; if not, Pacemaker will mark it as down but will not isolate it, which can lead to a split‑brain situation. In either case, Pacemaker will stop the VM on the failed node and start it on another healthy node. After a few seconds, check the status:

pve-ha-manager status
qm status 101

In a lab environment with a fast network and a responsive storage backend, the VM typically comes back up in less than 30 seconds. In production, the time can be longer if the storage is slow or the cluster is heavily loaded, so treat the 30‑second figure as an optimistic reference rather than a hard guarantee.

Trade‑offs and Limitations

Consideration	Impact
Shared Storage	Must be truly redundant; otherwise it becomes a single point of failure for the entire HA setup.
Fencing	Critical to avoid split‑brain. An unconfigured or unreliable fencing agent can cause data corruption.
Failover Latency	Depends on network, storage I/O, and Pacemaker tuning. Heavy workloads may see delays of several minutes.
Application‑Level Resilience	HA only restarts the VM when the host fails. Internal crashes are not automatically handled unless you add health‑check scripts or use application‑level clustering.
Operational Complexity	Requires understanding of Corosync, Pacemaker, fencing, and shared storage. Misconfiguration can lead to extended downtime.

Checklist Before Enabling HA

1. Confirm that every node can mount the shared datastore (e.g., ls /mnt/proxmox-data).
2. Run pvecm status to ensure quorum is achieved.
3. Test fencing: pve-ha-manager fence nodeX should trigger the agent and log a state change.
4. Verify that the VM’s disk resides on shared storage before adding it to HA.
5. Use pve-ha-manager status to confirm all HA resources are in the Started state.
6. Set up alerting on Pacemaker and Corosync logs (e.g., journalctl -u pve-ha-lrm, journalctl -u corosync).

When to Use HA

• Workloads that cannot tolerate even a few minutes of downtime.
• All VM disks are on a shared, highly available storage backend.
• Reliable fencing is in place to prevent split‑brain.

When to Skip HA

• Stateless or easily restartable services.
• Environments where shared storage cannot be made highly available.
• Lack of fencing hardware or software.

Bottom Line

Proxmox VE’s HA feature can turn a cluster of ordinary nodes into a resilient platform where VMs survive node outages automatically. The key to success is:

• Ensuring that shared storage is truly redundant.
• Testing fencing thoroughly in a lab before production.
• Monitoring Pacemaker and Corosync logs for rapid detection of failures.
• Adding application‑level health checks if you need to recover from internal VM crashes.

With these safeguards in place, you can expect failover times that are often under a minute, but always validate the performance in your own environment.