Proxmox VE High Availability: Architecture, Boundaries, and Failure Planning

The Problem: Survival Is Not Just Shared Storage

Proxmox VE High Availability (HA) promises that virtual machines keep running when a host dies. In practice, that promise depends on three independent systems agreeing: corosync for cluster membership, shared storage for disk images, and a watchdog for fencing. If any one of these is misconfigured, the cluster either fails to recover or—worse—writes to the same disk from two nodes at once.

The useful takeaway: a production HA cluster needs at least three voting members, physically or logically separated network planes, and hardware that can actually reset itself. Everything else is tuning.

Minimum Requirements

Proxmox VE HA relies on corosync, a cluster communication engine, to maintain quorum—a majority agreement on which nodes are alive. A cluster that loses quorum becomes read-only to prevent split-brain, a condition where partitioned nodes both believe they are in charge.

• Three nodes minimum. Two-node clusters cannot maintain quorum during a single node failure unless you add a separate quorum device (qdevice) as a tie-breaker.
• Shared storage. Every node must access the same disk images. NFS, iSCSI, or Ceph are common choices.
• Hardware watchdog. Each host needs a watchdog timer that the kernel can trigger to force a hardware reset. This is how the cluster "fences" a suspected-dead node.
• Redundant cluster network. Corosync requires low-latency, redundant links. Routing cluster traffic across a WAN or a congested overlay can trigger false fencing.

Cluster Size and Quorum Behavior

Topology	Quorum on single node failure	Assessment
2 nodes, no qdevice	Lost	Not suitable for HA
2 nodes + qdevice	Maintained	Acceptable for small deployments
3 nodes	Maintained	Smallest standard production design

The Smallest Suitable Design

Three physical hosts with the following layout:

• Two dedicated NICs (or a bond) on a management VLAN for corosync and pmxcfs, the Proxmox clustered file system that stores configuration.
• One or more NICs on a separate storage network.
• A standard Linux bridge for guest traffic.
• Watchdog enabled in BIOS and loaded in the kernel.

For storage, NFS or iSCSI is simpler to operate than Ceph, but it introduces a single point of failure at the storage layer unless the NFS server or iSCSI target is itself redundant. Ceph replicates data across nodes and eliminates the storage SPOF, at the cost of more moving parts.

Trust and Data Boundaries

Separate the cluster into three planes. The control plane carries corosync heartbeats and pmxcfs traffic on the management VLAN. The storage plane carries disk I/O on a dedicated network. The guest plane carries VM traffic on its own bridge.

This separation matters for two reasons. First, a noisy VM saturating the guest bridge should not delay corosync heartbeats and trigger false fencing. Second, a compromised guest should not be able to reach the cluster control plane or storage network directly. VLANs and firewall rules enforce these boundaries.

Operational Checks

Run these commands as root on any cluster node. They are read-only and safe for routine use.

Command	What to verify	Risk if skipped
pvecm status	All expected nodes are online and the cluster reports a quorate state.	Undetected partition or missing node.
corosync-quorumtool -l	A majority of votes is present and the ring is operational.	Split-brain during network faults.
ha-manager status	HA-managed services report started on the correct node.	Silent HA service failure.
pvesm status	Shared storage pools are active and reachable from this node.	Failed VM recovery due to missing disks.
ls -l /dev/watchdog	The device node exists and the appropriate kernel module is loaded.	Fencing fails; dead node may not reboot.

Concrete Diagnostic Decision

After isolating a node from the cluster network to simulate failure, run ha-manager status on a surviving node. If the VMs from the isolated node show stopped rather than started on a healthy host, do not manually start them. Investigate fencing logs first. Manually starting an HA-managed VM while the original node might still be alive risks split-brain disk access. If the isolated node is confirmed dead and VMs remain stopped, check that shared storage is reachable from the surviving node with pvesm status before forcing a start.

Failure Modes

• Total quorum loss. The cluster becomes read-only. HA cannot fence nodes or migrate services. This is the expected behavior of a two-node cluster without a qdevice when one node fails.
• Storage partition. If shared storage becomes unreachable, VMs cannot restart on surviving nodes because their disk images are unavailable.
• Watchdog timeout. An overly aggressive watchdog reboots a healthy node. Conversely, if the watchdog is emulated in a VM rather than backed by hardware, it may fail to reset the physical host, leaving a suspected-dead node running.
• Corosync link degradation. High latency or packet loss on the cluster network can cause nodes to be declared dead prematurely.

Conditions That Change the Design

Geographic redundancy. Spanning sites introduces latency that corosync tolerates poorly. You would need a qdevice at a third location and asynchronous storage replication. HA restarts become slow because disk access crosses the WAN.

Zero RPO/RTO. HA restarts VMs rather than live-migrating them. A restart means downtime and potential data loss for applications that are not crash-consistent. If you require zero recovery point objective (RPO) and near-zero recovery time objective (RTO), you need live migration and synchronous storage instead of HA restart.

Two-node constraint. If hardware limits you to two nodes, add a separate qdevice. Without it, a single node failure removes quorum and halts HA protection entirely.

Practical Verification

Test the design by disconnecting the cluster network cable from one node. Confirm that the remaining nodes maintain quorum and that affected VMs restart within the configured HA timeout. Reconnect the cable afterward. This drill changes cluster state, so perform it during a maintenance window and verify that the returning node rejoins cleanly with pvecm status.