Proxmox VE Cluster Design: Sizing, Quorum, and HA Fencing Decisions

The Challenge: Ensuring Stability in Multi-Node Clusters

Moving from a single Proxmox VE host to a production cluster introduces the risk of "split-brain" scenarios—where nodes lose communication but remain powered on, potentially writing to the same shared storage and causing catastrophic data corruption. To prevent this, you must make specific decisions regarding node count, quorum mechanisms, and fencing hardware.

Decision Matrix: Cluster Topology and Quorum

The primary constraint in Proxmox VE clustering is maintaining a majority (quorum) to allow the cluster to make decisions. In an even-node cluster, a 50/50 split results in a loss of quorum, freezing the cluster's ability to manage resources.

Topology	Node Count	Quorum Method	Fencing Requirement	Live Migration Constraint
Minimal HA	3	Built-in majority	IPMI / APC / SSH	Identical CPU types, shared storage
Balanced HA	4 + QDevice	External QDevice	IPMI / APC	Identical CPU types, shared storage
Large Scale	5–7	Built-in majority	IPMI / APC / Redfish	Identical CPU types, shared storage
Max Capacity	8 + QDevice	External QDevice	Redfish / IPMI	Identical CPU types, shared storage

Trade-offs and Engineering Constraints

• Node Count vs. Complexity: A 3-node cluster is the simplest HA-ready design. Adding nodes increases total resource capacity but adds more fencing endpoints and network overhead.
• The QDevice Necessity: For clusters with an even number of nodes (2, 4, 6, 8), a QDevice (Quorum Device) is required. This is a lightweight service running on an external host (like a Raspberry Pi or a small VM) that acts as a tie-breaker.
• Fencing Reliability: Fencing (or STONITH—Shoot The Other Node In The Head) is mandatory for High Availability (HA). IPMI is the standard for server hardware. SSH fencing is risky because it requires the target node's OS to be responsive to execute the shutdown command.
• CPU Homogeneity: Live migration preserves VM state but requires the destination CPU to support the same instruction sets as the source. Mixed CPU generations will cause migration failures unless the CPU type is set to a generic baseline, which reduces performance.

Resource Allocation for Performance Isolation

To prevent a single VM from starving others in a cluster, use these three allocation mechanisms:

• CPU Shares (cpuunits): A relative weight for the kernel scheduler. A VM with 2048 units gets twice the CPU time of a VM with 1024 units during contention.
• Memory Ballooning: Allows the host to reclaim unused memory from a guest to allocate it elsewhere.
• NUMA Pinning: Binds a VM to a specific physical CPU socket and its local memory bank to eliminate the latency of cross-socket memory access.

Example Configuration: To set a VM (ID 100) with high priority and NUMA pinning, run the following on any cluster node:

# Run as root on any cluster node
qm set 100 --cpuunits 2048 --memory 16384 --balloon 8192 --numa 0

Implementing HA Fencing

Fencing ensures a failed node is powered off before its VMs are restarted on another node. Configure fence devices in /etc/pve/ha/fence.cfg. This file is replicated across the cluster via the Proxmox Cluster File System (pmxcfs).

Example IPMI Configuration:

# /etc/pve/ha/fence.cfg
fence0: ipmi,ip=192.168.10.11,user=admin,pass=secret,port=1
fence1: ipmi,ip=192.168.10.12,user=admin,pass=secret,port=1

Verification: Before enabling HA for production VMs, test the fence action to ensure the BMC (Baseboard Management Controller) credentials and network paths are correct:

# Run on any node to test the fence device for node 0
ha-manager fence fence0 --test

The expected result is an OK response and a corresponding power-cycle event in the hardware BMC logs.

Validation and Practical Checks

Once the cluster is configured, perform these checks to verify the state:

1. Quorum Status: Run pvecm status. Ensure Quorate: Yes is displayed.
2. QDevice Connectivity: For even-node clusters, run corosync-qdevice-tool -m to confirm the external witness is connected.
3. Failover Simulation: Perform a graceful shutdown of one node using systemctl poweroff. Monitor journalctl -u ha-manager on the remaining nodes to verify that VMs are automatically relocated and started.
4. Resource Verification: Inside the guest VM, use numactl --hardware to confirm that memory and CPU are correctly pinned to the intended NUMA node.

Limitations and Risks

• Fencing Failure: If a fence device is misconfigured, the HA manager may fail to restart VMs, or worse, two nodes may attempt to run the same VM, leading to disk corruption.
• Network Partitions: If the network between nodes fails and quorum is lost, all HA resources will be frozen to protect data. Ensure your Corosync network is redundant.
• Storage Saturation: Simultaneous failovers of multiple large VMs can saturate shared storage I/O, potentially causing other nodes to time out and trigger unnecessary fencing.