Limits of Proxmox VE HA retry handling for in-flight VM writes

Question

The goal is to determine whether Proxmox VE’s HA manager can safely retry a failed VM resource without causing duplicate writes to shared storage when the original node loses contact.
Current documentation shows that HA stops the failed resource before relocation and relies on the storage layer for consistency, but it does not provide an idempotent retry mechanism for in‑flight disk writes. The safety of a retry therefore depends on fencing effectiveness, quorum state, and the guest’s ability to recover from incomplete I/O.

Does HA guarantee exactly‑once write semantics for a VM after a node failure, or can a retry lead to duplicate writes?
What verification steps can confirm that no duplicate VM instances start on the same shared‑storage disk during a failover scenario?
How does the reliability of fencing and quorum configuration affect the possibility of unsafe retries?

Tasadduq Burney · Answer

Short answer
No. Proxmox VE HA does not guarantee exactly-once write semantics for a VM's in-flight disk writes, and a retry after node failure can result in duplicate writes or concurrent access to the same shared-storage image if the original node is not fully isolated. HA stops and relocates the resource, but it relies on fencing, quorum, and the storage layer rather than an idempotent retry mechanism.
Confirmed behavior

The HA manager tracks VM resources through pve-ha-crm and pve-ha-lrm. When a node loses contact, the resource is marked failed and HA may start it on another node.
HA does not replay, deduplicate, or serialize in-flight QEMU disk writes. Consistency depends on the disk cache mode, storage backend, and guest filesystem journaling.
Failover is gated by corosync quorum and a fencing agent. The original node should be fenced before the VM starts elsewhere.

Where unsafe retries come from
If fencing fails, is slow, or is misconfigured, the original node can remain online and continue writing to the shared disk while HA starts a second VM instance on another node. Quorum loss can stall HA decisions or cause the manager to act without a reliable majority, widening the window for split-brain. File-based shared storage such as NFS generally offers weaker VM-level locking than Ceph RBD or a clustered filesystem, so a fencing gap is more dangerous there.
Verification steps

Identify the fencing agent configured for each node and check HA logs for successful fencing during prior events. Run a controlled fencing test on a non-production node to confirm isolation completes before HA would relocate a VM.
Confirm corosync quorum is healthy and observe quorum state during a simulated network isolation. HA should not proceed with failover without quorum.
Perform a controlled failover of a non-production VM on the actual shared-storage backend. From the destination node, check storage-level access logs or VM status to confirm only one QEMU process has the disk image open.
Review the VM's QEMU disk cache setting, such as cache=writeback versus cache=none, and ensure the guest filesystem uses journaling so incomplete writes can be recovered after a crash.

Assumes a standard Proxmox VE HA stack with corosync, pve-ha-manager, pve-ha-crm, and pve-ha-lrm; behavior may differ in older releases or custom HA policies.
One missing detail changes the recommendation: which shared-storage backend and lock manager you use. Ceph RBD with exclusive locking may reject a second attach even if fencing lags, while NFS or plain block storage without a distributed lock manager makes fencing effectively the only protection. Specify your storage backend before finalizing the failover procedure.

Limits of Proxmox VE HA retry handling for in-flight VM writes

1 answer

Short answer

Confirmed behavior

Where unsafe retries come from

Verification steps

Your perspective belongs here.

0 question comments