HA VM remains stopped after migration on NFS v4 shared storage
0 reputation · 21 Aug 2021, 05:01 UTC
0 reputation · 21 Aug 2021, 05:01 UTC
The goal is to confirm whether a Proxmox VE HA-managed VM can reliably transition to the running state after live migration when its disk resides on an NFS v4 export, given that the same configuration succeeds in a single‑node test but fails in a multi‑node production cluster.
The uncertainty stems from the HA resource agent’s reliance on flock‑based locking, which may not behave as expected under certain NFS v4 server settings, and it is not documented whether the agent automatically detects such lock limitations and falls back to an alternative fencing mechanism.
Does the HA resource agent currently probe the NFS server for lock capability before attempting to start a VM?
Is there a configurable option to enable automatic fallback to a different locking or fencing strategy when lock acquisition fails on NFS v4?
Should administrators treat NFS v4 as unsuitable for HA workloads unless specific export options are applied, or can the cluster be made safe without manual storage changes?
Short answer: no, the Proxmox VE HA resource agent does not probe the NFS server for lock capability before starting a VM, and there is no built-in option to automatically fall back to a different locking strategy when lock acquisition fails on NFSv4. The HA stack assumes the storage layer behaves correctly; if it doesn't, the start attempt fails and the VM is left stopped. That is almost certainly what you are seeing, and it explains why the single-node test passed — with one node there is no cross-host lock contention to expose the problem.
Confirmed behavior (well-established in current PVE releases): the HA manager (pve-ha-crm/lrm) migrates the VM configuration and then issues a normal start on the target node. If the disk image cannot be opened — because of a stale lock, an idmapping mismatch, or the storage not being active/shared on that node — the start fails, the error is logged, and the CRM does not retry indefinitely. The VM stays stopped. There is no storage-capability probe and no pluggable fallback fencing mechanism for this path.
Likely explanation for your case, in order of probability:
The HA logs tell you exactly which case you have:
journalctl -u pve-ha-lrm -u pve-ha-crm --since "-1h"
pvesm status
qemu-img info /mnt/pve/<storage>/images/<vmid>/<disk>
Look for the start attempt and its error on the target node. A lock/permission error points at NFS; a "storage not available" error points at configuration. Confirm the image is readable from the target host with the qemu-img info check above.
Not inherently, but it is not "safe by default." Proxmox HA's correctness relies on watchdog-based node fencing plus storage that presents consistent locking and identity semantics on every node. The cluster cannot be made safe purely through HA settings if the NFS export behaves inconsistently — you must align the export and mount options (including idmapd domain) across all nodes. Treat that storage configuration work as a prerequisite, not an optional tuning step.
One detail that would sharpen this: the exact error line from pve-ha-lrm on the target node at migration time. If it shows a lock/permission failure, focus on NFS options and stale locks; if it shows the storage as unavailable, fix the storage definition instead.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 21 Aug 2021, 10:52 UTC
The Proxmox VE HA resource agent does not flock the VM disk image itself; it creates a separate lock file (named after the VM ID) on the shared storage and uses flock() on that file to serialize start/stop actions. If the NFSv4 export cannot honor POSIX locks (e.g., missing lock mount option or inactive rpc.lockd/rpc.statd services), the flock call fails and the VM remains stopped after migration.
You can verify lock support independently of the disk by testing a dummy file on the same share:
flock -n /mnt/pve/nfs-storage/test.lock echo success
A failure indicates the NFSv4 server/client lock subsystem needs attention (enable the lock mount option and ensure rpc.statd and rpc.lockd are running on both client and server).