Mechanism of Read-Only Transition
The transition of /etc/pve to read-only mode is triggered by a QUORUM_CHANGE event emitted by the Corosync cluster engine. When the number of active votes falls below the required quorum threshold, Corosync signals the Proxmox Cluster File System (pmxcfs), which is a FUSE-based filesystem.
Upon receiving this signal, pmxcfs enforces a read-only state at the FUSE layer. This is a safety mechanism designed to prevent "split-brain" scenarios, where two isolated parts of a cluster might attempt to modify the same configuration files simultaneously, leading to permanent data corruption. This transition happens instantaneously and does not require a kernel-level remount of the underlying disk, as the restriction is handled by the pmxcfs process itself.
Restoring Write Access via pvecm expected
The pvecm expected command allows an administrator to manually override the number of votes required to achieve quorum. This is primarily used during recovery when a majority of nodes are permanently offline.
When you execute pvecm expected 1 (on a single remaining node), the following sequence occurs:
- The command updates the expected quorum value within the Corosync membership state.
- Corosync re-evaluates the current membership and determines that the node now meets the (lowered) quorum requirement.
- Corosync signals pmxcfs that quorum has been restored.
- pmxcfs releases its internal lock and switches the FUSE mount of
/etc/pve back to read-write mode.
Verification and Recovery Steps
To diagnose and resolve a read-only /etc/pve filesystem, use the following scoped commands:
# Check current quorum status and vote count
pvecm status
# Verify if /etc/pve is mounted as read-only (ro)
mount | grep /etc/pve
# Force quorum to 1 vote to restore write access
pvecm expected 1
If the filesystem remains read-only after adjusting the expected votes, a restart of the cluster filesystem service may be required:
systemctl restart pve-cluster
Operational Assumptions
This behavior assumes a standard Proxmox VE installation (version 6.x or 7.x+). Note that using pvecm expected while a network partition actually exists—rather than a total node failure—can lead to split-brain corruption if the other partition also maintains its own quorum. Always verify network connectivity before forcing quorum.
Missing Diagnostic Detail: Are you attempting this recovery on a node that still has network connectivity to other cluster members, or is this a standalone recovery of a degraded site?
proxmox