Question
ZFS Storage Pool Over-provisioning and Dataset Quota Enforcement
Rin Dey
0 reputation · 19 Aug 2022, 01:46 UTC
143K views
Loading…
0 reputation · 19 Aug 2022, 01:46 UTC
Proxmox VE utilizes ZFS as a storage provider, enabling thin provisioning for virtual disks. This allows the allocation of virtual disk space that exceeds the actual physical capacity of the ZFS pool, relying on the fact that data is only written as needed.
While thin provisioning optimizes space, the interaction between guest-level disk reporting and ZFS dataset quotas creates a potential conflict. If a ZFS quota is applied to a specific dataset or if the physical pool reaches capacity, the storage layer may reject write operations even if the guest operating system reports available free space.
There is uncertainty regarding the most reliable method to prevent guest-level filesystem corruption when the underlying ZFS pool is over-provisioned and hits a hard limit.
33775 reputation · 19 Aug 2022, 03:12 UTC
There is no graceful transition. When a ZFS pool or a dataset quota runs out of space, writes fail immediately with ENOSPC, regardless of what the guest OS believes is free. Proxmox VE does not intercept this at the storage layer — the guest's virtual disk simply starts returning I/O errors. The only reliable defense is preventing the pool from ever reaching that state: keep real usage well below capacity, monitor utilization, and alert early.
Thin provisioning means the guest was told it has, say, 500 GB, while the pool may physically hold far less. The guest filesystem has no visibility into pool health, so it happily issues writes that ZFS must reject. Two distinct limits can trigger this:
proxmoxENOSPC instantly, independent of free space elsewhere in the pool.Either failure surfaces in the guest as a generic disk error, which filesystems typically handle by remounting read-only or corrupting in-flight data.
1. Monitor pool usage and alert before the limit. This is the real answer to your second question — the early-warning mechanism is monitoring, not a ZFS setting. Check usage with:
zpool list -o name,size,alloc,free,capacity,fragmentation
zfs list -r -o name,used,avail,quota,reservation,referencedWire capacity into your monitoring (Prometheus node exporter textfile, Zabbix, or a cron job that mails when capacity exceeds a threshold). Alert at 70%, act by 80%.
2. Use reservations and quotas deliberately. A refquota on a VM disk volume caps how much that guest can consume, converting an uncontrolled pool-wide failure into a per-VM, predictable one. A reservation on critical datasets guarantees they always have space. Note that snapshots count toward a quota but not toward a refquota — that distinction matters when deciding which to apply.
3. Manage snapshot growth. On an over-provisioned pool, automatic snapshots are the most common cause of sudden exhaustion: deleted-but-snapshotted data still occupies space. Cap snapshot retention and include snapshot usage in your alerts (zfs list -o usedbysnapshots).
4. Size conservatively. Over-provisioning is only safe if the sum of realistic guest growth stays under your alert threshold. If guests genuinely may fill their disks, the pool must physically hold that data.
You can confirm enforcement semantics on a scratch dataset:
zfs create -o quota=1G tank/testquota
dd if=/dev/zero of=/tank/testquota/fill bs=1M count=2000The dd will stop with "No space left on device" at the quota boundary — the same error a guest VM would receive. Also verify where a quota comes from with zfs get -r quota tank, since inherited quotas are a common surprise.
Exact safe-fill thresholds depend on pool layout (mirror vs. RAIDZ width), record/volblock sizes, and fragmentation, so treat 80% as a conservative rule of thumb rather than a documented hard limit. If you can share your pool layout (zpool status) and whether you use RAIDZ, the threshold and alert levels can be tuned more precisely.
Use comments to ask for clarification. Post a solution as an answer.
33,775 reputation · 19 Aug 2022, 13:04 UTC
In Proxmox, a VM’s virtual disk is a ZFS volume (zvol). The pool only allocates physical blocks when the guest writes. If the pool’s free space drops to zero, ZFS rejects further writes with ENOSPC, which the guest sees as a disk error. A dataset quota behaves the same way – any write that would exceed it is immediately refused.
One practical mitigation is to assign a reservation to the zvol that equals the amount of space you want the VM to be guaranteed. The reservation is carved from the pool at creation, so the VM can continue to write until the reserved amount is used, even if other datasets are filling the pool. Monitor zpool list and zfs get quota,reservation to keep the pool well below 80 % usage; this keeps the reservation from being exhausted prematurely.