ZFS ARC cap configuration on Proxmox VE with undefined VM memory headroom

Question

Proxmox VE hosts using ZFS for storage rely on the Adaptive Replacement Cache for read caching. By default ARC can consume a large share of host RAM, which creates tension with memory allocated to VMs and containers. The goal is to select an ARC limit that preserves guest memory availability while retaining ZFS read performance.

There is no single documented correct cap. The appropriate value depends on workload read patterns, total RAM, and acceptable hit ratio. ARC pressure is observable via kernel statistics, but translating those observations into a safe persistent limit involves a workload-specific trade-off. Default sizing behavior also varies with the OpenZFS version shipped with Proxmox VE releases.

What measurement indicates ARC is the primary memory bottleneck for VMs rather than another subsystem? How should an acceptable ARC hit ratio be balanced against guaranteed VM memory headroom? What is the documented permission boundary for changing the ARC maximum on a production Proxmox VE host?

Tasadduq Burney · Answer

1.  When is ARC the real memory bottleneck?
Use arcstat and /proc/spl/kstat/zfs/arcstats to see the current ARC size, hit and miss counts.  The key indicators are:

Host free memory (or free -m) is low (e.g. < 10 % of total RAM).
ARC size is a large fraction of RAM (e.g. > 30 % of total).
Guest VMs are not swapping or ballooning.  If free -m shows guest swap usage, the bottleneck is the guest over‑commit, not ARC.
Host swap I/O is minimal.  High swap I/O while ARC is large suggests ARC is competing with guests.

If all three conditions hold, ARC is the primary memory pressure source for your VMs.

2.  Balancing hit‑ratio against headroom
The ARC hit‑ratio is available in arcstat as hit vs miss.  A high ratio (e.g. > 95 %) means most reads are served from RAM, but it also means the ARC consumes more memory.
Practical guidance for Proxmox hosts:

Start with a cap of 25–50 % of total RAM if you need a lot of guest memory.  For example, on a 64 GB host set zfs_arc_max to 16 GB (0.25 × 64 GB).
After applying the cap, monitor the hit‑ratio.  If it drops below ~90 % and your disks are SSD/NVMe, the performance impact is usually acceptable.  On spinning disks, keep the ratio higher.
Adjust the cap in 10 % increments, re‑check arcstat and guest performance, and iterate until you hit the sweet spot.

3.  Safely changing the ARC limit on a production Proxmox VE host

Check the current value: cat /sys/module/zfs/parameters/zfs_arc_max.  The output is in bytes.
Set the new cap in a persistent file:
echo "options zfs zfs_arc_max=17179869184" | sudo tee /etc/modprobe.d/zfs.conf

Replace 17179869184 with the desired byte value.
Rebuild the initramfs so the change survives reboot:
sudo update-initramfs -u

Reboot the node (recommended in a maintenance window): sudo reboot.
After reboot, verify the new cap:
cat /sys/module/zfs/parameters/zfs_arc_max

Re‑run arcstat to confirm the hit‑ratio and free memory behaviour.

Runtime adjustment without reboot is possible on many OpenZFS versions via /sys/module/zfs/parameters/zfs_arc_max, but the change will be lost on reboot unless the modprobe.d file is updated.

4.  One diagnostic detail that might change the recommendation
Do your VMs currently use host swap or ballooning?  If they are swapping or ballooning heavily, the host memory pressure is coming from the guests, not ARC, and you should consider increasing guest RAM or reducing over‑commit rather than tightening the ARC cap.

ZFS ARC cap configuration on Proxmox VE with undefined VM memory headroom

1 answer

1. When is ARC the real memory bottleneck?

2. Balancing hit‑ratio against headroom

3. Safely changing the ARC limit on a production Proxmox VE host

4. One diagnostic detail that might change the recommendation

Your perspective belongs here.

0 question comments