Diagnosing ENOSPC When df Shows Free Space on Unix

Recognizable condition

A write operation fails with ENOSPC ("No space left on device") while df -h on the same mount point reports ample free space. The symptom is immediate on file creation or write, not an intermittent I/O error, and often only affects non‑root users; root may still succeed.

Cause‑to‑check mapping

Possible cause	What to look for
Inode exhaustion	df -i shows IFree = 0 (or very low) even though blocks remain.
Reserved blocks (ext‑family)	Free space is below the root‑reserved threshold; non‑root gets ENOSPC while root can write.
Deleted‑but‑open file	A process holds an unlinked inode; df counts the blocks but the space is not reclaimable until the file is closed.
Quota limit	Per‑user/group/project quota reports limit reached; df -h still shows free space because quota is independent of filesystem allocation.

Ordered checks

1. Confirm the mount and filesystem type: df -hT /path/to/mount and stat -f /path/to/mount. Note the device, type, and block/inode totals.

2. Check inode usage: df -i /path/to/mount. If IFree is 0, proceed to inode‑specific fixes.

3. Look for deleted‑but‑open files (requires root): lsof -nP +L1 /path/to/mount. Entries with a SIZE/OFFSET column and a flag DEL indicate the problem.

4. Examine quota limits: repquota -a (or the appropriate quota tool for XFS, Btrfs, etc.) for the affected user/group/project.

5. For ext2/3/4, read the reserved‑block setting: tune2fs -l /dev/sdxY | grep -i 'reserved block count' or dumpe2fs -h /dev/sdxY.

Fixes tied to findings

Inode exhaustion

Identify directories with many small files (e.g., /var/cache, /tmp, application caches). Remove or archive unneeded files, or if the workload creates many tiny files permanently, consider recreating the filesystem with a larger inode ratio (mkfs.ext4 -i bytes-per-inode).

Reserved blocks causing ENOSPC for non‑root

If the free space reported by df -h is above zero but below the root reserve, either:

• Consume some of the reserved space by writing as root (temporarily) to lower the free‑space percentage, then retry as the original user; or
• Adjust the reserve percentage with tune2fs -m percent /dev/sdxY after understanding that the reserve protects root from filesystem‑full situations. Reducing it too low may cause the system to become unresponsive when the disk truly fills.

Deleted‑but‑open file

From the lsof output note the PID and command. Signal the process to close the file (e.g., kill -HUP pid or restart the service). After the file descriptor is released, the space becomes available and df -h should show increased free space.

Quota limit

If repquota shows the user has reached their quota, either:

• Increase the quota for that user/group/project with the appropriate quota tool (setquota, xfs_quota, etc.); or
• Clean up files owned by that user under the mount point.

Escalation criteria

Escalate to storage or platform owners when:

• System logs (dmesg, journalctl) show I/O errors, filesystem corruption, or the device going read‑only.
• Freeing space or closing deleted files does not clear the ENOSPC condition.
• The mount point is network‑based (NFS, autofs) and the server side may be full; verify on the server.
• Adjusting reserves or quotas requires coordinated downtime or affects multiple users.

Verification steps

1. As the affected non‑root user, run a test write: dd if=/dev/zero of=/mnt/test/probe bs=1M count=1 conv=fsync. Capture the exit status; ENOSPC appears as exit code 1 with No space left on device in stderr.
2. Repeat the same command as root. If root succeeds while the user fails, the reserved‑block hypothesis is supported.
3. After applying a fix, rerun the test as the original user. Success indicates the condition is resolved.
4. For deleted‑but‑open files, run lsof -nP +L1 /mnt/test before and after the fix; the entry should disappear and free space reported by df -h should increase by roughly the file size.

Limitations and practical checks

The df command reflects the filesystem’s view of allocation; it does not account for per‑user quota, namespace restrictions, or stale automount caches. On network filesystems, df may show cached information; always verify the server’s actual usage when the client reports ENOSPC but df shows free space. Never run fsck on a mounted read‑write filesystem; use the offline repair path or take a snapshot first.