No Space Left on Device While df Shows Free Space: ENOSPC Diagnostic for ext4 and XFS

The problem you can see in logs, not in df -h

Applications, package managers and systemd report ENOSPC: No space left on device on writes or new file creation. Reads work. df -h shows free gigabytes on the mount. The failure is specific to writes and new inodes.

Takeaway: free blocks is not the only limit. Inodes, reserved blocks for root, user/project quotas and filesystem metadata limits can all trigger ENOSPC while df -h looks healthy.

Recognizable condition

• Write syscalls fail with ENOSPC, e.g. write(2): No space left on device, mkdir: cannot create directory: No space left on device.
• df -h reports free space on the affected mount.
• Reads, listing and existing file updates may succeed. Failures appear on new files, log rotation, package install, or user uploads.

Cause to diagnostic table

Ordered checks

1. Confirm the mount and error scope

Run as root on the host where the mount is visible.

df -h /mountpoint
df -i /mountpoint
findmnt -n -o SOURCE,FSTYPE,TARGET /mountpoint

Expected check: note filesystem type ext4 or xfs and the mountpoint that fails.

2. Ext4 specific checks

Requires root and device path from findmnt.

tune2fs -l /dev/sdXN | egrep 'Block count|Free blocks|Reserved block count|Inode count|Free inodes'

Reserved block count is a percentage of total blocks reserved for root. Non-root writes fail once usable blocks drop below the reserve even though df -h shows free space.

Check open deleted files that keep inodes busy:

lsof +L1 /mountpoint

3. XFS specific checks

xfs_info /mountpoint
xfs_quota -x report -h /mountpoint

xfs_info shows data and metadata usage. xfs_quota report shows user/project limits. XFS does not preallocate inodes; ENOSPC can come from metadata space exhaustion.

4. Quota checks

Quota must be enabled on the filesystem.

quotaon -p /mountpoint
repquota -a /mountpoint
quota -v username

Expected check: usage >= soft or hard limit for the affected UID/project.

Fixes tied to findings

Inode exhaustion

Remove or consolidate small files, move data off the mount, or archive logs. Deleting files does not free inodes immediately if processes hold open file descriptors. Verify with lsof +L1 before assuming cleanup succeeded.

Permanent fix requires recreating the filesystem with a different inode ratio, e.g. mkfs.ext4 -i 16384 /dev/sdXN. This is destructive and requires unmount and a maintenance window.

Reserved blocks on ext4

For non-root data filesystems, lower the reserve:

tune2fs -m 0 /dev/sdXN

Run as root on an unmounted or read-only filesystem for safety. Reducing reserve on root removes safety margin for root recovery. Test in staging first.

Alternative: run the failing workload as root for recovery, then address cleanup.

Quota exhaustion

Raise limits temporarily:

setquota -u username 0 0 0 0 /mountpoint

Or edit with edquota username. Verify with quota -v username and a test write.

XFS metadata or size limits

Grow the filesystem if underlying block device has free space:

xfs_growfs /mountpoint

Requires root. XFS cannot shrink online. Adjust project quota with xfs_quota -x -c 'limit -p bsoft=...' as needed.

Limitations and verification

Do not run e2fsck or xfs_repair on a mounted read-write filesystem; unmount or use live environment to avoid corruption.

After remediation re-run df -h and df -i to confirm free blocks and inode usage below thresholds and attempt a test file creation in the affected directory.

• Validate quota changes with quota -v for the affected user and confirm writes succeed without ENOSPC.
• Monitor dmesg and syslog for 24 hours for recurring ENOSPC or filesystem error messages.

Escalation criteria

• Filesystem remounts read-only.
• Kernel messages about ext4 error, XFS metadata corruption, or I/O errors in dmesg.
• Repeated ENOSPC after cleanup and inode/space checks show no exhaustion.
• Need for offline fsck or xfs_repair.

Escalate to storage team and schedule a maintenance window before destructive operations. Preserve a copy of tune2fs -l, xfs_info, repquota output for post-mortem.