Diagnosing Linux Inode Exhaustion When df Shows Free Space

The condition: ENOSPC with free blocks

An application fails to write with "No space left on device" (ENOSPC), but df -h shows gigabytes free on the filesystem. This mismatch almost always means the filesystem ran out of inodes, not blocks. An inode is the metadata record every file and directory needs; on ext4 the inode table is sized when the filesystem is created (via the bytes-per-inode ratio, typically one inode per 16 KB) and cannot be grown afterward. Millions of tiny files can consume every inode while using a fraction of the disk's capacity.

Confirm the diagnosis in one step:

df -i /var

Run this as any user, substituting the mount point that rejected the write. If IUse% is at or near 100 while df -h /var shows free space, you have inode exhaustion. If IUse% is low, stop here — the cause is something else (reserved blocks, quotas, or on Btrfs, metadata exhaustion) and the fixes below will not help.

Likely causes at a glance

Common inode hoarders: unrotated log directories, session and cache directories, package manager caches, mail spools, container overlay layers, and CI build artifacts. Anything that creates one file per request, session, or build will get there eventually.

Ordered checks

1. Confirm inodes are the problem

Run df -i on the affected mount as shown above. Note the exact filesystem — on servers with separate /var or /tmp mounts, only one may be exhausted.

2. Find which directories hold the files

Count files per top-level directory. This requires read permission on the tree, so run it as root or with sudo:

sudo find /var -xdev -type d -print0 | while IFS= read -r -d '' d; do
  printf '%s\t%s\n' "$(find "$d" -maxdepth 1 -type f | wc -l)" "$d"
done | sort -rn | head -20

The -xdev flag keeps the search on one filesystem so you do not count files on other mounts. On very large trees this takes minutes; a faster first pass is counting only immediate children of a suspected parent, e.g. for d in /var/spool/*; do echo "$(find "$d" | wc -l) $d"; done.

3. Rule out deleted-but-open files

These consume blocks rather than inodes, but they frequently co-occur with logging problems and produce the same ENOSPC symptom:

sudo lsof +L1

Any row with a large SIZE/OFF and a deleted path is a file whose space returns only when the holding process closes it or exits.

Fixes tied to findings

If a cache, session, or build-artifact directory is the offender

Delete or archive the contents, but verify ownership first — blindly running find -delete on a cache directory can break a running application mid-write. Safer pattern: stop the owning service, clear the directory, restart:

sudo systemctl stop myapp
sudo find /var/cache/myapp -xdev -type f -mtime +7 -delete
sudo systemctl start myapp

The -mtime +7 filter removes only files older than seven days, which protects entries the application may still be using. Adjust the path and age to your case; the commands change state, so the rollback is your backup or the application's ability to regenerate its cache — confirm which before deleting.

If logs are the offender

Configure logrotate with a size cap and a bounded retention count rather than date-based rotation alone. A minimal drop-in at /etc/logrotate.d/myapp:

/var/log/myapp/*.log {
    size 50M
    rotate 5
    compress
    missingok
    copytruncate
}

copytruncate truncates the live file in place, which avoids restarting the application but can lose a few log lines during the copy. Test with logrotate -d /etc/logrotate.d/myapp (debug mode, makes no changes) before relying on it.

If deleted-but-open files were found

Restart the holding process identified by lsof +L1 — for example sudo systemctl restart rsyslog. Do not reboot the machine to fix this; a targeted restart releases the same space with less disruption.

If the workload is inherently many-small-files

Cleanup only buys time. The structural fix is to recreate the filesystem with a denser inode ratio — mkfs.ext4 -i 4096 /dev/sdXN creates one inode per 4 KB — or to migrate the data to XFS, which allocates inodes dynamically and has no fixed table to exhaust. Both require a backup, reformat, and restore, so plan downtime. Note that on XFS, df -i reports a dynamic figure and near-100% IUse% is not the same alarm signal.

Verifying the fix

Re-run df -i on the mount and confirm IUse% dropped well below 100. Then retry the exact write that failed — restart the application job or re-run the deploy — and check dmesg | tail for unrelated I/O errors that would indicate a different problem.

To see the failure mode safely in a test environment, create a small loopback filesystem and fill it with empty files:

dd if=/dev/zero of=/tmp/inode-test.img bs=1M count=64
mkfs.ext4 /tmp/inode-test.img
sudo mkdir -p /mnt/inode-test && sudo mount -o loop /tmp/inode-test.img /mnt/inode-test
df -i /mnt/inode-test
sudo sh -c 'cd /mnt/inode-test && i=0; while touch "f$i" 2>/dev/null; do i=$((i+1)); done; echo "created $i files"'
df -h /mnt/inode-test   # blocks remain; df -i shows 100%

Clean up with sudo umount /mnt/inode-test && rm /tmp/inode-test.img. The loop exits when touch fails with ENOSPC, demonstrating free blocks with zero free inodes.

When to escalate

Hand off to a storage or filesystem specialist when: inode exhaustion recurs shortly after a thorough cleanup (suggesting a runaway process creating files faster than you can remove them); the exhausted filesystem is the root filesystem and cannot be unmounted for reformatting without downtime you cannot schedule; or ENOSPC appears alongside fsck errors or I/O errors in dmesg, which points to corruption rather than simple exhaustion. Bring the df -i, df -h, and directory-count outputs to that conversation — they cut the diagnostic loop short.