Diagnosing and Fixing Inode Exhaustion on Unix Filesystems

Recognizable Condition

Applications start failing with errors like "No space left on device" (ENOSPC) even though df -h shows plenty of free blocks. The real clue comes from df -i, which reports 0% or very low free inodes on the affected mount point.

Cause/Diagnostic Table

Possible Cause	Typical Symptom
Excessive small files (mail queues, application caches, /tmp)	High file count, low inode availability
Broken or misbehaving process creating many temporary files	Sudden inode spike after a cron job or service restart
Filesystem quotas limiting inode allocation	Quota tools show inode limit reached despite free space
Log rotation not compressing or deleting old logs	Directories like /var/log filled with many small log files

Ordered Checks

1. Identify the inode‑full filesystem: run df -i on all mounts and note any with IUse% near 100%.

   # df -i

2. Compare total file count to inode usage on the suspect mount (replace /mnt/data with your mount point):

   # find /mnt/data -type f | wc -l   # approximate file count
   # df -i /mnt/data | tail -1 | awk '{print $3}'   # used inodes

3. Locate directories with the highest file counts (requires read access to the tree):

   # find /mnt/data -type d -exec sh -c 'echo "$(find "$1" -type f | wc -l) $1"' _ {} \; | sort -nr | head -n 10
   

4. Review recent logs or cron entries that may have generated many small files (e.g., /var/log/syslog, user crontabs).

Fixes Tied to Findings

• If a specific directory (e.g., /var/spool/clientmqueue) holds millions of queued mail messages, safely remove old entries after confirming they are not needed:

  # find /var/spool/clientmqueue -type f -mtime +7 -delete   # delete files older than 7 days
  

  Permission: run as root or with sudo. Risk: deleting active mail queues can lose messages; verify with -print first.

• For application caches (e.g., /var/www/html/cache), clear the cache according to the application’s documentation or remove files older than a safe threshold.

• Adjust log rotation: edit /etc/logrotate.conf or a service‑specific file in /etc/logrotate.d/ to enable compression (compress) or increase maxage.

• If the filesystem was created with a low inode density and cannot be resized, plan a migration: create a new filesystem with a higher -i ratio (e.g., mkfs.ext4 -i 4096 /dev/sdb1) and copy data using rsync -aHX.

Escalation Criteria

• After cleanup, df -i still shows >95% inode usage.
• Critical services fail to start due to ENOSPC and cannot be worked around by temporary file relocation.
• In such cases, schedule a maintenance window to either recreate the filesystem with a higher inode count or add a new mount point and migrate the affected data.

Verification

1. Run df -i on the cleaned mount; free inode percentage should be comfortably above a safe threshold (e.g., >10%).
2. Monitor for rebound: watch -n 5 df -i /mnt/data for a few minutes.
3. Trigger the previously failing application or job and confirm it no longer reports ENOSPC.
4. Check system logs (/var/log/messages, syslog) for absence of recurring ENOSPC messages.

Limitations and Safety Notes

• Do not delete files in system directories (/bin, /sbin, /etc) without understanding their purpose; removal can break boot or services.
• Always perform a dry‑run with -print before using -delete or rm -rf.
• Increasing inode count on a mounted filesystem is not possible; it requires unmounting and running mkfs or, for ext*, resize2fs only if the underlying block device supports it.