Resolving Disk Space and Inode Exhaustion on CentOS

The Problem: System Instability Due to Full Partitions

When a CentOS system reaches 100% disk utilization, critical services often fail to start, databases crash due to an inability to write commit logs, and SSH access may be denied. The most common culprit is the /var/log partition, which can be exhausted either by massive individual log files or by a sheer volume of small files consuming all available inodes.

The primary takeaway: Disk capacity is not just about bytes; it is also about index nodes (inodes). If you see "No space left on device" but df -h shows available gigabytes, you are likely facing inode exhaustion.

Diagnostic Matrix

Symptom	Command to Verify	Probable Cause
Disk usage at 100% (Bytes)	df -h	Large log files or runaway application dumps.
Disk usage low, but "No space left" error	df -i	Inode exhaustion (too many small files).
df shows full, but du shows empty	lsof +L1	Deleted files still held open by active processes.

Step-by-Step Recovery Process

Step 1: Identify the Exhaustion Type

Run these commands as root or via sudo to determine if the issue is capacity or metadata.

# Check human-readable disk space
df -h

# Check inode utilization
df -i

If IUse% is near 100%, you have too many files. If Use% is near 100%, you have files that are too large.

Step 2: Locate the Culprits

If the /var partition is the bottleneck, isolate the specific directory using du (disk usage). The -s flag provides a summary, and -h makes it human-readable.

# Find the largest directories in /var/log
du -sh /var/log/* | sort -rh | head -n 10

If you suspect inode exhaustion, count the files in directories to find the source of the fragmentation:

# Count files in each subdirectory of /var/log
find /var/log -maxdepth 1 -type d -exec sh -c "echo -n '{}: '; ls -1 '{}' | wc -l" \;

Step 3: Apply the Fix Based on Findings

Scenario A: Large Log Files

Do not use rm on a log file that is currently being written to by a service (like httpd or mysqld). Deleting the file removes the directory entry, but the process keeps the file handle open, meaning the space is not reclaimed until the process restarts.

The Fix: Truncate the file. This empties the content without breaking the process handle.

# Truncate a specific log file to zero bytes
> /var/log/large_app_log.log

Scenario B: Inode Exhaustion (Too many small files)

This often happens in /var/spool or session directories. Identify the directory with the highest file count and remove unnecessary temporary files.

# Remove files older than 7 days in a specific temp directory
find /var/log/app_temp/ -type f -mtime +7 -delete

Scenario C: "Ghost" Space (Deleted but Open)

If df shows the disk is full but du cannot find the large files, a process is holding a deleted file open.

# List deleted files still held open by processes
lsof +L1

The Fix: Restart the service associated with the PID found in the lsof output.

# Example: Restarting rsyslog to release handles
systemctl restart rsyslog

Preventing Recurrence with Logrotate

CentOS uses logrotate to manage log growth. If logs are filling up too quickly, your rotation interval or retention count is too high. Edit the configuration in /etc/logrotate.conf or the specific app config in /etc/logrotate.d/.

Example Configuration:

/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
}

In this example, logs are rotated daily, only 7 days of history are kept, and old logs are compressed to save space.

Verification and Limitations

After applying fixes, verify the recovery:

1. Run df -h and df -i to ensure Use% has dropped below 90%.
2. Check that the application is now able to write to its logs by tailing the file: tail -f /var/log/messages.

Limitations: Truncating logs is a temporary fix. If the application is logging at a DEBUG level in production, the disk will fill up again rapidly. You must change the application's logging level to INFO or WARN for a permanent solution.

Rollback Procedure

Because truncating files (> file) and deleting old temp files are destructive operations, they cannot be undone. To prevent accidental data loss, always back up a critical log file to an external mount or a different partition before truncating:

# Backup to a different partition before clearing
cp /var/log/critical.log /mnt/backup_drive/critical.log.bak