Resolving Disk Space Exhaustion in CentOS /var/log Partitions

The Problem: System Instability Due to Full /var/log

When the /var/log partition reaches 100% utilization, CentOS systems experience critical service failures. Common symptoms include the inability to start new services, failures in SSH authentication, and application crashes because the system cannot write to its log files. The immediate goal is to reclaim space without crashing active processes or losing critical forensic data.

Diagnostic Matrix: Identifying the Culprit

Before deleting files, determine whether the space is consumed by traditional text logs or the systemd binary journal.

Symptom	Likely Cause	Diagnostic Tool
Large files ending in .log or .log.1	Logrotate failure or verbose app logging	du -sh /var/log/* | sort -h
Massive /var/log/journal directory	Uncapped systemd-journald storage	journalctl --disk-usage
df shows full, but du shows space	Deleted files held open by active processes	lsof +L1

Step 1: Immediate Space Recovery

If the system is unresponsive, use these methods to reclaim space safely. Avoid using rm on active log files, as the disk space will not be released until the process writing to the file is restarted.

Vacuuming the Systemd Journal

If journalctl --disk-usage reveals high consumption, use the vacuum command. Run these as root or with sudo on the local terminal.

• By Time: Remove logs older than 2 days:
  journalctl --vacuum-time=2d
• By Size: Limit logs to a specific size:
  journalctl --vacuum-size=500M

Truncating Large Text Logs

To clear a text log without breaking the file handle held by a running service, truncate the file to zero bytes:

# Run as root
truncate -s 0 /var/log/large_application.log

Step 2: Implementing Long-Term Limits

To prevent recurrence, you must define hard limits for the systemd journal and verify log rotation schedules.

Capping Journald Storage

Edit the configuration file to ensure the journal does not grow indefinitely. Open /etc/systemd/journald.conf and locate the [Journal] section.

# Example configuration in /etc/systemd/journald.conf
[Journal]
SystemMaxUse=500M
SystemMaxFileSize=50M

Required Action: Apply changes by reloading the daemon and restarting the service:

systemctl daemon-reload
systemctl restart systemd-journald

Verifying Logrotate

Check if logrotate is failing for specific applications. Ensure the configuration in /etc/logrotate.d/ for the offending application includes a rotate count and a compress directive to minimize footprint.

Step 3: Verification and Validation

Confirm the fix is active and the system is stable using these checks:

1. Disk Check: Run df -h /var/log to verify that the percentage used has dropped below 80%.
2. Config Check: Run grep "SystemMaxUse" /etc/systemd/journald.conf to ensure the limit is persisted.
3. Service Check: Run systemctl status systemd-journald to confirm the service is active (running).

Escalation Criteria

If the following conditions occur, the issue is likely not a logging configuration error but a deeper system failure:

• Rapid Refill: Disk space returns to 100% within minutes of vacuuming (indicates a "log loop" or critical application error).
• I/O Wait: High iowait in top despite available space (indicates failing hardware or filesystem corruption).
• Permission Denied: logrotate fails despite correct config (indicates SELinux policy violations or filesystem read-only mode).