Diagnosing Filebeat Harvester Stop‑Reading After Log Rotation

Recognizable Condition

Filebeat has been ingesting /var/log/app.log successfully, but after the log file is rotated to /var/log/app.log.1 and a new /var/log/app.log is created, Filebeat stops sending events.

Common Causes & Quick Evidence

Cause	Symptom	Evidence in Filebeat Logs
File not found after rotation – old inode retained	No new events after rotation	"File not found" or "Harvester not following"
Incorrect file path or glob pattern	New file outside monitored directory	"Filebeat is not watching" or missing file entry
Permission denied on the new file	Filebeat cannot open the file	"Permission denied" in logs
Harvester pool capped – new file never assigned	Filebeat continues polling old file only	"Harvester pool limit reached"
Inotify watcher limit exceeded	File creation events are missed	"Inotify watch limit reached" or missing watch events
Buffer size exceeded by large rotated file	Harvester stalls or drops data	"Buffer size exceeded" or “harvester buffer full”

Ordered Diagnostic Checks

1. Check Filebeat Logs
   journalctl -u filebeat -n 200 | grep -i "harvester"
   Look for entries about missing files, permission errors, or pool limits.
2. Verify File Path & Glob
   Inspect the filebeat.yml section:

   filebeat.inputs:
     - type: log
       paths:
         - /var/log/app.log
         - /var/log/app.log.*
   

   Ensure the rotated file matches the pattern and resides within the watched directory.
3. Inspect Permissions
   stat -c "%a %U %G %n" /var/log/app.log
   The new file should have at least read permission for the Filebeat user and be owned by a group that user belongs to.
4. Check Harvester Pool Size
   Filebeat defaults to 500 harvesters. If you have many logs, increase it:

   harvester_limit: 1000

   Restart Filebeat after changing.
5. Verify Inotify Limits
   cat /proc/sys/fs/inotify/max_user_watches
   If your system has many files, raise it:

   sysctl -w fs.inotify.max_user_watches=1048576

6. Check Buffer Size
   Large rotated files can exhaust the harvester buffer. Adjust:

   harvester_buffer_size: 64mb

   or disable harvester_buffer_size if you prefer polling.

Fixes Tied to Findings

• Old Inode Retention – Ensure filebeat setup --register is run after rotation or use filebeat -E harvester.inode_tracking=true (default in 7.10+). Restart Filebeat if the file is still not picked up.
• Path Mismatch – Update the paths glob to include rotated names or move the rotated file into the watched directory.
• Permission Issues – Change ownership or add read permission:

  chown filebeat:filebeat /var/log/app.log
  chmod 640 /var/log/app.log

• Harvester Pool Full – Increase harvester_limit or reduce the number of monitored files.
• Inotify Quota Exceeded – Raise max_user_watches or switch to filebeat -E harvester.poll_interval=30s to use polling.
• Buffer Exhaustion – Raise harvester_buffer_size or split large logs into smaller chunks before rotation.

Verification Steps

1. Run Filebeat in debug mode:

   filebeat -e -d "*"

   Watch for inotify events for file creation and opening.
2. After applying a fix, tail the Filebeat log:

   journalctl -u filebeat -f

   Confirm that new events appear for the rotated file.
3. Use stat to compare inode before and after rotation:

   stat -c "%i" /var/log/app.log
   # after rotation
   stat -c "%i" /var/log/app.log.1
   

   The inode should change; Filebeat should detect this change.

Escalation Criteria

• After all local checks, if Filebeat still ignores the new file, consider the following:
  • Filebeat version is older than 7.10 – upgrade to benefit from inode tracking.
  • Operating system lacks inotify support (e.g., minimal containers) – confirm fallback to polling works.
  • Large rotated files consistently exhaust buffer – review log rotation policy or use multiline settings.
• Contact Elastic support or file a bug if the issue reproduces on a clean 7.10+ installation with default settings.

Limitations & Practical Tips

• Filebeat must run as a user with read access to all rotated files; non‑privileged containers may need --user adjustments.
• In minimal containers lacking inotify, Filebeat falls back to polling; consider increasing harvester.poll_interval.
• Large log files can cause memory pressure; monitor filebeat.log for buffer warnings and adjust harvester_buffer_size accordingly.
• Always test changes in a staging environment before applying to production.