Filebeat Harvester Rotation Issues – Diagnostic Guide

When Filebeat stops capturing new log data after a file rotation, pinpoint the cause quickly with this diagnostic guide. Follow the ordered checks, apply fixes, and know when to elevate the issue.

Recognizable Condition

Filebeat has been ingesting /var/log/app.log successfully, but after the log file is rotated to /var/log/app.log.1 and a new /var/log/app.log is created, Filebeat stops sending events.

Common Causes & Quick Evidence

Cause	Symptom	Evidence in Filebeat Logs
File not found after rotation – old inode retained	No new events after rotation	"File not found" or "Harvester not following"
Incorrect file path or glob pattern	New file outside monitored directory	"Filebeat is not watching" or missing file entry
Permission denied on the new file	Filebeat cannot open the file	"Permission denied" in logs
Harvester pool capped – new file never assigned	Filebeat continues polling old file only	"Harvester pool limit reached"
Inotify watcher limit exceeded	File creation events are missed	"Inotify watch limit reached" or missing watch events
Buffer size exceeded by large rotated file	Harvester stalls or drops data	"Buffer size exceeded" or “harvester buffer full”

Ordered Diagnostic Checks

Check Filebeat Logs
journalctl -u filebeat -n 200 | grep -i "harvester"
Look for entries about missing files, permission errors, or pool limits.
Verify File Path & Glob
Inspect the filebeat.yml section:
```
filebeat.inputs:
  - type: log
    paths:
      - /var/log/app.log
      - /var/log/app.log.*
```
Ensure the rotated file matches the pattern and resides within the watched directory.
Inspect Permissions
stat -c "%a %U %G %n" /var/log/app.log
The new file should have at least read permission for the Filebeat user and be owned by a group that user belongs to.
Check Harvester Pool Size
Filebeat defaults to 500 harvesters. If you have many logs, increase it:
```
harvester_limit: 1000
```
Restart Filebeat after changing.
Verify Inotify Limits
cat /proc/sys/fs/inotify/max_user_watches
If your system has many files, raise it:
```
sysctl -w fs.inotify.max_user_watches=1048576
```
Check Buffer Size
Large rotated files can exhaust the harvester buffer. Adjust:
```
harvester_buffer_size: 64mb
```
or disable harvester_buffer_size if you prefer polling.

Fixes Tied to Findings

Old Inode Retention – Ensure filebeat setup --register is run after rotation or use filebeat -E harvester.inode_tracking=true (default in 7.10+). Restart Filebeat if the file is still not picked up.
Path Mismatch – Update the paths glob to include rotated names or move the rotated file into the watched directory.

Permission Issues – Change ownership or add read permission:

chown filebeat:filebeat /var/log/app.log
chmod 640 /var/log/app.log

Harvester Pool Full – Increase harvester_limit or reduce the number of monitored files.
Inotify Quota Exceeded – Raise max_user_watches or switch to filebeat -E harvester.poll_interval=30s to use polling.
Buffer Exhaustion – Raise harvester_buffer_size or split large logs into smaller chunks before rotation.

Verification Steps

Run Filebeat in debug mode:
```
filebeat -e -d "*"
```
Watch for inotify events for file creation and opening.
After applying a fix, tail the Filebeat log:
```
journalctl -u filebeat -f
```
Confirm that new events appear for the rotated file.
Use stat to compare inode before and after rotation:
```
stat -c "%i" /var/log/app.log
# after rotation
stat -c "%i" /var/log/app.log.1
```
The inode should change; Filebeat should detect this change.

Escalation Criteria

After all local checks, if Filebeat still ignores the new file, consider the following:
- Filebeat version is older than 7.10 – upgrade to benefit from inode tracking.
- Operating system lacks inotify support (e.g., minimal containers) – confirm fallback to polling works.
- Large rotated files consistently exhaust buffer – review log rotation policy or use multiline settings.
Contact Elastic support or file a bug if the issue reproduces on a clean 7.10+ installation with default settings.

Limitations & Practical Tips

Filebeat must run as a user with read access to all rotated files; non‑privileged containers may need --user adjustments.
In minimal containers lacking inotify, Filebeat falls back to polling; consider increasing harvester.poll_interval.
Large log files can cause memory pressure; monitor filebeat.log for buffer warnings and adjust harvester_buffer_size accordingly.
Always test changes in a staging environment before applying to production.

To supplement the diagnostic checks, it is important to consider the close_inactive setting. Even if the inode is correctly tracked, Filebeat may keep a harvester open on the rotated file (e.g., app.log.1) until a specific period of inactivity is reached.

If close_inactive is set to a high value, the harvester for the old file remains active, which can contribute to harvester pool exhaustion if rotations occur frequently. Conversely, setting it too low can cause performance overhead due to constant harvester restarts for slow-writing logs.

Verification Tip

You can verify if a harvester is still clinging to a rotated file by checking the Filebeat Monitoring API or inspecting the registry file (typically in /var/lib/filebeat/registry). Look for the inode of the rotated file to see if its offset has reached the end of the file (EOF) and if it is still marked as active.

Diagnosing Filebeat Harvester Stop‑Reading After Log Rotation