Optimizing Filebeat Harvester Settings to Prevent Log Loss and File Descriptor Exhaustion

Why Filebeat Drops Log Lines When Your Application Writes Infrequently

When a service logs only a few lines per hour, Filebeat may stop reading the file after a short silence period, causing missing events in Elasticsearch. This blog shows how to choose the right harvester settings to keep logs flowing without exhausting file descriptors.

How the Harvester Works

Filebeat creates a single harvester for each open log file. The harvester holds a file descriptor until one of the close events occurs: close_inactive (silence timeout), close_renamed, close_removed, or close_eof. While the descriptor is open, the harvester reads new lines and sends them to the output queue.

close_inactive vs ignore_older

close_inactive stops harvesting after the configured period of no file modification. If the application appends data later, Filebeat must reopen the file, which can generate duplicate events if the old harvester re‑reads the same lines.

ignore_older tells Filebeat to skip files whose last modification time is older than the threshold. This is useful for rotating logs that stay unchanged for days, but it may also skip delayed writes from slow‑running applications that still need to be indexed for audit purposes.

Limiting Open File Descriptors with harvester_limit

Each harvester consumes a file descriptor. On hosts that tail many log files, the default limit can be reached, causing new files to wait in a queue and potentially miss data. Setting harvester_limit caps the number of concurrent harvesters; excess files are opened only when a slot frees. Adjust this value based on the expected number of simultaneous log files and the OS limit on open files.

Worked Example: Tuning for a Low‑Traffic Service

Run the configuration in a test directory and verify with:

1. `filebeat -c /etc/filebeat/filebeat.yml -d "harvester"` – starts Filebeat with debug output showing open/close events. The Beat user must have read permission on the log paths (e.g., be in the adm group or the files must be world‑readable).
2. `filebeat test config` – validates syntax and Elasticsearch connectivity.
3. `filebeat test output` – checks that events are shipped without error.

After deployment, compare the number of Filebeat events with the line count in the source log during a known rotation window to ensure no lines are lost.

Limitations and Verification

Setting close_inactive too low can cause Filebeat to reopen rotated files, leading to duplicate events. Setting it too high may keep descriptors open longer, increasing the risk of descriptor exhaustion. ignore_older may skip logs from applications that write rarely but are required for compliance; monitor the “ignored files” metric in Stack Monitoring.

To verify the effect of harvester_limit, generate load with a script that opens many log files simultaneously and watch the Beat’s stats endpoint or Kibana’s “Harvester” metric. A rising queue size indicates the limit is being hit.

Actionable Checklist

• Identify the typical inactivity period of your log files.
• Set close_inactive to a value that balances duplicate risk and descriptor usage.
• Use ignore_older only when you are sure delayed writes are irrelevant.
• Adjust harvester_limit to stay below the OS‑level open‑file limit (often 1024 or 65535).
• Run `filebeat -d "harvester"` in a staging environment and review open/close logs.
• Validate with `filebeat test config` and `filebeat test output` before rolling out.