Filebeat harvester pauses due to internal queue reaching capacity

Operators want Filebeat to ship log events reliably while avoiding alert fatigue caused by frequent notifications about internal queue backpressure. The goal is to distinguish transient queue fluctuations that are harmless from sustained conditions that could lead to data loss or processing delays.

Filebeat’s memory queue offers low latency but can lose events if the process crashes before flushing, whereas the disk spool persists events across restarts at the cost of higher I/O and potential disk exhaustion when the downstream remains offline. Backpressure automatically pauses the harvester when the queue reaches its configured capacity, which reduces throughput but protects the pipeline.

What queue occupancy threshold or watermark should trigger an alert that signals a genuine risk of data loss without firing on short‑lived spikes?

Is a hybrid configuration—using a small in‑memory buffer backed by a disk spool—recommended to balance performance and durability, and how can its effectiveness be measured in production?