Filebeat harvester pauses due to internal queue reaching capacity

Question

Operators want Filebeat to ship log events reliably while avoiding alert fatigue caused by frequent notifications about internal queue backpressure. The goal is to distinguish transient queue fluctuations that are harmless from sustained conditions that could lead to data loss or processing delays.

Filebeat’s memory queue offers low latency but can lose events if the process crashes before flushing, whereas the disk spool persists events across restarts at the cost of higher I/O and potential disk exhaustion when the downstream remains offline. Backpressure automatically pauses the harvester when the queue reaches its configured capacity, which reduces throughput but protects the pipeline.

What queue occupancy threshold or watermark should trigger an alert that signals a genuine risk of data loss without firing on short‑lived spikes?

Is a hybrid configuration—using a small in‑memory buffer backed by a disk spool—recommended to balance performance and durability, and how can its effectiveness be measured in production?

Siddharth · Answer

Direct AnswerFor the memory queue, alert on sustained occupancy above 70–75% for 5+ minutes. Short spikes to 90%+ lasting under 30 seconds are normal burst behavior and should not trigger alerts. For the disk queue, alert when disk usage exceeds 70% of queue.disk.max_size sustained for 10+ minutes, or when write_ahead flush latency exceeds 500 ms at p99.No Built-In Hybrid Queue ExistsFilebeat offers two mutually exclusive queue types: queue.mem (in-memory) and queue.disk (on-disk spool). There is no single pipeline that simultaneously buffers in memory and persists to disk. You must choose one at startup.Recommended Alerting ThresholdsQueue TypeWarningCriticalSmoothingMemory (queue.mem)≥70% occupancy sustained 5 min≥85% occupancy sustained 2 minPrometheus avg_over_time (5m window)Disk (queue.disk)≥70% of max_size sustained 10 min≥85% of max_size sustained 5 minPrometheus avg_over_time (10m window)Harvester pauses occur at 100% capacity (hard limit). Backpressure appears in logs as "harvester paused" messages and increments the filebeat.harvester.paused counter.Measuring Effectiveness in ProductionEnable monitoring: monitoring.enabled: true and ship metrics to Prometheus or Elasticsearch.Track occupancy ratio over time:Memory: filebeat.queue.mem.events / filebeat.queue.mem.capacityDisk: filebeat.queue.disk.used_bytes / queue.disk.max_sizeCorrelate with output latency (filebeat.output.events.acked latency) and harvester pause duration metrics.Simulate downstream outage (stop Logstash/Elasticsearch) to observe fill rate and pause behavior.Verify disk queue persistence: kill Filebeat during ingestion and confirm events deliver after restart.One Diagnostic Detail NeededWhich queue type are you currently running (queue.mem or queue.disk)? The exact threshold and verification steps differ between them.

Filebeat harvester pauses due to internal queue reaching capacity

1 answer

Direct Answer

No Built-In Hybrid Queue Exists

Recommended Alerting Thresholds

Measuring Effectiveness in Production

One Diagnostic Detail Needed

Your perspective belongs here.

0 question comments

Queue Type	Warning	Critical	Smoothing
Memory (`queue.mem`)	≥70% occupancy sustained 5 min	≥85% occupancy sustained 2 min	Prometheus `avg_over_time` (5m window)
Disk (`queue.disk`)	≥70% of `max_size sustained 10 min`	≥85% of `max_size sustained 5 min`	Prometheus `avg_over_time` (10m window)