Diagnosing Logstash Persistent Queue Disk Saturation and Backpressure

Recognizable Condition

Logstash pipelines suddenly show a sharp drop in events per second. The path.data/queue directory contains growing queue.persistence files that consume all free space on the volume. Logstash logs emit repeated backpressure warnings, and upstream agents (Filebeat, Metricbeat, etc.) report connection reset or timeout errors. Downstream outputs (Elasticsearch, Kafka) appear idle or show high latency.

Cause / Diagnostic Table

Symptom	Likely Cause	What to Verify
Queue files grow unbounded, disk fills	queue.max_bytes unset or larger than free disk	Check path.data/queue size vs. df -h free space
High await or %util on the queue volume	Storage I/O latency bottleneck	Run iostat -x 1 on the device hosting path.data
Downstream output stall (e.g., Elasticsearch indexing throttling)	Output backpressure fills PQ as a buffer	Inspect output plugin stats via Monitoring API
CPU saturated while intake remains high	pipeline.workers exceeds core count, processing can't keep up	Compare pipeline.workers × pipeline.batch.size memory footprint to heap and CPU usage

Ordered Verification Steps

1. Check disk space and I/O on the queue volume

   Run on the Logstash host with root or sudo privileges:

   df -h $(logstash -e 'puts path.data' 2>/dev/null | xargs dirname)

   Then monitor latency:

   iostat -x 1 $(mount | grep 'path.data' | awk '{print $1}')

   Look for await > 5 ms or %util > 80 % as signs of storage pressure.

2. Validate Persistent Queue settings

   Examine logstash.yml (usually /etc/logstash/logstash.yml):

   grep -E 'queue\.type|queue\.max_bytes|queue\.page_capacity|queue\.checkpoint\.writes' /etc/logstash/logstash.yml

   If queue.max_bytes is missing or set to a value larger than the free space minus 10 % headroom, note it.

3. Assess output plugin health

   Enable the Monitoring API (if not already) and fetch pipeline stats:

   curl -s http://localhost:9600/_node/stats/pipelines?pretty

   Check events.out and events.queue_push_duration_in_millis for the affected pipeline. Rising push latency with flat or falling output events indicates downstream backpressure.

4. Review worker and batch sizing vs. CPU and heap

   Calculate theoretical memory use:

   echo $(( $(grep -oP 'pipeline\.batch\.size\s*:\s*\K\d+' /etc/logstash/pipelines/*.yml) * $(grep -oP 'pipeline\.workers\s*:\s*\K\d+' /etc/logstash/pipelines/*.yml) * 1024 ))

   Compare the result (in KiB) to the JVM heap (-Xmx in jvm.options). If the product approaches or exceeds the heap, reduce workers or batch size.

Fixes Tied to Findings

• Storage bottleneck – Move the persistent queue to a dedicated NVMe or SSD with higher IOPS. Update path.queue in logstash.yml to point to the new mount, then restart Logstash. Verify with iostat that await drops below 5 ms under load.
• Unbounded queue growth – Set a sensible upper bound. Example configuration (adjust percentages to your environment):

# logstash.yml
queue.type: persisted
queue.max_bytes: 5gb          # ~50‑70 % of free disk on a 10 GB volume
queue.page_capacity: 64mb
queue.checkpoint.writes: 1024 # default; keep unless durability vs. throughput trade‑off is needed

After editing, validate syntax:

logstash --configtest_and_exit

Then restart the service. Monitor du -sh path.data/queue to confirm size stabilizes near the limit.

• Downstream output backpressure – Resolve the root cause:
• Elasticsearch: check indexing throttling (indices.store.throttle.max_bytes_per_sec) or cluster health; increase shard count or adjust refresh interval.
• Kafka: verify broker lag (kafka-consumer-groups --describe) and increase batch.size or linger.ms in the output plugin.
• Excessive workers/batch size – Reduce pipeline.workers to ≤ number of CPU cores (e.g., pipeline.workers: 4 on a 4‑core host). Optionally lower pipeline.batch.size from the default 125 to 64‑96 if heap is tight. Restart and observe CPU usage drop while throughput stabilizes.

Escalation Criteria

• Persistent queue disk usage exceeds 85 % of the volume for more than 15 minutes despite backpressure being applied to inputs.
• Repeated QueueFullException entries in logstash.log.
• Risk of data loss: if queue.checkpoint.writes is set to 0 (no fsync) and the node crashes, events in the queue may be lost.
• Upstream agents cannot buffer locally (e.g., Filebeat filebeat.registry.file on a full filesystem) and begin dropping events.

When any of these conditions persist, consider:

• Introducing a dedicated buffering layer (Kafka or Redis) upstream of Logstash.
• Scaling out Logstash horizontally with a load‑balancing upstream.
• Increasing the retention or size of the upstream buffer to absorb longer output outages.

Limitations and Practical Verification

The guidance assumes Logstash 7.x–8.x with the default persistent queue implementation (memory‑mapped files). Changing queue.max_bytes requires a full restart; there is no hot‑reload. Setting queue.checkpoint.writes: 1 guarantees durability but can halve throughput on rotational disks—test on a staging node before production rollout.

To verify that a fix is effective:

1. Enable periodic metrics (metrics.periodic: 5 in logstash.yml).
2. Monitor events.queue_push_duration_in_millis and events.queue_pop_duration_in_millis via the Monitoring API; both should stay low (< 10 ms) under load.
3. Simulate an output stall (e.g., pause Elasticsearch indexing with curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"indices.store.throttle.max_bytes_per_sec":"0mb"}}') and confirm that queue growth stops at the configured queue.max_bytes and that backpressure propagates to inputs (rising events.in latency).
4. After changes, run logstash --configtest_and_exit to ensure no syntax errors before restarting.

Rollback: revert the modified logstash.yml lines to their previous values, run the config test, and restart Logstash. Verify that queue size returns to prior behavior.