Diagnosing and Fixing Logstash Pipeline Stalls: A Practical Backpressure Guide

Problem Overview

When Logstash stops sending events to its output destination while the input plugin continues to receive data, the pipeline is said to be stalled. The most common cause is backpressure—the output plugin cannot keep up with the input rate, filling internal queues and eventually blocking the entire pipeline.

Symptom Checklist

• Input plugin logs show continuous activity, but output logs are silent.
• Persistent queue files grow rapidly and consume disk space.
• CPU usage spikes during filter processing.
• Logstash API /node/stats reports a large number of events “in flight” and “queued”.
• Garbage Collection logs show frequent pauses or “out of memory” errors.

Diagnostic Flow

1. Check the /node/stats endpoint to see current queue depths and worker status.

   curl http://localhost:9600/_node/stats?pretty

2. Inspect logstash-plain.log for circuit breaker or OOM messages.

   grep -i \"circuit breaker\" /var/log/logstash/logstash-plain.log

3. Use top or htop to correlate CPU spikes with filter execution.
4. Verify heap size with jcmd GC.heap_info or jstat -gc .
5. Check disk I/O on the persistent queue directory (/var/lib/logstash/queue) with iostat -xz 1.

Common Causes and Fixes

Cause	Diagnostic Indicator	Fix
Output plugin rate limit exceeded	Large queue depth in /node/stats	Increase output rate (e.g., bulk_max_size for Elasticsearch) or add more output workers.
Persistent Queue I/O bottleneck	High disk latency on queue directory	Move queue to SSD, or disable PQ (queue.type: memory) if data loss is acceptable.
Filter worker saturation	CPU > 80% on filter threads, long pipeline execution times	Increase pipeline.workers up to the number of logical CPUs, or move heavy filters to a separate pipeline.
Insufficient heap allocation	Frequent GC pauses or OOM errors	Allocate more heap (e.g., -Xms4g -Xmx4g) while keeping -Xmn2g to reduce GC overhead.
Excessive worker count	Context switching spikes, low throughput per worker	Limit pipeline.workers to CPU cores - 1.

Example: Tweaking the Persistent Queue

Below is a minimal Logstash configuration that enables a persistent queue on a fast SSD and sets a reasonable queue size.

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { \"message\" => \"%{COMBINEDAPACHELOG}\" }
  }
}

output {
  elasticsearch {
    hosts => [\"localhost:9200\"]
    index => \"weblogs-%{+YYYY.MM.dd}\"
  }
}

# Queue settings
queue.type => \"persisted\"
queue.max_bytes => \"1gb\"
queue.max_events => 100000
queue.checkpoint_interval => 60

Checking the Result

After applying a change, restart Logstash and run the /node/stats query again. The events_in_queue field should drop below the threshold you set (e.g., queue.max_bytes).

curl http://localhost:9600/_node/stats?pretty | jq '.node.stats.queue.events_in_queue'

Also confirm that CPU usage stabilizes and no new OOM messages appear in the log.

Escalation Criteria

• If queue depth remains high after all local fixes, consider scaling horizontally: add more Logstash instances behind a load balancer.
• Persistent OOM errors after heap tuning indicate a memory leak in a plugin; report to the plugin maintainer.
• Repeated I/O bottlenecks on SSD suggest hardware failure; run SMART tests or replace the drive.

Summary

Pipeline stalls are a symptom of backpressure. By systematically checking queue depth, worker saturation, heap health, and disk I/O, you can identify the root cause and apply targeted fixes. Always verify changes with the /node/stats API and logs before scaling or altering production settings.