Diagnosing Logstash Pipelines That Stall With Rising Persistent Queue Depth

Recognizable Condition: Alive but Not Emitting

Logstash appears alive—the process is running and CPU is idle—but events stop reaching the output. The persistent queue depth climbs steadily, and the monitoring API reports events.in greater than zero while events.out stays flat. This pattern usually means the pipeline is blocked somewhere between input and output, not that Logstash has crashed.

The useful takeaway: before restarting or adding workers, compare input and output rates in the monitoring API and scan recent logs for retry or blocking warnings. Restarting a pipeline with a large persistent queue can trigger recovery work that delays throughput further.

Common Causes and Diagnostic Signals

Cause	Diagnostic Signal	First Check
Output backpressure	Elasticsearch bulk rejections, cluster red, repeated output retry logs	Cluster health and output plugin logs
Filter blocking	Input rate normal, output rate zero; filter logs show network waits	Compare events.in vs events.out; inspect filter config
Persistent queue issues	Queue depth grows without bound; startup logs mention page recovery	Disk space and queue file integrity

Ordered Diagnostic Checks

1. Scan Logstash logs for repeated output retry or worker-blocked warnings. On the Logstash host, run:
   

   grep -iE "retry|blocked|warning" /var/log/logstash/logstash-plain.log

   Replace the path with your actual log location. Requires read permission on the log files.
2. Query the monitoring API to compare rates. On the Logstash host, run:
   

   curl -s "http://localhost:9600/_node/stats/pipelines?pretty"

   Replace localhost:9600 if the API is bound elsewhere. Inspect the events and queue sections for the affected pipeline. Expected finding: events.in rising while events.out is flat and queue depth climbs.
3. Verify Elasticsearch cluster health and output bulk settings. A red cluster or repeated bulk rejections will cause backpressure.
4. Inspect the dead letter queue if enabled. A growing dead letter queue indicates events are being dropped after retries.
5. Review recent config or template changes. A new index template or filter plugin update is a common trigger.

Fixes Tied to Specific Findings

Output Backpressure: Bulk Rejections or Cluster Red

If Elasticsearch returns bulk rejections because of a mapping conflict, Logstash retries and the queue grows. A frequent cause is an index template managed by Logstash that conflicts with existing mappings. You can stop Logstash from managing the template and maintain mappings yourself:

output {
  elasticsearch {
    hosts => ["https://your-es-cluster:9200"]
    index => "app-logs-%{+YYYY.MM.dd}"
    manage_template => false
  }
}

Risk: With manage_template => false, Logstash will no longer install or update the index template. You must pre-create the correct mappings in Elasticsearch, or bulk requests will continue to fail. Rollback: revert the setting and restart Logstash if you want template management restored.

Blocking Filters: Network I/O Without Timeouts

DNS, JDBC, or Ruby filters that perform synchronous network calls without timeouts can freeze a pipeline worker. Adding pipeline.workers does not help; it increases contention. The fix is to add explicit timeouts where the plugin supports them, or offload the blocking work to a separate pipeline that communicates through a queue or message bus.

Persistent Queue Problems: Full Disk or Corrupted Pages

If the disk holding the queue is full, Logstash cannot write pages and the queue depth stops updating or grows until writes fail. Check available space on the Logstash host:

df -h /var/lib/logstash

Replace /var/lib/logstash with the queue.path defined in logstash.yml. If the disk is full, either free space or increase queue.max_bytes and restart Logstash.

Critical caution: Do not delete or edit persistent queue files while Logstash is running. Doing so can cause silent data loss or unrecoverable corruption. If corruption is suspected, stop Logstash, back up the entire queue directory, and restart to trigger recovery. Only clear corrupted pages after a backup and with Logstash stopped. Rollback: restore the backed-up queue directory if recovery fails.

Escalation Criteria

• Data loss is confirmed or the dead letter queue is growing.
• Persistent queue corruption is suspected after an unclean shutdown.
• The pipeline remains blocked after targeted fixes for more than one business cycle.
• The root cause shifts to Elasticsearch capacity, requiring index lifecycle management or cluster scaling.

Limitations and Verification

Persistent queue defaults, dead letter queue enablement, and monitoring API response fields vary across major Logstash versions. Validate settings against your exact deployed version before changing production configuration.

Monitoring API metrics can lag behind real-time throughput. Correlate API numbers with log timestamps before declaring recovery. The dead letter queue is local to each Logstash instance and is not automatically reprocessed; plan a separate replay job if you need to recover those events.

To verify a fix, confirm that events.in and events.out converge and queue depth stabilizes or shrinks over several minutes. In a staging environment, you can force an output failure and confirm that pipeline stats show events.out dropping to zero while queue depth rises, reproducing the stall before testing remediation.