Logstash Persistent Queues: Surviving Downstream Outages Without Data Loss

The Problem: Ingestion Outpaces Processing

Firewall logs arrive in bursts. Your Elasticsearch cluster undergoes a rolling restart. Without buffering, Logstash either drops events or blocks the Beats shipper, causing backpressure that ripples to the edge. Persistent queues solve this by writing events to disk before they reach filters and outputs, decoupling ingestion from downstream availability.

How Persistent Queues Work

When queue.type: persisted is set in logstash.yml, every event is appended to segment files under data/queue (default path). The queue acts as a write-ahead log: inputs acknowledge only after the event is fsynced to disk. Filters and outputs then consume from the queue at their own pace. If the output stalls, the queue fills until queue.max_bytes (default 1 GB) is reached, at which point the input pauses — Beats will log blocked connections, providing end-to-end flow control without data loss.

Configuration Walkthrough

Enable persistent queues for a pipeline that ingests firewall logs via Beats and ships to Elasticsearch:

# logstash.yml
queue.type: persisted
queue.max_bytes: 4gb          # bound disk usage to 4 GB
queue.page_capacity: 64mb     # segment size (default)
queue.checkpoint.writes: 1024 # fsync every 1024 writes (default)
path.queue: /var/lib/logstash/queue  # dedicated mount point recommended

Restart Logstash (required; changing queue.type cannot be done hot). After restart, send a test burst:

# On a Beats host
filebeat test output -c filebeat.yml  # verify connectivity
# Generate 500k test events
for i in {1..500000}; do echo '{"msg":"test '$i'"}' | nc localhost 5044; done

Verify queue files appear:

ls -lh /var/lib/logstash/queue/
# Expect .log files (segment data) and .checkpoint files

Confirm backpressure: pause the Elasticsearch output (e.g., systemctl stop elasticsearch on a single node), continue sending events, and watch Beats logs for connection refused or blocked messages. The queue should absorb events until queue.max_bytes.

Trade-offs and Monitoring

• Disk I/O latency: Each event incurs an fsync (typically 1–5 ms). On magnetic disks this can become a bottleneck; use SSDs or a dedicated NVMe mount for path.queue.
• Disk saturation: Sustained load exceeding output throughput will fill the queue. Monitor queue.fill_percent via the Monitoring API:

GET _node/stats/pipelines/main/queue
# Look for "fill_percent" and "events_count"

• No encryption at rest: Queue files are plaintext. For sensitive logs, enable filesystem encryption (LUKS, BitLocker, or cloud volume encryption).
• Restart required: Any change to queue.type or path.queue needs a full Logstash restart. Plan a maintenance window.

When to Enable

Enable persistent queues when:

• Ingestion is bursty (firewall, DNS, proxy logs).
• Downstream systems undergo planned maintenance (rolling restarts, schema changes).
• You cannot tolerate event loss during transient outages.

Start with queue.max_bytes sized to survive your longest expected outage (e.g., 30 minutes of peak throughput). Monitor fill_percent for a week, then adjust. If you see sustained fill >80%, either increase disk allocation or scale the output pipeline.