Persistent Queues in Logstash: Turning Volatile Buffers into Durable Storage

The Volatility of In‑Memory Queues

When Logstash starts, it creates an internal buffer between the input and the filter/output stages. By default this buffer is an in‑memory queue that can hold a limited number of events. If the Logstash process crashes, the host loses power, or a container is killed, every event that is sitting in that memory buffer is lost permanently.

For many log sources a few seconds of missing data is acceptable, but for audit trails, financial logs, or security events that must never be dropped, this volatility is a hard failure.

Persistent Queues: What They Do

A persistent queue changes the buffer from volatile to disk‑backed. Events are written to a directory on the local filesystem before they enter the filter stage. Logstash keeps a checkpoint file that records the position of the last successfully processed event. On a clean shutdown Logstash simply moves the checkpoint forward. On an unclean shutdown it reads the checkpoint on restart and resumes from that point, re‑processing any events that were written to disk but not yet delivered.

Because the queue is written to disk, the data survives process restarts. The trade‑off is that the queue now incurs disk I/O and consumes space on the host.

Configuring a Persistent Queue

Persistent queue settings are defined per‑pipeline, usually in pipelines.yml or a pipeline’s own config file. The following example shows a minimal configuration for a single pipeline named critical_logs:

# pipelines.yml
- pipeline.id: critical_logs
  path.config: /etc/logstash/conf.d/critical.conf
  queue.type: persisted
  queue.max_bytes: 1073741824   # 1 GB maximum
  queue.checkpoint.writes: 200  # checkpoint after every 200 events
  queue.checkpoint.interval: 30   # or every 30 seconds, whichever comes first
  queue.path: /var/lib/logstash/queue

Key points:

• queue.type must be set to persisted (the default is memory). The setting is available in Logstash 7.3+.
• queue.max_bytes limits the on‑disk queue size. When the size is reached, the input blocks until space frees up.
• queue.checkpoint.writes and queue.checkpoint.interval control how often the checkpoint is updated. A lower value reduces the number of events that could be replayed after a crash, but increases disk writes.
• queue.path defaults to data/queue inside the Logstash installation directory. Override it only if you need a dedicated mount point.

What to Expect: Trade‑Offs and Limitations

1. Throughput vs. Latency – Disk writes add latency. A persistent queue can reduce the maximum events‑per‑second compared to an in‑memory queue, especially on spinning disks.

2. Disk Space – The queue can grow up to queue.max_bytes. If your output is slow, the queue will fill and eventually block the input, creating backpressure that can hide downstream problems until the disk is full.

3. At‑Least‑Once Delivery – Because checkpoints are written in batches, an unclean shutdown can cause events to be processed twice. Downstream systems should therefore be idempotent or use stable document IDs.

4. No Protection Against Disk Failure – The persistent queue protects against process loss, not against the underlying storage failing. Store the queue on a reliable volume (RAID, SAN, or a cloud block device with redundancy).

Monitoring the Queue State

Logstash exposes queue metrics through its HTTP monitoring API. By default the API listens on port 9600 and is enabled when logstash.monitoring.enabled: true is set in logstash.yml. The endpoint for pipeline statistics is:

curl -sS http://localhost:9600/_node/stats/pipelines?pretty | jq '.pipelines["critical_logs"].queue'

The JSON output contains:

• type – should be persisted
• events_count – number of events currently queued
• queue_size_in_bytes – size of the on‑disk queue
• queue_max_bytes – configured maximum size

Monitoring these values lets you detect when the queue is growing unexpectedly, which usually indicates that an output is slower than the input.

Practical Test: Verify Replay After a Crash

Below is a safe, illustrative workflow you can run on a test environment:

1. Start Logstash with the persistent queue enabled.
2. Send 1,000 JSON events to the input (e.g., via tcp or http).
3. Immediately kill the Logstash process (e.g., kill -9 $(pgrep logstash)).
4. Restart Logstash.
5. Query the downstream sink (e.g., Elasticsearch) to count the events that arrived. The count should be at least 1,000, but may be higher if duplicates were re‑sent.
6. Check the monitoring API to confirm that queue_size_in_bytes returned to zero after the restart, indicating that the queue was drained.

Because this is a conceptual test, do not rely on the exact numbers. The goal is to observe that events survive the crash and that the queue drains on restart.

Actionable Checklist

• Confirm the Logstash version supports persistent queues (≥ 7.3).
• Set queue.type: persisted in the pipeline that requires durability.
• Choose queue.max_bytes based on available disk space and expected output latency.
• Adjust queue.checkpoint.writes or queue.checkpoint.interval to balance replay risk against I/O overhead.
• Enable the monitoring API and create a simple alert that fires when queue_size_in_bytes approaches queue.max_bytes.
• Verify downstream idempotency or use stable document IDs to mitigate duplicate events.
• Periodically delete the queue directory (or configure Logstash to purge it on startup) after confirming the pipeline is healthy.

Implementing a persistent queue is a straightforward way to add resilience to Logstash pipelines that cannot tolerate data loss. By carefully sizing the queue, monitoring its state, and handling duplicates downstream, you can achieve at‑least‑once delivery without sacrificing too much performance.