Persisted Queues and Dead Letter Queues in Logstash

Enabling a persisted queue and dead‑letter queue in Logstash gives crash‑resilient, at‑least‑once pipelines while requiring local disk storage and idempotent outputs.

Decision: Enable Persisted Queues with Dead Letter Queue for Crash Resilience

Logstash ships with an in‑memory queue by default. Events acknowledged by inputs sit in RAM until outputs write them. An abrupt shutdown or crash discards everything in that buffer, breaking at‑least‑once guarantees. The alternative is to switch queue.type to persisted and enable the dead letter queue (dead_letter_queue.enable: true). This combination writes acknowledged events to local disk before acknowledging them, and captures events that outputs reject so they can be inspected and reprocessed.

Constraints and Trade‑offs

Option	Durability	Disk I/O	Operational Overhead	Delivery Semantics
In‑memory queue (default)	Lost on crash	None	Minimal	At‑most‑once
Persisted queue only	Survives restart	Moderate (checkpoint writes)	Monitor `path.queue` space	At‑least‑once
Persisted queue + DLQ	Survives restart; failed events retained	Moderate + DLQ writes	Monitor both directories; manual reprocess	At‑least‑once with visibility into failures

Key Trade‑offs

Disk requirements: path.queue must be on local storage with sufficient IOPS and free space. Network filesystems (NFS, SMB) are not supported because Logstash relies on local disk semantics for queue reliability.
Backpressure: When the persisted queue fills (disk or memory limits), acknowledged inputs such as Beats slow down, protecting Logstash memory.
Duplicates: Persisted queues provide at‑least‑once, not exactly‑once. Downstream outputs must be idempotent or tolerate duplicates after a failure.
DLQ handling: Dead‑lettered events are not automatically returned to the pipeline. An operator must reprocess them via a DLQ input or custom tooling. DLQ disk usage is bounded by dead_letter_queue.max_bytes; when full, the oldest dead‑lettered events are discarded.
Migration: Switching queue.type on a node with unprocessed events may require draining or clearing the queue directory before restart.

Concrete Configuration

Edit logstash.yml (typically /etc/logstash/logstash.yml) and add:

queue.type: persisted
path.queue: "/var/lib/logstash/queue"
queue.max_bytes: 4gb
queue.checkpoint.writes: 1024
queue.checkpoint.interval: 1000

dead_letter_queue.enable: true
dead_letter_queue.path: "/var/lib/logstash/dead_letter_queue"
dead_letter_queue.max_bytes: 1gb

Run this as root or the Logstash service user. Ensure the directories exist and are owned by the Logstash user:

mkdir -p /var/lib/logstash/queue /var/lib/logstash/dead_letter_queue
chown -R logstash:logstash /var/lib/logstash

Verification Steps

Start Logstash and check startup logs for lines confirming persistent queue and DLQ activation:
```
grep -E "Persistent queue|Dead letter queue" /var/log/logstash/logstash-plain.log
```
Confirm queue files exist:
```
ls -la /var/lib/logstash/queue/
```
You should see *.log and *.checkpoint files.
Trigger a DLQ entry by sending an event that an output will reject (e.g., Elasticsearch mapping mismatch). Then verify the DLQ directory appears and contains files:
```
ls -la /var/lib/logstash/dead_letter_queue/
```
Crash‑recovery test: While events are flowing, stop Logstash forcefully (kill -9 <pid>). Restart the service and verify in‑flight events continue processing after recovery. Check logs for "Recovering from checkpoint" messages.
Backpressure observation: Under load, monitor pipeline logs for backpressure warnings and confirm acknowledged inputs (Beats, TCP) slow accordingly.

Monitoring and Alerts

Track queue.used_bytes and dead_letter_queue.used_bytes via the Monitoring APIs or Prometheus exporter.
Alert when queue usage exceeds 70 % of queue.max_bytes or DLQ usage exceeds 70 % of dead_letter_queue.max_bytes.
Alert on disk space for the filesystem holding path.queue and dead_letter_queue.path.

Limitations

Persisted queues do not provide exactly‑once semantics; design downstream systems for idempotency.
DLQ reprocessing is manual; automate with a separate Logstash pipeline reading from the DLQ path if needed.
Network filesystems are unsupported for path.queue; use local NVMe/SSD for production workloads.
A full disk stalls the entire pipeline and blocks acknowledged inputs.

Using Persisted Queues and Dead Letter Queues in Logstash for Crash‑Resilient Pipelines