Logstash Dead-Letter Queue: A Quarantine, Not a Retry Queue

When an Elasticsearch output rejects a document because a field's type changed, Logstash logs an error and moves on. The event is gone. The dead-letter queue (DLQ) is the feature that writes those failed events to disk instead of dropping them silently. The useful takeaway: treat the DLQ as a quarantine and an evidence trail, not as automatic retry or a delivery guarantee.

What the DLQ actually captures

The DLQ is per pipeline. It stores events that fail during pipeline processing—most commonly output failures such as mapping conflicts, rejected documents, or serialization errors. It does not capture events that never entered the pipeline, events removed by a drop filter, or events lost because an upstream shipper could not connect.

Enabling it is a configuration change. In many Logstash 7.x and 8.x installations, you set dead_letter_queue.enable: true in logstash.yml, or per pipeline in pipelines.yml. The exact key and location vary by version and distribution, so confirm against your version's reference before rolling it out. The target directory must be writable by the user that runs Logstash—often logstash. If it is not, the pipeline may fail to start or the DLQ may remain inactive.

A worked example: catching a mapping conflict

Assume a pipeline reads from Beats and writes to Elasticsearch. A field named user is mapped as keyword in the index template, but a new event sends user as an object. Elasticsearch rejects the document. Without a DLQ, the event is lost after the error log.

Enable the DLQ in /etc/logstash/logstash.yml (path and setting names are version-dependent):

dead_letter_queue.enable: true
path.dead_letter_queue: /var/lib/logstash/dlq
dead_letter_queue.max_bytes: 1gb

The max-size setting bounds disk use (for example, dead_letter_queue.max_bytes in recent 7.x/8.x releases; confirm the exact key for your version). When the limit is reached, new failed events are not retained; the DLQ does not automatically spill to another disk or retry. Plan capacity and monitor it.

Check permissions before restarting. Run as a user with read access to the Logstash config and directory metadata:

ls -ld /var/lib/logstash/dlq
namei -l /var/lib/logstash/dlq

The directory should be owned or group-writable by the Logstash service account. Then validate the configuration without starting the pipeline:

sudo -u logstash /usr/share/logstash/bin/logstash \
  --path.settings /etc/logstash \
  --config.test_and_exit

Look for a clean configuration parse and any log line that mentions the dead-letter queue path. A successful parse confirms the setting is recognized; it does not prove the directory is writable at runtime. After restart, send a deliberately failing event in a staging environment—for example, a document that violates the mapping—and check the DLQ directory and the Logstash logs for capacity or permission warnings.

Reading and reprocessing the DLQ

The DLQ is not a human-readable log by default. Use the dead_letter_queue input plugin in a separate pipeline to read and reprocess events. For multi-pipeline deployments, specify the pipeline ID so you read the correct queue.

input {
  dead_letter_queue {
    path => "/var/lib/logstash/dlq"
    pipeline_id => "main"
    commit_offsets => true
  }
}
output {
  stdout { codec => rubydebug }
}

Start with stdout to inspect the shape of failed events, then route them to a corrected index or a repair queue. Do not point the reprocessing pipeline back at the same failing output without fixing the root cause; you will simply refill the DLQ.

Trade-offs and limits

DLQ does	DLQ does not
Retain failed events on disk for inspection	Retry automatically
Bound disk use with a max-size setting	Guarantee delivery after the limit is hit
Provide a per-pipeline quarantine	Cover events lost before the pipeline

Two operational risks matter. First, a misconfigured path can prevent startup or cause silent drops. Second, an unbounded or unmonitored DLQ can fill a disk and affect the whole host. Treat the DLQ as a bounded buffer with an owner and a review process.

What to do next

1. Enable the DLQ in a staging pipeline with a dedicated path and a max-size limit.
2. Validate with --config.test_and_exit and check directory ownership with ls -ld and namei -l.
3. Trigger one known failure and confirm the event appears in the DLQ and a warning appears in the logs if capacity is reached.
4. Build a separate reprocessing pipeline using the dead_letter_queue input, starting with stdout.
5. Monitor DLQ size and disk usage; define what happens when the limit is reached.

To disable the feature, set dead_letter_queue.enable: false and restart. Existing DLQ files remain on disk until you remove them manually; confirm that no reprocessing pipeline still depends on them before cleanup.