Logstash Dead Letter Queue: How to Stop Losing Events When Outputs Fail

When a Logstash output plugin can't deliver an event — Elasticsearch is down, an HTTP endpoint returns an error, a document fails mapping — the default behavior for many failure modes is to drop the event or retry forever. The Dead Letter Queue (DLQ) gives you a third option: persist the failed event to disk so you can reprocess it later instead of losing it silently.

What the DLQ actually does

The DLQ is a built-in Logstash feature that captures events an output plugin has given up on and writes them to local files. Each event is stored with its original payload plus metadata about why it failed (the plugin that rejected it and a reason). Nothing is reprocessed automatically — the DLQ is a holding area, and you decide when and how to drain it.

Two important scope notes:

• The DLQ only captures events the output plugin explicitly rejects as unrecoverable. It does not help with events lost because Logstash itself crashed mid-pipeline (that's what persistent queues are for) or with transient errors the output keeps retrying.
• Support varies by output plugin. The Elasticsearch output has the most complete DLQ integration (for example, HTTP 400/404 mapping errors). Check your specific output plugin's documentation before assuming coverage.

Enabling it: a worked configuration

DLQ is enabled globally in logstash.yml, not per pipeline:

# logstash.yml
dead_letter_queue.enable: true
dead_letter_queue.path: /var/lib/logstash/dlq
dead_letter_queue.max_bytes: 2048mb

What each setting means:

• dead_letter_queue.enable — turns the feature on. Default is off.
• dead_letter_queue.path — directory where DLQ segment files are written. Put this on persistent, monitored storage — not /tmp or a container's ephemeral layer.
• dead_letter_queue.max_bytes — cap on total DLQ size. When the cap is reached, Logstash stops writing new dead-lettered events (and those events are dropped), so size this generously and alert well before you hit it.

Your pipeline config needs no DLQ-specific changes. A normal Elasticsearch output works as before:

output {
  elasticsearch {
    hosts => ["http://es-node-1:9200"]
    index => "app-logs-%{+YYYY.MM.dd}"
  }
}

When Elasticsearch rejects a document (say, a mapping conflict returning HTTP 400), Logstash writes that event into a .log segment file under the DLQ path instead of dropping it.

Reprocessing dead-lettered events

Draining the DLQ requires a second pipeline using the dead_letter_queue input plugin. Run it on demand, after you've fixed the root cause:

input {
  dead_letter_queue {
    path => "/var/lib/logstash/dlq"
    commit_offsets => true
    pipeline_id => "main"
  }
}

output {
  elasticsearch {
    hosts => ["http://es-node-1:9200"]
    index => "app-logs-recovered-%{+YYYY.MM.dd}"
  }
}

Notes on this:

• commit_offsets => true lets the input track what it has already read, so a restart doesn't replay everything. Set it to false if you want to replay the full queue each run.
• pipeline_id filters to events dead-lettered by a specific pipeline. Without it, you read everything in the directory.
• Consider routing recovered events to a separate index first (as above) so you can inspect them before merging into the main data stream — the original failure may have been a data problem that will simply fail again.

Operational limits and common mistakes

Disk is the hard limit. If the downstream system stays broken, the DLQ grows until it hits max_bytes or fills the disk. A full disk can stall or crash the whole Logstash process, affecting healthy pipelines too. Monitor the DLQ directory size and alert on growth, not just on the cap.

Enabling without draining. The most common mistake: teams turn on the DLQ, feel safe, and never build the reprocessing pipeline or an operational runbook. The DLQ then becomes a write-only graveyard. Decide upfront who drains it, how often, and what "done" looks like.

Volatile storage. Pointing the DLQ at /tmp, a tmpfs mount, or un-persisted container storage means a reboot silently deletes your safety net. Use a persistent volume.

Assuming all failures are captured. Only non-retryable, plugin-reported rejections land in the DLQ. Network timeouts are usually retried by the output instead. Test with your actual output plugin rather than assuming behavior.

Reprocessing without fixing the cause. If events failed because of malformed data, replaying them unchanged just refills the DLQ. Inspect a sample of dead-lettered events first; add a filter in the recovery pipeline to repair or drop known-bad records.

Verifying it works

You can validate the setup end to end without waiting for a real outage:

1. Enable the DLQ and restart Logstash. Confirm the DLQ directory is created.
2. Force a rejection — for example, send a document that violates an Elasticsearch index mapping, or temporarily point the output at an index with a strict mapping that rejects your test event.
3. Check the DLQ path for new segment files and confirm the Logstash logs mention dead-letter writes.
4. Fix the cause, run the recovery pipeline with the dead_letter_queue input, and confirm the events arrive at the output.
5. Watch disk usage on the DLQ path during the failure window to confirm consumption matches your sizing expectations.

Version note: DLQ behavior and per-plugin support have evolved across Logstash 6.x through 8.x, so confirm the settings and plugin coverage against the documentation for your exact version before rolling this into production.