Using Logstash Dead Letter Queue to Rescue Failed Events

When a filter blows up, your data doesn’t have to

Imagine a Logstash pipeline that parses application logs with a grok filter. One day a new log format appears, the grok pattern fails, and the filter throws an exception. Without a safety net, those events disappear from your output and you lose visibility into the problem. Logstash’s Dead Letter Queue (DLQ) provides that safety net: it writes the original event to disk before the pipeline aborts, letting you inspect the failure and replay the event once the issue is fixed.

Enabling the DLQ

Add the following settings to logstash.yml (or to a specific pipeline file if you use pipeline‑specific config):

dead_letter_queue.enable => true
# Optional: limit how much disk the DLQ can consume
dead_letter_queue.max_bytes => 1024mb
# Default directory is path.data/dead_letter_queue
# You can override it with:
dead_letter_queue.path => "/var/log/logstash/dlq"

After saving the file, restart Logstash. The process will create the directory if it does not exist and begin writing any event that causes a non‑recoverable error to a .log file inside it.

What gets written to the DLQ?

When a filter or output plugin returns an error, Logstash serializes the original event (as it arrived at the pipeline input) to a JSON line and adds a _dlq_ metadata object. The object contains at least:

• _dlq_.reason – the exception message or error string
• _dlq_.timestamp – when the event was queued
• _dlq_.pipeline – the pipeline ID that produced the failure

All original fields (including @timestamp from the source) are preserved, but any modifications made by filters that succeeded before the failure are not retained. When you replay the event, the full pipeline runs again from the start.

Worked example: catching a failing ruby filter

1. Create a test pipeline (test-dlq.conf) that intentionally fails:

   input {
     generator {
       lines => [
         '{"message":"good event"}',
         '{"message":"bad event"}'
       ]
       count => 2
     }
   }
   filter {
     if [message] =~ /bad/ {
       ruby { code => "raise 'intentional failure for bad event'" }
     }
   }
   output {
     stdout { codec => rubydebug }
   }
   

2. Start Logstash with DLQ enabled (using the config from the previous section) and point it at the test pipeline:

   bin/logstash -f test-dlq.conf --path.settings ./config
   

   You will see the good event printed to stdout, while the bad event triggers the ruby exception and disappears from the output.

3. Check the DLQ directory (default ./data/dead_letter_queue) for a newly created .log file. Inspect a line:

   cat ./data/dead_letter_queue/log.1663587200000.log | head -1
   

   You should see JSON similar to:

   {"message":"bad event","@timestamp":"2026-09-19T22:13:20.000Z","_dlq_":{"reason":"intentional failure for bad event","timestamp":1663587200000,"pipeline":"main"}}

4. Fix the pipeline by removing or correcting the ruby filter, then restart Logstash with the DLQ input plugin to replay the queued events:

   input {
     dead_letter_queue {
       path => "./data/dead_letter_queue"
       # commit_offsets => true   # optional, saves progress
     }
   }
   filter {
     # no ruby filter – events will pass through cleanly
   }
   output {
     stdout { codec => rubydebug }
   }
   

   Run:

   bin/logstash -f replay-dlq.conf --path.settings ./config
   

   The previously failed "bad event" should now appear in the stdout output, demonstrating successful recovery.

Trade‑offs and operational considerations

While the DLQ prevents data loss, it introduces operational overhead:

• Disk usage – a misbehaving plugin can fill the queue quickly. Setting dead_letter_queue.max_bytes provides a hard ceiling; once the limit is reached, Logstash will stop accepting new events and begin logging warnings. Monitor the directory size (du -sh path.data/dead_letter_queue) and set up alerts.
• Ordering – DLQ files are written per‑batch and per‑pipeline; the dead_letter_queue input does not guarantee global ordering across multiple files. If strict ordering is required, you must re‑sequence events downstream (e.g., using a sort buffer or application‑level logic).
• Reprocessing semantics – because only the original event is stored, any enrichment that happened before the failure is lost. If you rely on intermediate fields, you must either adjust the pipeline to be idempotent or store those fields elsewhere before they might be lost.

Actionable next steps

1. Enable the DLQ in your production logstash.yml with a sensible max_bytes (e.g., 10‑20 % of your expected daily ingest volume).
2. Set up a simple health check that logs the current DLQ size every hour and triggers an alert if it exceeds 80 % of the configured limit.
3. Document a run‑book for the replay process: stop the problematic pipeline, fix the offending filter/output, start a temporary pipeline with the dead_letter_queue input, and verify that events reappear in the expected output.
4. After a successful replay, consider cleaning up the processed DLQ files (Logstash does not auto‑delete them) to reclaim disk space.

By treating the DLQ as a configurable safety net rather than an afterthought, you gain visibility into processing failures and a reliable path to recover lost data without redesigning your entire logging architecture.