Logstash Dead Letter Queues: Catching Failed Events Before They Vanish

The Silent Data Loss Problem

Logstash pipelines process millions of events daily, but when an output rejects a document—malformed JSON, schema mismatch, or a downstream service returning 4xx—the event often disappears. By default, Logstash logs a warning and moves on. No retry, no archive, no alert. The Dead Letter Queue (DLQ) changes that: it writes every discarded event to a local file so you can inspect, fix, and replay without losing data.

How the DLQ Works

Enable the DLQ per pipeline by adding a dead_letter_queue path in logstash.yml or the pipeline-specific pipelines.yml. When any output or filter calls event.cancel (or the Elasticsearch output receives a non-retryable HTTP error), Logstash serializes the full event—metadata, fields, and the error context—into a numbered file under that directory. Each pipeline gets its own subdirectory, so multiple pipelines on the same host stay isolated.

Worked Example: Elasticsearch Mapping Rejection

Imagine an Elasticsearch output that rejects documents because a timestamp field arrived as a string instead of a date. The pipeline configuration:

output {
  elasticsearch {
    hosts => ["https://es-cluster:9200"]
    index => "logs-%{+YYYY.MM.dd}"
    dlq_policy => "all"          # send every failure to DLQ
  }
}

In logstash.yml:

dead_letter_queue.enable: true
dead_letter_queue.path: "/var/lib/logstash/dead_letter_queue"
# optional rotation
dead_letter_queue.max_bytes: 1073741824   # 1 GB per segment
dead_letter_queue.flush_interval: 5000    # ms

After a mapping error, you’ll see files like /var/lib/logstash/dead_letter_queue/main/0.log. Each line is a JSON object containing the original event plus [@metadata][dlq][error] with the Elasticsearch response.

Inspecting and Replaying

Use the bundled CLI to list pending events without stopping Logstash:

# run on the Logstash host as the logstash user
/usr/share/logstash/bin/logstash --dead-letter-queue-path /var/lib/logstash/dead_letter_queue/main --dead-letter-queue-list

Output shows event count, first/last timestamps, and the error message. To replay after fixing the mapping:

/usr/share/logstash/bin/logstash --dead-letter-queue-path /var/lib/logstash/dead_letter_queue/main --dead-letter-queue-replay

Replay reads events in order and feeds them back into the same pipeline inputs. If you’ve corrected the Elasticsearch template, they index cleanly.

Trade-offs and Operational Limits

• Disk growth: DLQ segments rotate at max_bytes but never auto-delete. You must archive or prune old segments (e.g., a daily cron that moves files older than 7 days to cold storage).
• Worker races: With pipeline.workers > 1, multiple threads write to the same DLQ file. Logstash uses append-only writes with OS-level atomicity, but extremely high throughput can cause contention. Consider a dedicated DLQ disk or reducing workers for pipelines with frequent failures.
• No built-in alerting: The DLQ is passive. Pair it with a file-watcher (Filebeat tailing the DLQ directory) that ships a summary to your monitoring stack so you know when the queue grows.

Verify It’s Working

1. Send a deliberately malformed event (e.g., curl -XPOST localhost:5044 -d '{"timestamp":"not-a-date"}').
2. Check /var/lib/logstash/dead_letter_queue/main/ for a new segment file.
3. Run the --dead-letter-queue-list command and confirm the error matches the Elasticsearch mapping rejection.
4. Fix the mapping, replay, and verify the document appears in the target index.

If the segment appears and replay succeeds, the DLQ is operational. Schedule the archive job and add the Filebeat watcher before you consider the pipeline production-ready.