Preventing Data Loss with the Logstash Dead Letter Queue

The Silent Data Drop

In a high-volume logging pipeline, a single malformed event—like a truncated JSON string or a field that exceeds a destination's size limit—can cause an event to be dropped. By default, when Logstash cannot deliver an event to an output plugin after repeated retries, that data is gone. For teams handling compliance or financial data, this "silent drop" is unacceptable.

The solution is the Dead Letter Queue (DLQ). The DLQ acts as a safety net, capturing events that fail to be processed or delivered and storing them on disk. This allows you to inspect the failure, fix the underlying configuration or data issue, and replay the events without losing information.

Enabling the DLQ

The DLQ is not active by default. It must be enabled in the logstash.yml configuration file. Because it involves disk I/O, you should define limits to prevent a flood of bad data from consuming your entire filesystem.

Add the following settings to your logstash.yml:

dead_letter_queue.enable: true
# Limit the DLQ size to 1GB to prevent disk exhaustion
dead_letter_queue.max_bytes: 1073741824
# Optional: Specify a custom path if the default path.data is on a small partition
# dead_letter_queue.path: "/var/lib/logstash/dlq"

After updating the configuration, restart the Logstash service. You can verify the feature is active by checking for the existence of the dead_letter_queue directory within your Logstash data path.

Handling a Failure: A Worked Example

Consider a scenario where you are using a stdin input and an Elasticsearch output, but a specific event is too large for the Elasticsearch mapping, causing a 400 Bad Request error.

Step 1: Triggering the DLQ

Run Logstash with the DLQ enabled. Send a malformed or oversized event that you know will trigger an output failure. In the logs, you will see an entry indicating that the event has been written to the DLQ:

# Expected log pattern
INFO [logstash.outputs.elasticsearch] Event failed to be indexed... writing to dead letter queue

Step 2: Inspecting the Failure

The DLQ stores events as JSON lines on disk. You can use cat or tail on the files in the path.data/dead_letter_queue directory to identify why the event failed. This allows you to determine if you need to update your Elasticsearch mapping or add a Logstash filter to truncate the offending field.

Step 3: Replaying the Data

Once the destination is fixed, you can recover the data using the dead_letter_queue input plugin. It is recommended to do this in a separate, temporary pipeline to avoid mixing live data with recovered data.

input {
  dead_letter_queue {
    path => "/var/lib/logstash/data/dead_letter_queue"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "recovered-events"
  }
}

Trade-offs and Operational Risks

While the DLQ prevents data loss, it introduces specific engineering overhead:

• Disk I/O: Writing failed events to disk adds latency and I/O pressure. On systems with slow disks, a high volume of failures can degrade overall pipeline performance.
• The Infinite Loop: If you replay events from the DLQ into the same pipeline that caused the failure—without fixing the root cause—the events will simply fail again and be written back to the DLQ.
• Monitoring Gap: The DLQ is a "passive" storage. If you do not monitor the size of the dead_letter_queue directory or track the logs for DLQ write events, you may not realize you are losing data until the max_bytes limit is hit and new failures start being dropped.

Verification and Cleanup

To verify your DLQ setup is working, inject a deliberately broken event (e.g., a string that violates a strict schema in your output) and confirm the file size of the DLQ directory increases.

Rollback: To disable the DLQ, set dead_letter_queue.enable: false in logstash.yml and restart Logstash. Note that disabling the feature does not automatically delete existing DLQ files; you must manually remove the dead_letter_queue directory if you wish to reclaim disk space.