Dead Letter Queue Thresholds: Defining Alert Triggers

Short answer

There is no universally correct DLQ depth threshold — RabbitMQ deliberately leaves this to operators. A defensible starting point for a typical production workload is max(100 messages, 1% of peak hourly throughput) as a warning, with a critical alert at 3× that value, plus a separate alert on the oldest message age. Then tune over 2–4 weeks based on your observed false-positive rate. The threshold is a policy decision about your error budget, not a number the broker can give you.

Why a single static number is not enough

DLQ depth alone conflates three different situations: a sudden burst of failures, a slow leak of poison messages, and a stalled operations process that never drains the queue. Each deserves a different response, so a good alerting setup combines three signals:

• Absolute depth — catches accumulation. Warning at your baseline threshold, critical at 3×.
• Rate of change — catches spikes before they hit the depth limit. For example, alert if depth increases by more than 50 messages/minute sustained for 5 minutes.
• Oldest message age — catches neglect. If the oldest message is older than 24h, the problem is the response process, not the application. Page on this independently.

Static vs dynamic thresholds

For stable, predictable traffic, a static threshold is simpler and easier to reason about — start there. Dynamic thresholds (for example, alerting when depth exceeds 3 standard deviations above a 1-hour moving average, using PromQL functions like stddev_over_time) reduce noise for workloads with strong daily or weekly cycles, but they have two requirements and one real risk:

• You need at least about 7 days of history before the baseline is meaningful.
• The PromQL is more complex and harder for on-call engineers to interpret at 3 a.m.
• Risk: a moving average can mask gradual degradation — a steady 10 msg/min leak slowly raises the baseline and never alerts. Always pair a dynamic threshold with an absolute floor.

Retention policy

Set a TTL directly on the DLQ with the x-message-ttl argument — 7 to 14 days is a common range — and add x-max-length to cap disk and memory usage. Two caveats:

• Expired messages are dropped silently. If you need an audit trail, configure a dead-letter exchange on the DLQ itself so expired messages route to an archive queue instead of vanishing.
• TTL is applied per-message at enqueue time; changing it requires redeclaring the queue, so plan a maintenance window.

Also monitor queue_message_bytes alongside depth: 10,000 tiny messages may matter less than 100 large ones filling the disk.

Verification

1. Enable the RabbitMQ Prometheus plugin and confirm rabbitmq_queue_messages{queue="my-dlq"} appears in your Prometheus targets.
2. Build a Grafana panel showing depth, rate of increase, oldest message age, and message bytes over 24h.
3. Simulate a burst (publish ~200 messages to a failing consumer) and confirm the alert fires within expected latency. Note that a default 15s scrape interval can miss micro-spikes.
4. Cross-check Prometheus against the broker:

   rabbitmqctl list_queues name messages messages_ready messages_unacked message_bytes

5. Review alert history monthly. Count false positives and false negatives; a false-positive rate under ~10% is a reasonable target. Revisit the threshold quarterly or when traffic shifts by more than ~20%.

Assumptions and open points

This assumes classic queues and Prometheus/Grafana scraping. If you use quorum queues, dead-lettering and TTL enforcement behave differently — test in staging before relying on these numbers. One detail that would change the recommendation: your peak hourly message throughput. If you share it, the 1%-of-throughput baseline can be replaced with a concrete number for your workload. Finally, depth-based alerts cannot distinguish retryable failures from permanent poison messages; consider a retry-count header and a separate "retry exhausted" queue so alerts stay actionable.