Should a RabbitMQ consumer nack-and-requeue or dead-letter a message when local processing times out?
0 reputation · 24 Nov 2023, 01:45 UTC
0 reputation · 24 Nov 2023, 01:45 UTC
I am designing a RabbitMQ consumer (RabbitMQ 3.12+, .NET and Java clients) that processes messages with manual acknowledgements and a bounded basic.qos prefetch. Each delivery may occasionally exceed a local processing deadline, and I need an explicit, deterministic policy for what the consumer does at that point rather than leaving messages unacked.
The tension I cannot resolve: a nack with requeue fits transient faults (a slow downstream dependency), but if the failure is deterministic the message can loop hot and indefinitely. Rejecting to a dead-letter exchange fits poison messages, but RabbitMQ does not natively count delivery attempts, so a transient timeout could prematurely park a message that would have succeeded on retry. I know the redelivered flag exists and that x-death headers on the DLX can be inspected, but the broker cannot infer my business intent.
Constraints: processing is idempotent, side effects complete before ack, and connection recovery is enabled so duplicates are already possible.
Which criteria should decide between requeue and dead-letter on a local timeout? Is bounding retries via x-death inspection on the DLX the accepted way to cap attempts, or should attempt counting live in application headers? How do I verify the policy behaves correctly when a consumer is killed mid-processing?
31775 reputation · 24 Nov 2023, 05:41 UTC
When a local processing timeout occurs, you should dead-letter the message (basic.nack or basic.reject with requeue=false) rather than requeuing it to the original queue.
The deciding factor is the risk of a "hot loop." Because a timeout is often deterministic—caused by a specific payload size, a complex calculation, or a consistently unresponsive downstream resource—requeuing the message immediately to the head of the queue can saturate your consumer's prefetch buffer. This prevents other valid messages from being processed and spikes CPU/IO without making progress.
Using the x-death header is the standard way to cap attempts without modifying the original message payload. The workflow is as follows:
x-dead-letter-exchange argument on your primary queue.basic.reject(requeue=false).x-death header (which contains a list of deaths, including the count).If you require strict control or cannot use a DLX, maintain a custom header (e.g., x-retry-count). The consumer increments this value and republishes the message to the queue before acknowledging the original. This is more deterministic than x-death but increases broker load due to the explicit publish/ack cycle.
If a consumer is killed while processing a message (before it sends an ack, nack, or reject), RabbitMQ detects the connection loss. Because the message was never acknowledged, the broker automatically requeues it. This is a broker-level safety mechanism and is independent of your timeout policy.
x-death header to ensure the count increments correctly after each rejection.kill -9 the consumer process, and verify the message returns to the Ready state in the primary queue.Diagnostic Note: Does your downstream dependency exhibit "cascading failure" (where retries make the outage worse)? If so, you must implement an exponential backoff in your retry consumer rather than immediate requeueing.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.