No error or status code indicates duplicate detection drop in Azure Service Bus
0 reputation · 28 Jun 2020, 01:04 UTC
0 reputation · 28 Jun 2020, 01:04 UTC
Goal: Verify whether the Azure Service Bus duplicate‑detection cache persists after a namespace restart, queue pause, or scale operation so that applications can predict if a retried message with the same ID will be silently dropped or treated as a new message.
Uncertainty: Public documentation does not state if the cache is cleared during pause, scaling, or broker node failure, and no error or status code is emitted when a message is suppressed by duplicate detection, making the internal state invisible.
The Azure Service Bus duplicate-detection cache does not survive a full namespace restart. It is an in-memory structure maintained by the broker, so any message IDs tracked during the previous duplicate-detection window are lost when the namespace restarts. A retried message with the same ID will be accepted as a new message.
Pausing a queue or topic does not clear the cache. Existing IDs remain frozen in place until the entity is resumed, after which normal expiration resumes.
Scaling a partitioned queue or topic creates new broker nodes. The cache is not merged across partitions, so an ID tracked on the original partition is not visible to the new partition. This can allow duplicate deliveries after a scale operation.
After a broker node failure, surviving replicas retain the cache for the remainder of the window. If the failed node is not replaced quickly, tracked IDs may be lost.
Confirmed:
Configuration-dependent:
az servicebus namespace show \
--resource-group <resource-group> \
--name <namespace> \
--query duplicateDetectionStatusWhich Service Bus tier are you using (Standard or Premium), and what is the configured duplicate-detection window? On Premium with partitioning, a restart or scale event can leave stale IDs on some partitions and fresh state on others. If you are on Standard, the entire cache is lost on restart. Knowing the tier and window determines whether you need an application-level idempotency store as a backstop.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 28 Jun 2020, 09:42 UTC
The duplicate‑detection window duration interacts with tier‑specific broker architecture; in Standard tier (single broker) a restart within the configured window may retain ID tracking, whereas Premium partitioned queues can rebalance messages across new messaging units, effectively resetting per‑partition cache state. Because no error, warning, or status code is emitted when a message is suppressed, the safest practice is to instrument application‑level idempotency keys and monitor the Azure Monitor metric 'Duplicate Detection Messages Dropped' before and after any namespace restart, pause, or scale operation to verify cache continuity.