Azure Service Bus ↔ Azure Functions: Choosing Between Duplicate Detection and Custom Idempotency Store

Goal

The objective is to enable robust retry logic between Azure Service Bus and Azure Functions while guaranteeing that each business operation is applied only once, even when transient failures trigger function re‑invocations.

Constraints and Uncertainty

Service Bus duplicate detection can silently drop messages that share a MessageId within a configurable window, but this window is limited (minutes to hours) and fails to protect against duplicates that arrive after the window expires. Azure Functions can be configured with built‑in retry policies, yet each retry re‑enters the function, potentially causing duplicate writes if the handler is not idempotent. Implementing an application‑level idempotency key (e.g., a hash or correlation ID stored in Cosmos DB or Azure Table) adds operational complexity, latency, and cost, yet offers a persistent guard against duplicates across arbitrarily long periods.

Unresolved Decision

Should the solution rely primarily on Service Bus’s duplicate detection (simpler, but limited) or invest in a custom idempotency store (more flexible, but heavier)? The trade‑off involves cost, latency, and reliability under varying message volumes and retry patterns.

Specific Questions

• What Service Bus duplicate detection window size best aligns with typical Azure Function retry intervals to minimize duplicate re‑processing?
• Under what traffic or failure scenarios does a custom idempotency store become necessary to prevent duplicate writes?
• How can we balance the operational overhead of a durable idempotency store against the risk of missed duplicates when relying solely on Service Bus duplicate detection?