Azure Service Bus ↔ Azure Functions: Choosing Between Duplicate Detection and Custom Idempotency Store

Answer the Decision First

Use Service Bus duplicate detection when your function’s retry interval is shorter than the duplicate‑detection history window (typically 5‑10 minutes) and you can guarantee that the MessageId uniquely identifies the business operation. Switch to a custom idempotency store when you need a longer deduplication period, deduplication on a business key rather than MessageId, or want to guard against duplicates that arrive after the Service Bus window expires.

Why this matters

• Duplicate detection is free and requires only a queue/topic setting.
• It only discards messages that share the same MessageId within the configured window.
• A custom store adds latency (network round‑trip) and cost (storage, read/write ops) but gives full control over the deduplication window and key semantics.

1. Aligning the Duplicate‑Detection Window with Function Retries

Azure Functions’ built‑in retry policy (e.g., maxRetryCount and intervalInSeconds) usually keeps retries within a few minutes. Service Bus duplicate detection can be configured up to 5 minutes by default, and up to 24 hours for premium tiers. In practice:

• If your retry interval is ≤ 5 minutes, set DuplicateDetectionHistoryTimeWindow to 5 minutes. This covers most transient failures.
• For workloads with longer retry windows (e.g., 15–30 minutes), either increase the history window (if your tier supports it) or move to a custom store.
• Always set a MessageId that is stable across retries; otherwise duplicate detection will not trigger.

2. When a Custom Idempotency Store Becomes Necessary

Consider a custom store in these scenarios:

• Extended deduplication period: You need to prevent re‑processing for hours or days (e.g., a user‑initiated workflow that may retry after a system outage).
• Business‑key deduplication: The MessageId is not unique enough. For example, idempotency on a transaction ID that can be reused across different messages.
• Cross‑queue deduplication: The same operation may arrive on multiple queues or topics; a shared store is required.
• Compliance or audit requirements: You must log every processed key for audit trails, which Service Bus does not provide.

3. Balancing Operational Overhead and Risk

To keep overhead low while protecting against missed duplicates, follow this hybrid approach:

1. Enable duplicate detection on all queues/topics. Set the history window to the maximum you can tolerate for typical retry patterns.
2. For critical or long‑lived operations, add a lightweight idempotency check in the function body:
   • Use Azure Table Storage or Cosmos DB with a single write per key.
   • Employ an upsert or conditional insert to avoid race conditions.
   • Log the key in a separate audit table if required.
3. Monitor DuplicateDetectionHistorySize via Service Bus Explorer to confirm entries expire as expected.
4. Periodically audit the idempotency store for stale keys and purge them if they are no longer needed.

Cost comparison (per 1 M operations, 2026 pricing):

Approach	Storage Cost	Latency (avg)
Service Bus Duplicate Detection	None	Negligible
Azure Table Store Idempotency	$0.10 / GB‑month	~20 ms
Cosmos DB (RU‑based)	$0.08 /RU‑month	~15 ms

4. Quick Implementation Steps

1. Enable duplicate detection:

   az servicebus queue update \
     --name my-queue \
     --resource-group rg \
     --duplicate-detection-history-time-window 00:05:00 \
     --enable-duplicate-detection true
   

2. Set MessageId in producer code (e.g., Node.js):

   const msg = new ServiceBusMessage({
     body: payload,
     messageId: transactionId // stable across retries
   });
   await sender.sendMessages(msg);
   

3. Optional idempotency check in function (C#):

   public async Task RunAsync(ServiceBusReceivedMessage message)
   {
     var key = message.MessageId;
     var exists = await tableClient.ExistsAsync(key);
     if (exists) return; // duplicate
     await tableClient.AddAsync(new IdempotencyEntity(key, DateTime.UtcNow));
     // process business logic
   }
   

5. Diagnostic Detail Needed

To refine the recommendation, please share:

• Maximum expected retry interval (seconds or minutes).
• Estimated message volume per hour.
• Whether the same business operation can appear on multiple queues or topics.

These details will help decide if the 5‑minute duplicate detection window suffices or if a custom store is required.