How should I design MessageId and PartitionKey for Azure Service Bus duplicate detection to safely retry sends without creating duplicate messages in a partitioned queue?

Direct Answer

To guarantee that a resent message is treated as a duplicate and not stored twice, you must: 1) enable duplicate detection on the queue or topic by setting DuplicateDetectionHistoryTimeWindow to a value that covers your maximum retry interval; 2) assign a deterministic MessageId that uniquely identifies the logical business operation (e.g., a UUID or a hash of the payload plus a correlation key); 3) choose a PartitionKey that is independent of MessageId and is derived from a business key or a hash of the payload to spread load across partitions; 4) reuse the same MessageId on every retry while keeping the PartitionKey unchanged if the message must stay in the same partition, or change it only when the business context changes.

Why This Works

• Duplicate detection is global – it looks at MessageId across the entire queue or topic, regardless of partition.
• PartitionKey only affects routing – it does not influence duplicate detection.
• Deterministic MessageId ensures that retries use the same ID so Service Bus can recognize duplicates.
• Independent PartitionKey prevents accidental hot‑partitioning and keeps ordering semantics intact.

Practical Steps

1. Verify queue properties:

   # Azure CLI
   az servicebus queue show \
     --resource-group MyRG \
     --namespace-name MyNS \
     --name MyQueue \
     --query "{duplicateDetectionHistoryTimeWindow:duplicateDetectionHistoryTimeWindow, isPartitioned:partitioningState}"

2. Create a deterministic MessageId in your application code. Example in C#:

   var messageId = Guid.NewGuid().ToString(); // for new messages
   // For retries, reuse the same messageId stored in your database or correlation header.

3. Generate a PartitionKey based on a stable business key (e.g., customerId) or a hash of the payload if ordering is not required:

   string partitionKey = customerId; // or SHA256(payload).Substring(0, 8)

4. Send the message:

   var msg = new ServiceBusMessage(body)
       {
           MessageId = messageId,
           PartitionKey = partitionKey
       };
       await sender.SendMessageAsync(msg);

5. On retry, retrieve the original MessageId from your idempotency store and resend with the same value and the same PartitionKey if the message must stay in the same partition.

Considerations & Edge Cases

• If DuplicateDetectionHistoryTimeWindow is shorter than your retry window, a retry after the window will be treated as a new message. In that case, increase the window or implement an application‑level idempotency key that persists beyond the Service Bus history.
• Never use a mutable value (e.g., timestamp) as MessageId unless you can guarantee it is identical across retries.
• When business context changes (e.g., different customer), generate a new MessageId but keep the same PartitionKey if ordering per partition is required.

Diagnostic Question

What is the current value of DuplicateDetectionHistoryTimeWindow on your queue or topic? Knowing this will confirm whether the window covers your retry strategy.