HTTP 429 Request Rate Too Large during Cosmos DB write retries

Direct Answer

429 responses never indicate a committed write. When Cosmos DB returns HTTP 429 (Request Rate Too Large), the request exceeded the provisioned RU/s and the server did not apply the mutation. The SDK's automatic retry of a 429 is therefore inherently safe—it cannot create duplicates or overwrite state from that same request.

The duplicate-write risk comes from timeouts or network failures (408, 503, connection reset), where the client cannot know whether the server committed before the response was lost. Retrying such a write is safe only if the operation is idempotent.

How the SDK Handles 429 Retries

• The SDK reads the x-ms-retry-after-ms header and waits that duration before retrying.
• Default policy: up to 9 retries with a cumulative wait of 30 seconds (configurable via MaxRetryAttemptsOnRateLimitedRequests and MaxRetryWaitTimeOnRateLimitedRequests in .NET v3/v4, Java v4, etc.).
• The retried request carries the same payload and activity ID; no application logic is required.

Idempotency with Upsert + If-Match

Upsert with a stable, client-generated id is idempotent for creates. Retrying the same upsert (same id, same payload) converges to a single document.

Adding If-Match (ETag) breaks that idempotency by design. If the first attempt actually succeeded but the client saw a failure (timeout, not 429), the document's ETag has changed. The SDK's retry sends the stale ETag, and the server correctly returns 412 Precondition Failed. This is optimistic concurrency working as intended—it detects the earlier success.

Recommended Pattern for High-Throughput Writes

1. Use deterministic, operation-scoped IDs (e.g., "order-12345-item-7") so any retry path collapses to one server-side effect.
2. Prefer plain Upsert (no If-Match) for idempotent creates/updates.
3. If you need optimistic concurrency, handle 412 after a retry by re-reading the document, comparing content/ETag, and deciding whether to re-apply or treat as success.
4. Do not disable 429 retries or set retry-after to zero; aggressive immediate retries worsen throttling.

Verification Steps

• Enable SDK diagnostics/logging and confirm 429 responses include x-ms-retry-after-ms and that retried requests carry the same activity ID and payload.
• Force a client-side timeout after a committed upsert, then retry with the same ID—verify exactly one document exists with the expected content.
• Test If-Match retry: commit a replace, re-issue the same request with the stale ETag, confirm 412 and that the document reflects the first write.
• Inspect CosmosDiagnostics to count retries and confirm whether a failed request reached the server (status vs. substatus and request charge).

One Diagnostic Detail Needed

Which SDK language and version are you using (.NET v3/v4, Java v4, Node, Python)? Default retry limits and option names differ across generations, and the exact configuration surface changes the tuning steps.