Diagnosing and Resolving Azure Cosmos DB Provisioned Throughput Throttling (HTTP 429)

Recognizable condition

Applications using Azure Cosmos DB SDK start seeing intermittent 429 Too Many Requests responses, accompanied by rising p99 latency and increased retry logs. Azure Monitor shows the container’s Normalized RU consumption hovering near 100 % with sharp spikes, while the overall request rate stays roughly constant. A subset of operations reports a high request charge per operation.

Cause‑diagnostic mapping

Observable signal	Likely cause
Normalized RU ≈ 100 % with spikes, 429 count rising	Provisioned RUs insufficient for sustained load
High request charge on specific operations, uneven partition key distribution	Hot partition key causing localized RU exhaustion
Large item size or high index cost per read	Inflated request charge per operation
Queries without partition key or with non‑sargable filters	Fan‑out across partitions consuming extra RUs
Burst traffic exceeding provisioned RUs and SDK retry budget	Short‑lived traffic spikes

Ordered checks

1. Inspect core metrics – In Azure Monitor, query the AzureDiagnostics table for the container:

   AzureDiagnostics
   | where Category == "DataPlaneRequests"
   | summarize avg_NormalizedRU = avg(NormalizedRUConsumptionPercentage),
               total_429 = count_if(StatusCode == 429),
               total_RU = sum(RequestCharge)
               by bin(TimeGenerated, 5m), ContainerName
   

   Run this in the Azure Portal Log Analytics workspace (requires Log Analytics Reader role). Look for periods where avg_NormalizedRU ≥ 95 % and total_429 > 0.
2. Review SDK logs – Enable diagnostic logging (e.g., CosmosTraceLevel.Info) and capture RequestCharge and retry count per operation. Correlate high‑charge entries with the partition key value logged by the SDK.
3. Sample operations via Query Metrics – In Data Explorer, open the container, select Query Metrics, run a representative query and note the PartitionKeyRangeId distribution and Request Charge. Repeat for suspect queries to spot a single partition key consuming a disproportionate share of RUs.
4. Check item size and index metrics – In Data Explorer, enable Item Size column and view the size histogram. Open Metrics → Index to see average index size per item; large values indicate costly indexing.
5. Validate client retry policy – Verify the SDK’s ConnectionPolicy.RetryOptions (MaxRetryAttempts, MaxRetryWaitTime) and connection mode (Direct vs Gateway). Ensure max concurrency settings do not exceed the provisioned RU budget when retries are enabled.

Fixes tied to findings

• Sustained load exceeds provisioned RUs – Increase throughput or enable autoscale. Example Azure CLI (run in a terminal with az login and Microsoft.DocumentDB/databaseAccounts/sqlContainers/write permission):

  az cosmosdb sql container throughput update \
    --resource-group  \
    --account-name  \
    --name  \
    --container-name  \
    --throughput 5000
  

  After the change, monitor NormalizedRUConsumptionPercentage; it should drop below 80 % for the observed traffic window. Rollback: decrease the throughput back to the previous value using the same command with the original --throughput figure if cost becomes prohibitive.
• Hot partition key – Redesign the key to increase cardinality (e.g., append a synthetic suffix or use a composite key). Avoid monotonic keys like timestamps. If immediate change is not possible, consider splitting the logical container into two containers with a range‑based key and using a fan‑out write pattern.
• Large items or expensive queries – Reduce item size by removing unnecessary properties or compressing large strings. Add covering indexes for frequent query paths (IndexingPolicy.IncludedPaths). Rewrite queries to include the partition key in the WHERE clause and use sargable filters (=, >, <, AND) to avoid cross‑partition scans.
• Burst traffic – Enable SDK server‑side retries (ConnectionPolicy.RetryOptions.MaxRetryAttempts) and implement client‑side exponential backoff with jitter. For bulk writes, use the BulkExecutor library to batch operations and reduce per‑operation overhead.

Escalation criteria

• Throttling persists above 5 % of requests after throughput increase and retry tuning, or NormalizedRUConsumptionPercentage remains high on a single logical partition indicating a data‑model hotspot that requires redesign.
• Cross‑partition queries are business‑critical and cannot be rewritten to be partition‑key targeted, or latency SLO breaches continue despite autoscale and client changes, signalling the need for an architecture review (e.g., sharding, migration to a different Cosmos DB API, or introducing a caching layer).

Verification

1. After applying a fix, check Azure Monitor for the 429 Substatus metric; it should return to baseline (near zero) for at least one full traffic cycle (e.g., one hour of typical load).
2. Enable SDK diagnostic logging and confirm that the average RequestCharge per operation has decreased and retry counts have dropped.
3. Use Data Explorer to run Query Metrics on the previously suspect queries and verify that the request charge per execution is now within expected bounds.
4. Monitor NormalizedRUConsumptionPercentage and p99 latency for sustained improvement; if they remain elevated, revisit the checks.