Azure Cosmos DB: Latency spikes during concurrent request bursts

Question

Throughput Throttling and Latency
Azure Cosmos DB manages resource consumption via Request Units (RUs). When concurrent requests exceed the provisioned RU/s limit, the system returns HTTP 429 (Too Many Requests) responses to protect the database from overloading.
While these responses are expected, the interaction between high concurrency and SDK-level retry policies can introduce significant end-to-end latency. When the SDK automatically retries throttled requests, the resulting queuing behavior may manifest as a latency spike rather than an immediate failure, potentially masking the underlying throughput bottleneck.
This behavior is particularly complex in environments using autoscale throughput, where the RU/s limit fluctuates dynamically up to a defined maximum ceiling.
Design Uncertainty
It is unclear how to distinguish between latency caused by physical resource contention (such as hot partitions) and latency induced by the SDK's internal retry mechanisms during transient bursts.

How does the SDK retry policy influence the observed p99 latency during concurrent bursts compared to the raw server-side latency?
  Is there a documented threshold where autoscale throughput adjustments cannot keep pace with sudden concurrency spikes, leading to inevitable 429-induced latency?

Zoha · Answer

Direct answer
Your end-to-end p99 during concurrent bursts is usually not the raw server-side latency. When demand exceeds provisioned or autoscale RU/s, Cosmos DB returns HTTP 429. The SDK then retries with backoff, so client-observed p99 can include queued retries and multiple round trips even when server-side latency remains normal. There is no universal documented threshold where autoscale inevitably produces 429-induced latency; the boundary depends on your maximum RU/s, per-request cost, concurrency, and partition distribution.
Likely explanation vs confirmed facts
Likely explanation: a retry storm. High concurrency produces many 429s; each retry adds load and delay, and the slowest requests are retried longest, inflating p99. Confirmed: Cosmos DB enforces RU/s per container or database and returns 429 when exceeded. Autoscale adjusts only within the configured maximum ceiling and cannot serve sustained instantaneous demand above it.
Steps for this case

Reproduce a controlled burst while capturing SDK diagnostics: record start/end time, HTTP status, request charge, retry count, and time spent in backoff for each operation.
For the same window in Azure Monitor, compare server-side latency with request counts by status code (especially 429) and RU consumption. High 429s plus low server latency point to retry amplification.
Check partition-key-range statistics, such as normalized RU consumption and request volume, to see whether throttling is uniform or concentrated on a hot partition.
Re-run with client concurrency capped and retry backoff unchanged. If p99 drops sharply, the burst amplified retries rather than server contention.

The diagnostic that changes the recommendation
The single detail that changes the recommendation is the latency split between SDK retry/backoff wait and server processing. If most p99 time is client-side retry wait, tune concurrency/retry and reduce request cost. If most time is server-side and skewed toward one partition-key range, adjust capacity or partition design instead of retry settings.
Caveats

SDK retry defaults and backoff behavior are version- and language-specific; do not generalize one client's settings to all clients.
Raising RU/s or autoscale max can mask hot-partition skew and increase cost; verify partition distribution before scaling.
Autoscale reacts to consumption but is not a queueing guarantee for sudden bursts; it does not remove the need for client-side concurrency limits.

Because SDK defaults and autoscale behavior evolve, verify against your current SDK version and Azure Monitor data before changing production settings.

Azure Cosmos DB: Latency spikes during concurrent request bursts

Throughput Throttling and Latency

Design Uncertainty

1 answer

Direct answer

Likely explanation vs confirmed facts

Steps for this case

The diagnostic that changes the recommendation

Caveats

Your perspective belongs here.

0 question comments