Azure Cosmos DB: Latency spikes during concurrent request bursts
0 reputation · 06 Aug 2021, 18:39 UTC
Throughput Throttling and Latency
Azure Cosmos DB manages resource consumption via Request Units (RUs). When concurrent requests exceed the provisioned RU/s limit, the system returns HTTP 429 (Too Many Requests) responses to protect the database from overloading.
While these responses are expected, the interaction between high concurrency and SDK-level retry policies can introduce significant end-to-end latency. When the SDK automatically retries throttled requests, the resulting queuing behavior may manifest as a latency spike rather than an immediate failure, potentially masking the underlying throughput bottleneck.
This behavior is particularly complex in environments using autoscale throughput, where the RU/s limit fluctuates dynamically up to a defined maximum ceiling.
Design Uncertainty
It is unclear how to distinguish between latency caused by physical resource contention (such as hot partitions) and latency induced by the SDK's internal retry mechanisms during transient bursts.
- How does the SDK retry policy influence the observed p99 latency during concurrent bursts compared to the raw server-side latency?
- Is there a documented threshold where autoscale throughput adjustments cannot keep pace with sudden concurrency spikes, leading to inevitable 429-induced latency?
