Does increasing Cloud Run max concurrency reduce latency under concurrent load?
0 reputation · 10 Mar 2025, 19:23 UTC
Goal
Determine whether increasing Cloud Run maximum request concurrency reduces tail latency when many requests arrive at the same time, or whether it mainly moves queuing from Cloud Run scale-out into the container instance.
Constraints and uncertainty
Cloud Run fully managed routes multiple concurrent requests to one container instance up to the configured maximum concurrency, defaulting to 80. A single-threaded runtime or blocking event-loop task can cause head-of-line blocking inside that instance, while a higher limit may delay scale-out and let one slow request share an instance with others. Cloud Run documentation does not specify a deterministic latency model combining maximum concurrency, CPU throttling during scale-down, and idle timeout, so the optimal value depends on the application threading model and whether requests are CPU- or I/O-bound.
Assuming Cloud Run fully managed (1st generation) with a configurable concurrency limit above the default 80:
- Does raising max concurrency above 80 consistently reduce p99 latency under concurrent load?
- When does in-instance queuing outweigh the latency added by provisioning additional instances?
- What concurrency limit best matches a single-threaded application that cannot process requests in parallel?