Cloud Run Concurrency: Size It With Math, Not Defaults

Ship a small JSON API to Cloud Run, accept every default, and life is easy — until the first real traffic spike, when you're either paying for a fleet of half-idle instances or watching p99 latency climb while new containers boot. The knob most teams never touch is concurrency: how many requests a single container instance handles at the same time. It's the hinge between your traffic shape, your instance count, and your bill.

The thesis of this post: concurrency should be chosen from what your code actually does while handling a request — CPU-bound or I/O-bound — not left at the default. Here's how to reason about it, a worked sizing example, and the failure modes that bite people who raise it blindly.

What the number actually controls

Cloud Run's concurrency setting caps how many requests one instance processes simultaneously. The default has been 80 with a maximum of 1000, but these values have shifted over Cloud Run's history — confirm them in the console or with gcloud before hard-coding anything into your infrastructure.

The autoscaler watches in-flight requests against this capacity: higher concurrency means fewer instances for the same traffic, which means fewer cold starts and less per-instance overhead. Lower concurrency spreads load across more instances, which is what you want when requests contend for CPU.

Concurrency also sets a hard ceiling together with max-instances: the most simultaneous in-flight requests your service can accept is roughly max-instances × concurrency. Push past it and Cloud Run throttles with 429 responses. So the two knobs must be sized as a pair — raising concurrency without checking the ceiling (or lowering it without raising max-instances) is a classic way to discover throttling during an incident instead of a load test.

CPU-bound or I/O-bound decides the right number

A request handler that saturates a core — image resizing, PDF rendering, heavy computation — gains nothing from sharing an instance. Eighty concurrent resizes on one vCPU just queue behind each other, and tail latency explodes. A handler that spends most of its time waiting on a database or a downstream API, by contrast, leaves the instance mostly idle; high concurrency lets one instance absorb a lot of traffic for free.

Concurrency 1 deserves a mention as a pragmatic escape hatch: if your code can't safely handle parallel requests, setting it to 1 sidesteps the problem entirely — at the price of running many more instances.

A worked example: sizing an I/O-bound API

Suppose an orders API peaks at 120 requests/second with a p95 latency of 250 ms, most of it spent waiting on Postgres. A useful approximation (Little's law) is: in-flight requests ≈ arrival rate × latency. That's 120 × 0.25 = 30 in flight on average; plan for spikes of 2–3×, so roughly 60–90 concurrent requests at peak.

Concurrency 120 with max-instances 10 gives a ceiling of 1200 in-flight requests — comfortable headroom, and likely one or two instances doing the steady-state work. The deployment command, run from Cloud Shell or a machine with the gcloud CLI authenticated against your project (you need a role with Cloud Run Admin permission, such as roles/run.admin):

gcloud run deploy orders-api --image=REGION-docker.pkg.dev/PROJECT/REPO/orders-api:TAG --region=europe-west1 --concurrency=120 --max-instances=10 --no-traffic

Substitute your own project, repository, image tag, and region. The --no-traffic flag deploys the revision without sending it production traffic, so you can point a load test at the revision URL first and shift traffic once the numbers look right. To confirm the setting landed, describe the service and check the containerConcurrency field under spec.template.spec — it should read 120:

gcloud run services describe orders-api --region=europe-west1 --format='value(spec.template.spec.containerConcurrency)'

Then load-test with a tool like hey or k6 at realistic peak rates and watch two things in Cloud Monitoring: p95/p99 latency and the instance count. If latency stays flat and instance count stays low, the setting is doing its job. I haven't run this specific configuration, so treat the numbers as a starting hypothesis your load test confirms or refutes.

Two limits that bite in production

Your runtime has to actually multiplex. If the container runs a synchronous single-worker framework, it processes one request at a time no matter how many Cloud Run sends — the extra requests queue inside the container and latency climbs. High concurrency only pays off with a threaded, multi-worker, or async runtime that can genuinely use it.

Memory is per instance, not per request. Eighty concurrent requests each holding 20 MB at peak is 1.6 GB against a default 512 MiB limit — a recipe for OOM kills, which show up as latency spikes and instance churn rather than clean errors. Raise memory limits alongside concurrency and load-test before trusting it.

Billing interacts too: with request-based billing (CPU charged only while a request is being processed), idle concurrency headroom is nearly free. With always-allocated CPU, you pay per instance-second whether requests arrive or not, so over-provisioned concurrency is paid idleness. Comparing both models on a staging service with realistic traffic tells you which one fits your concurrency profile.

Make it a deliberate decision

Treat concurrency as a per-service decision recorded in your deployment config, not an invisible default. Before your next Cloud Run service ships: classify the workload as CPU-bound or I/O-bound, run the in-flight math against your peak traffic, size max-instances so the product covers that peak with headroom, and load-test one revision while watching p95/p99 and instance count. Revisit whenever the workload changes — an endpoint that gains a CPU-heavy code path has silently become a different sizing problem.