Cloud Run Concurrency: Picking the Right Number of Requests Per Instance

If your Cloud Run service gets slow under bursty traffic — or your bill looks bigger than the traffic justifies — the knob you probably haven't touched is request concurrency: how many simultaneous HTTP requests a single container instance is allowed to handle. It's one of the highest-leverage settings on the platform, and the default is rarely the right answer for every workload.

The thesis is simple: concurrency is a trade between cost efficiency and latency stability, and the right value depends on whether your requests are CPU-bound, I/O-bound, or thread-unsafe. Here's how to reason about it.

What concurrency actually controls

Cloud Run's autoscaler adds instances based on concurrent request load. If your concurrency limit is 80 and 160 requests arrive at once, the platform targets roughly two instances. If your limit is 1, those same 160 requests target 160 instances (subject to your max-instances cap).

This is a fundamentally different model from one-request-per-instance platforms. A single Cloud Run instance is a real container running your server, and it can juggle many requests at once — as long as your runtime and your code can handle it.

One caveat before numbers: Cloud Run's default and maximum concurrency values have changed across releases and vary by runtime. Historically the default has been high (up to 1000 for newer runtimes), but confirm the current default for your language runtime in the documentation before relying on any specific figure.

The failure modes at both extremes

Too high: requests queue on busy instances while the autoscaler spins up new ones. You see this as p95/p99 latency spikes during traffic bursts even though average CPU looks fine. Worse, if each request holds significant memory, high concurrency can push instances past their memory limit — and out-of-memory kills surface as intermittent 500s, not a clear error message.

Too low: the autoscaler creates far more instances than you need. Each instance bills for its lifetime, so an I/O-bound API that spends 90% of its time waiting on a downstream database will cost dramatically more at concurrency 1 than at concurrency 80, with no latency benefit to show for it.

There's also a correctness case: if your framework or code isn't thread-safe — a common pattern is a Python/Flask app mutating module-level global state per request — high concurrency isn't just slow, it's wrong. Setting concurrency to 1 is a legitimate engineering decision here: you trade instance count and cost for guaranteed isolation.

A worked example: two services, opposite answers

Consider two services on the same project:

• Image resizing API. Each request saturates roughly one vCPU for ~300ms. With concurrency 80, eighty requests would timeshare one vCPU and each would take many seconds. The right setting is 1–4, matching the vCPU allocation, so the autoscaler adds instances instead of queueing.
• API proxy. Each request spends ~200ms waiting on a downstream service and does almost no local CPU work. This service can comfortably run 80+ concurrent requests per instance; lowering concurrency just multiplies the bill.

Set it per revision at deploy time:

gcloud run deploy image-resizer \
  --image REGION-docker.pkg.dev/PROJECT/REPO/image-resizer:TAG \
  --concurrency 4 \
  --cpu 1 --memory 512Mi

Run this with permission to deploy to the service (roles/run.developer plus access to the image). Replace the region, project, repo, and tag placeholders. Changing --concurrency creates a new revision, so it's easy to roll back by shifting traffic to the previous revision if latency regresses.

Remember that the effective limit is the lower of the Cloud Run setting and what your server actually supports. If Gunicorn runs 2 workers with 8 threads each, setting Cloud Run concurrency to 100 doesn't give you 100 parallel requests — it gives you 16 with a queue in front.

How to verify instead of guessing

Don't tune this by feel. Deploy two revisions — one at concurrency 1, one at your candidate value — and run a load test with a tool like hey or wrk against each:

hey -n 5000 -c 100 https://SERVICE_URL/endpoint

Then compare in Cloud Monitoring: container instance count, request latency distribution (p95 especially), and billable instance time. Watch container logs during the test for OOM kills or timeouts — those tell you the setting exceeded real per-request memory. The right configuration is the one that hits your latency target at the lowest instance count, verified against your actual traffic shape.

Trade-offs worth knowing up front

• Concurrency tuning does nothing for cold starts. That's what minimum instances is for — a separate, always-billable feature.
• Under the default CPU-allocation mode, CPU is throttled between requests, so background threads may stall regardless of how you set concurrency. If you need background work, look at always-allocated CPU, which changes billing.
• Defaults and maximums shift between Cloud Run releases. Treat any specific number in this post (including the examples) as a starting hypothesis and confirm current values in the official docs before publishing or deploying.

The actionable version: classify each service as CPU-bound, I/O-bound, or thread-unsafe; set concurrency to match (low, high, or 1 respectively); then prove it with a load test and the instance-count metric. It's a fifteen-minute experiment that often cuts either your p99 or your bill in half.