Choosing Cloud Run Concurrency for Burst Traffic and Cost Efficiency

Requirements

When deploying a stateless HTTP service on Cloud Run you often need to handle two opposing goals:

1. Keep latency low during traffic spikes.
2. Minimize the number of running instances to control cost and reduce cold‑start frequency.

The concurrency setting (the maximum number of requests a single container instance can process simultaneously) is the primary lever that trades off these goals.

Smallest Suitable Design

Start with the service’s CPU allocation:

• If the service is CPU allocated (the default), Cloud Run grants up to 1 vCPU per instance.
• If you set --cpu=always or limit CPU to a fraction (e.g., --cpu=0.5), the effective parallelism is bounded by that share.

The smallest viable design is to pick a concurrency value that does not exceed the CPU share you have provisioned. For a 1 vCPU allocation, a safe starting point is concurrency = 40; this leaves headroom for context switches and I/O wait while still allowing the autoscaler to consolidate traffic.

Trust/Data Boundaries

Cloud Run isolates each request inside the container’s sandbox. The concurrency setting does not change trust boundaries; however, higher concurrency increases the blast radius of a faulty request (e.g., a runaway loop) because more requests share the same CPU and memory. Apply standard mitigations:

• Set per‑request timeouts in your code.
• Use resource limits (e.g., ulimit) inside the container if needed.
• Enable request‑level authentication (IAM, Firebase Auth, etc.) before the concurrency decision.

Operational Checks

After deploying with a chosen concurrency, verify that the instance stays within its CPU budget and that latency remains acceptable.

Metrics to watch

• request_count_per_instance – average number of requests handled by each instance.
• container_cpu_utilization – percentage of the allocated CPU used.
• request_latencies – distribution (especially p99).

Verification procedure

1. Deploy a simple HTTP service (e.g., a Go echo server) with a test concurrency:

# Replace PROJECT_ID, SERVICE_NAME, and REGION as needed
# Requires: Cloud Run Admin role (roles/run.admin) on the project
gcloud run deploy SERVICE_NAME \
  --image=gcr.io/PROJECT_ID/echo:latest \
  --platform=managed \
  --region=REGION \
  --allow-unauthenticated \
  --concurrency=40 \
  --cpu=1

1. Generate a steady load with a tool like hey or k6 targeting the service URL. Example with hey:

hey -z 2m -q 50 -c 100 https://SERVICE_NAME-REGION.a.run.app

This sends a constant rate of ~50 requests per second with a concurrency of 100 at the client side.

1. Open Cloud Monitoring and examine the two metrics:

• If request_count_per_instance hovers near the configured value (≈40) and container_cpu_utilization stays below ~65% (for a 1 vCPU allocation), the concurrency is appropriate.
• If CPU utilization regularly exceeds 80% or latency (p99) climbs sharply, lower the concurrency.
• If the metric shows many instances each handling only a few requests, consider raising concurrency to reduce instance count.

Failure Modes

Setting concurrency too high for the allocated CPU leads to:

• CPU throttling → increased request latency and possible HTTP 429 (Too Many Requests) responses from the runtime.
• Higher memory pressure if each request holds state, potentially causing OOM kills.
• More frequent cold starts when the autoscaler scales down to zero because each instance is busy longer.

Setting concurrency too low results in:

• Excessive instance count for modest traffic, raising the base cost.
• More frequent scaling events, which adds latency for sporadic traffic (each new instance incurs a cold start).

Conditions That Would Change the Design

• Switching to --cpu=always with a fractional CPU (e.g., 0.25 vCPU) reduces the safe concurrency ceiling proportionally.
• If the workload becomes CPU‑bound (e.g., heavy computation per request), lower concurrency to avoid throttling.
• If the service starts doing significant I/O (calls to external APIs, databases), you can safely raise concurrency because the CPU spends time waiting.
• Introducing a maximum instance limit (--max-instances) may require lower concurrency to avoid hitting the limit during bursts.

Limitations and Practical Verification

The verification steps above rely on metrics that are emitted with a short delay (typically <1 min). For real‑time tuning, combine metric observation with client‑side latency measurements. Always test with a load pattern that mirrors production (e.g., Poisson‑distributed bursts) before promoting a concurrency change to a critical service.

---

Changing the concurrency setting updates the service configuration and creates a new revision; rolling back is as simple as redeploying the previous revision or adjusting the flag again.