Choosing Cloud Run Concurrency: A Practical Guide to Balancing Cost and Latency

Problem & Takeaway

When a microservice receives thousands of requests per second on Cloud Run, the concurrency setting – how many requests a single container handles at once – determines both cost and response time. The key decision is: What concurrency value delivers the lowest cost while keeping latency within our service level objective (SLO)? The takeaway is to start with the default (80), collect empirical data, and adjust only if latency or CPU usage breaches thresholds.

Decision Context

Constraints

• High request rate (≥10 k req/s) with a mix of CPU‑bound and I/O‑bound requests.
• Maximum average latency SLO of 200 ms.
• Budget cap: cost per request should not exceed $0.0005.
• Cloud Run limits: default max‑instances 1000; maximum concurrency 1000.

Option Comparison

Concurrency	Approx. # Instances (for 10 k req/s)	CPU per Instance	Memory per Instance	Avg Latency (CPU‑bound)	Avg Latency (I/O‑bound)	Cost Impact	Risk
1	≈10 k	1 vCPU	256 MiB	>500 ms (queueing)	≈80 ms	High (many instances)	CPU starvation, high cold‑starts
10	≈1 k	1 vCPU	256 MiB	≈300 ms	≈120 ms	Moderate	Potential queueing if CPU spikes
80 (default)	≈125	1 vCPU	256 MiB	≈150 ms	≈90 ms	Low (few instances)	Risk of CPU contention for CPU‑bound loads
100	≈100	1 vCPU	256 MiB	≈170 ms	≈95 ms	Low	Higher chance of contention; still within SLO
500	≈20	1 vCPU	256 MiB	≈300 ms	≈150 ms	Very low	High CPU contention; may exceed SLO

Trade‑Offs

• CPU‑bound workloads – lower concurrency (≤10) reduces contention, keeping per‑request CPU time low but increases instance count and cost.
• I/O‑bound workloads – higher concurrency (80–100) is safe; the container spends most time waiting for network or disk, so CPU is idle and requests can be multiplexed.
• Higher concurrency reduces the number of active instances, lowering memory and CPU‑seconds billed. Cloud Run charges per vCPU‑second and per GB‑second, so fewer instances mean less recurring cost.
• Cold starts become more frequent when concurrency is low because more instances are spun up. If your image has a long startup time, consider increasing concurrency to amortize the cold‑start cost across more requests.
• Cloud Run auto‑scales based on request rate and concurrency. Setting concurrency too high can cause the system to keep fewer instances alive, which may lead to queueing if traffic spikes beyond the capacity of each instance.

Concrete Implementation & Validation

Step 1: Build & Push the Container

# Build the image
docker build -t gcr.io/PROJECT_ID/my-service .

# Authenticate Docker with GCR
gcloud auth configure-docker

# Push the image
docker push gcr.io/PROJECT_ID/my-service

Step 2: Deploy Revisions with Different Concurrency Values

Each deployment creates a new revision. Use the same image but vary the --concurrency flag.

# Concurrency 10
gcloud run deploy my-service-conc10 \
  --image gcr.io/PROJECT_ID/my-service \
  --platform managed \
  --region us-central1 \
  --concurrency 10 \
  --memory 256Mi \
  --cpu 1 \
  --allow-unauthenticated

# Concurrency 80 (default)
gcloud run deploy my-service-conc80 \
  --image gcr.io/PROJECT_ID/my-service \
  --platform managed \
  --region us-central1 \
  --concurrency 80 \
  --memory 256Mi \
  --cpu 1 \
  --allow-unauthenticated

# Concurrency 100
gcloud run deploy my-service-conc100 \
  --image gcr.io/PROJECT_ID/my-service \
  --platform managed \
  --region us-central1 \
  --concurrency 100 \
  --memory 256Mi \
  --cpu 1 \
  --allow-unauthenticated

Required permissions: run.revisions.create, run.services.update, storage.objects.get for image access.

Step 3: Generate Load for One Hour

Use a lightweight load generator such as hey or wrk. The following example uses hey to send 10 k requests per second for 60 minutes.

# Install hey (Linux)
sudo apt-get install hey

# Run load against the revision URL
hey -z 60m -q 10000 -c 100 -m GET https://my-service-conc80-xxxxxxxx.a.run.app

Replace the URL with the service’s actual endpoint. Run the command for each revision separately.

Step 4: Collect Metrics in Cloud Monitoring

Open the Cloud Monitoring dashboard and add the following metrics:

• run.googleapis.com/latency – request latency (average, 90th percentile)
• run.googleapis.com/cpu_utilization – CPU usage per instance
• run.googleapis.com/instance_count – number of active instances
• run.googleapis.com/uptime_check_error_count – error rate (optional)

Filter each metric by service_name (e.g., my-service-conc10) so you can compare revisions side‑by‑side. Use the Time series view to plot the data over the one‑hour period.

Step 5: Validate Against SLO & Cost

• Check that the average latency stays below 200 ms and that the 90th percentile latency is also under this threshold.
• Confirm that CPU utilization does not exceed 70–80% on average; higher values indicate contention.
• Review the instance_count metric. Fewer instances with acceptable latency mean lower cost.
• Use the run.googleapis.com/cpu_utilization and run.googleapis.com/memory_utilization metrics to estimate cost per request: multiply average CPU‑seconds and GB‑seconds by the Cloud Run pricing rates, then divide by the number of requests.
• If the concurrency value 80 yields latency within SLO and the lowest cost, keep it. If 10 or 100 offers a better balance (e.g., 10 reduces latency without a prohibitive cost increase), switch to that revision and set it as the default.

Step 6: Rollout the Chosen Revision

Once validation is complete, promote the best revision to production:

# Promote revision to default
gcloud run services update my-service \
  --platform managed \
  --region us-central1 \
  --revision my-service-conc100

Replace my-service-conc100 with the revision name that met the criteria.

Limitations & Practical Checks

• These guidelines assume a stateless HTTP service. Stateful or session‑aware workloads may require different concurrency handling.
• Cloud Run’s auto‑scaling algorithm may add or remove instances in the background; monitor run.googleapis.com/instance_count to see real‑time changes.
• Concurrency values above 1000 are rarely needed and may hit platform limits. Use --max-instances to cap scaling if you expect traffic spikes.
• Always test in a staging environment before changing production concurrency. A sudden drop in concurrency can increase cold starts and cost.
• Use Cloud Logging to capture request and error logs; correlate them with metrics to investigate outliers.

Conclusion

Choosing the right concurrency for Cloud Run is a balancing act between cost and latency. Start with the default 80, gather data, and adjust only when the metrics show a clear benefit. By following the steps above, you can make an evidence‑based decision that keeps your microservice within budget while delivering the performance your users expect.