Tuning Cloud Run Concurrency: When More Requests Mean Less Cost

Why Concurrency Matters in Cloud Run

When you deploy a container on Google Cloud Run, the platform automatically scales the number of instances that serve your traffic. By default, each instance can handle up to 80 concurrent requests. The max‑concurrency setting tells Cloud Run how many simultaneous requests a single instance should process before it spins up a new one.

Adjusting concurrency is the simplest way to squeeze more work out of each instance, reduce cold‑start frequency, and lower your bill. But if you set it too high or too low, you risk higher latency or wasted capacity. The key is to match the setting to your app’s CPU and memory profile.

Understanding the Trade‑Offs

Concretely, concurrency is a knob that balances two opposing goals:

• Cost efficiency – A higher concurrency value means each instance stays busy longer, so fewer instances are needed to serve the same traffic volume.
• Performance – A lower concurrency value reduces the chance that a single request will starve others for CPU or memory, keeping latency predictable.

When you increase concurrency, you also increase the risk of hitting memory limits. If an instance’s memory is exhausted while handling many requests, Cloud Run will terminate the instance with an OOMKilled event, which can spike latency for the next request.

Practical Example: Tuning a Node.js API

Suppose you have a Node.js Express API that makes external HTTP calls and spends most of its time waiting for network responses. The CPU usage per request is low, but you want to keep the cost down during peak traffic.

Step 1 – Deploy with Default Concurrency

gcloud run deploy my-node-api \
  --image gcr.io/my-project/my-node-api:latest \
  --region us-central1 \
  --platform managed \
  --allow-unauthenticated

After deployment, Cloud Run will use the default concurrency of 80.

Step 2 – Measure Baseline Metrics

Use the Cloud Console or gcloud monitoring to watch two metrics:

• container.googleapis.com/instance_count – how many instances are running.
• container.googleapis.com/request_latency – average request latency.

Run a load test that sends 400 requests per second (RPS) for 30 seconds. Record the instance count and latency.

Step 3 – Reduce Concurrency to 20

Now redeploy with a lower concurrency:

gcloud run deploy my-node-api \
  --image gcr.io/my-project/my-node-api:latest \
  --region us-central1 \
  --platform managed \
  --concurrency 20 \
  --allow-unauthenticated

Repeat the same load test. You should see:

• More instances (about 20‑30) because each instance handles fewer requests before scaling.
• Lower latency, especially under peak RPS, because each request gets more CPU time and less contention.

Step 4 – Push Concurrency Back Up to 80

Deploy again with the default concurrency to confirm the cost savings:

gcloud run deploy my-node-api \
  --image gcr.io/my-project/my-node-api:latest \
  --region us-central1 \
  --platform managed \
  --allow-unauthenticated

If the latency remains acceptable, you’ve reduced your instance count and bill while keeping performance within SLA.

When to Keep Concurrency Low

If your application is CPU‑intensive—say, a Go service that crunches data in memory—setting concurrency to 1 or 2 is safer. Each request will have the full CPU share, preventing timeouts and ensuring predictable latency. In such cases, the cost benefit of higher concurrency is outweighed by the risk of throttling.

Key Limitations and Safety Checks

• Memory OOMs – Monitor the OOMKilled metric. If you see spikes after increasing concurrency, drop the value or increase the memory allocation.
• CPU Throttling – Use Cloud Run’s cpu setting. If you set cpu=1 (always allocated), the instance gets full CPU even when idle, which can mitigate throttling when concurrency is high.
• Language Runtime – Node.js (single‑threaded) can handle many concurrent requests because the event loop stays responsive while waiting on I/O. Java or .NET, which use multiple threads, may benefit from lower concurrency to avoid thread contention.

Actionable Checklist

1. Profile your app’s CPU and memory usage per request.
2. Start with the default concurrency (80) and record baseline metrics.
3. Iteratively lower concurrency to see latency improvements, watching for OOMs.
4. If latency is acceptable, you can raise concurrency back up to reduce cost, again monitoring for resource events.
5. Automate the process with a simple CI pipeline that runs a load test and alerts if OOMs or latency exceed thresholds.

By following this cycle, you’ll find the sweet spot that balances cost and performance for your Cloud Run service.