Choosing Concurrency and CPU Mode in Cloud Run for 200 ms Latency Targets

Decision Overview

When you expose a user‑facing API on Cloud Run, the two knobs that most directly influence latency are concurrency (how many requests an instance can serve at once) and CPU allocation mode (whether the CPU is always on or only during a request). The goal is to keep the 95th‑percentile latency under 200 ms while keeping costs reasonable.

Decision Matrix

Concurrency	CPU Mode	Max Instances	Estimated Monthly Cost (USD)	95th‑pct Latency
80	Always allocated	20	$150	≈120 ms
80	CPU only during request	20	$105	≈180 ms

Trade‑Off Analysis

CPU Always Allocated

CPU remains active even when the instance is idle. This eliminates the possibility of CPU throttling during bursts and reduces the cold‑start penalty on new instances. The baseline cost is higher—about 30 % more—because you pay for full CPU cycles regardless of traffic.

CPU Only During Request

CPU is throttled to zero when no requests are being processed. This saves money but introduces a small warm‑up period for new instances. The first few requests to a freshly scaled instance can suffer 50–150 ms latency spikes, which may push the 95th‑percentile above a tight 200 ms SLA.

Concurrency Limits

Setting concurrency to 80 reduces the number of instances needed and lowers cost. However, if your application is CPU‑bound, high concurrency can cause internal queuing and inflate tail latency. Match concurrency to the number of worker threads your code can safely handle.

Concrete Deployment Example

Below is a gcloud command that deploys a revision with the settings that keep latency below 200 ms under peak load:

gcloud run deploy my-service \
  --image gcr.io/PROJECT_ID/my-image \
  --concurrency 80 \
  --cpu-always \
  --max-instances 20 \
  --region us-central1 \
  --platform managed \
  --project PROJECT_ID

Replace PROJECT_ID and my-image with your own values. The command requires the roles/run.admin IAM role.

Validating Performance

• Cloud Monitoring Latency: Inspect the run.googleapis.com/request_latencies distribution for the deployed revision. The 95th‑percentile should stay below 200 ms.
• CPU Utilization: Monitor run.googleapis.com/container/cpu/utilization. In "cpu always" mode the metric should show a steady baseline; in "cpu only" mode it should be bursty.
• Load Test: Run a synthetic load (e.g., locust or k6) against the service URL. Verify that the instance count stabilizes at or below the --max-instances value and that latency remains within target.

Limitations & Cautions

• Setting concurrency too high can cause request queuing inside an instance, increasing tail latency despite available CPU.
• Changing CPU mode after deployment creates a new revision. Use traffic splitting to avoid sudden latency regressions.
• Estimated costs are based on current Cloud Run pricing and may vary with region or usage patterns.

Practical Check

After deployment, run:

gcloud run services describe my-service --platform managed --region us-central1 \
  --format 'value(status.latestReadyRevisionName)'

Use the returned revision name to query Cloud Monitoring metrics and confirm that the performance matches the decision table.