Optimizing Cloud Run Concurrency for Latency and Cost

The Concurrency Trade-off

In Cloud Run, concurrency determines how many simultaneous HTTP requests a single container instance can handle. By default, Cloud Run scales instances based on request volume, but if concurrency is set too low, you trigger more frequent cold starts and increase your monthly bill. If set too high, requests compete for the same CPU and memory, causing latency spikes or Out-of-Memory (OOM) crashes.

The goal is to find the highest concurrency value your application can handle without degrading response times. For I/O-bound workloads (like APIs calling a database), high concurrency is efficient. For CPU-bound workloads (like image processing), low concurrency is necessary to prevent request queuing.

Mechanism: CPU Allocation and Concurrency

Concurrency is not a standalone setting; it is tied to how CPU is allocated. To use a concurrency value greater than 1, you must configure the service to have CPU always allocated. If you use the default "CPU allocated during request processing," the platform effectively forces concurrency to 1, regardless of your configuration, because the instance is throttled when not actively processing a specific request.

Deployment Example

To deploy a lightweight API capable of handling up to 80 simultaneous requests per instance with 2 vCPUs, run the following command from your local terminal with the Cloud SDK installed. You will need roles/run.admin and roles/iam.serviceAccountUser permissions.

gcloud run deploy myservice \
  --image gcr.io/[PROJECT_ID]/myservice \
  --concurrency 80 \
  --cpu 2 \
  --memory 512Mi \
  --no-cpu-throttling

Key Flags Explained:

• --concurrency 80: Tells Cloud Run to send up to 80 requests to one instance before starting a new one.
• --no-cpu-throttling: This ensures CPU is always allocated, which is a prerequisite for concurrency > 1.
• --cpu 2: Provides the necessary compute power to handle the parallel load.

Comparison: Concurrency Profiles

Workload Type	Recommended Concurrency	CPU Allocation	Primary Risk
I/O Bound (API Gateway)	High (50-80)	Always Allocated	Memory exhaustion (OOM)
CPU Bound (Data Processing)	Low (1-10)	Always Allocated	Request queuing/Latency
Simple Web Page	Default (1)	During Request	Frequent cold starts

Limits and Common Pitfalls

The 80-Request Cap

Cloud Run caps concurrency at 80 requests per instance. If your application is extremely lightweight and can handle more, you cannot increase this limit further via the platform settings. You must instead scale horizontally by allowing more instances to spin up.

Memory Pressure

Increasing concurrency does not increase the memory available to the instance. If each request consumes 10MB of RAM and you set concurrency to 80, you need at least 800MB of available memory just for the requests, plus the overhead of your application runtime. Setting high concurrency with low memory limits (e.g., 256Mi) is a common cause of random container restarts.

The "Hidden" Override

A frequent mistake is setting --concurrency 80 but forgetting --no-cpu-throttling. In this scenario, the deployment may succeed, but the platform will ignore your concurrency setting and treat the instance as concurrency=1. This leads to an unexpected surge in the number of instances created, significantly increasing costs.

Verification and Diagnostics

To verify that your concurrency settings are active and performing as expected, follow these steps:

1. Verify Effective Configuration

Run this command to check the actual limit applied to the service:

gcloud run services describe myservice --format='value(spec.template.spec.containers[0].resources.limits.concurrency)'

If the output is empty or 1 despite your deployment flags, check if CPU throttling is enabled.

2. Load Testing

Use a tool like k6 or hey to send a steady stream of concurrent requests. While the test is running, navigate to the Cloud Run Monitoring tab in the Google Cloud Console.

• Check: "Request count per instance."
• Expected Result: The graph should show the request count climbing toward your set limit (e.g., 80) before a new instance is provisioned.
• Warning Sign: If you see 429 Too Many Requests errors or a sharp spike in latency while the instance count is low, your concurrency is set higher than your CPU/Memory can actually support.

Rollback

If the service becomes unstable, revert to the default single-concurrency behavior by running:

gcloud run services update myservice --concurrency 1 --cpu-throttling