Diagnosing and Fixing High Latency and Cold Starts in Google Cloud Run

Problem Statement

When a Cloud Run service shows a steady rise in request latency or frequent “cold start” spikes, the user is often left wondering whether the issue is in the code, the container image, or the Cloud Run configuration. This guide walks through a systematic diagnostic flow that maps observable symptoms to root causes, offers concrete checks, and recommends targeted fixes. The goal is to reduce latency, eliminate unnecessary cold starts, and keep costs under control.

Root‑Cause Matrix

Symptom	Typical Cause	Key Metric / Log
Latency spikes > 200 ms on otherwise idle traffic	Cold start when scaling from zero	run.googleapis.com/container/startup_latencies, structured log field cold_start:true
Steady latency 200–500 ms even under load	CPU throttling (only during request allocation)	CPU usage < 10 % during request, high cpu_utilization during idle
Sudden increase in instance count + OOM logs	Memory pressure causing OOM kills	run.googleapis.com/container/exit_codes, Cloud Logging “Container terminated: OOM”
Latency 50–200 ms only on VPC‑connected requests	VPC connector latency or mis‑configuration	run.googleapis.com/vpc_connector_instance_count, connector health logs
Requests time out before first response	Slow container startup or missing health check	run.googleapis.com/request_latencies, “Health check failed” logs

Ordered Troubleshooting Checklist

1. Confirm the symptom in logs
   

   gcloud logging read "resource.type=cloud_run_revision AND labels.run.googleapis.com/service_name=YOUR_SERVICE" --limit 10 --format json
   

   Look for cold_start:true or Container terminated: OOM entries.
2. Check Cloud Monitoring metrics
   Navigate to Monitoring→Metrics Explorer and add:
   • run.googleapis.com/request_latencies (95th percentile)
   • run.googleapis.com/container/startup_latencies
   • run.googleapis.com/instance_count
   • run.googleapis.com/vpc_connector_instance_count
   Compare the 95th percentile latency with the observed spikes.
3. Inspect service configuration
   

   gcloud run services describe YOUR_SERVICE --region=YOUR_REGION --format=json | jq ".spec"
   

   Key fields: minInstances, maxInstances, concurrency, cpuAllocationMode, vpcAccess.
4. Profile container startup locally
   

   docker run --rm -i IMAGE sh -c "time your_app_startup_command"
   

   Note the image size: docker images IMAGE --format "{{.Size}}".
5. Validate VPC connector health
   

   gcloud compute networks vpc-access connectors describe CONNECTOR --region=YOUR_REGION
   

   Check ipCidrRange utilization and connector status.
6. Simulate load and monitor instance count
   

   hey -n 1000 -c 50 https://YOUR_SERVICE_URL
   

   Watch run.googleapis.com/instance_count and latency in real time.

Targeted Fixes

1. Reduce Cold Starts

• Set minInstances: 1 to keep a warm instance for baseline traffic. Risk: incurs cost even when idle.
• Use maxInstances with traffic splitting to pre‑allocate instances for burst traffic.
• Shrink the container image: use multi‑stage builds, distroless base images, and a comprehensive .dockerignore. Aim for < 500 MB.
• Enable cpuAllocationMode: "always_allocated" for background tasks, but monitor billing.

2. Mitigate CPU Throttling

• If the workload is CPU‑bound, set cpuAllocationMode: "always_allocated" or increase cpu per instance.
• Reduce concurrency to a value that matches the app’s ability to process requests in parallel.

3. Resolve Memory Pressure

• Increase memory per instance. Verify with run.googleapis.com/container/startup_latencies after change.
• Profile memory usage with docker stats or a language profiler.
• Remove in‑memory caches that grow unbounded; use Cloud Memorystore or Cloud Firestore instead.

4. Fix VPC Connector Issues

• Ensure the connector’s ipCidrRange has enough free IPs. If exhausted, extend the range or move to a new connector.
• Place the connector in the same region as the Cloud Run service.
• Add a healthCheck for the connector if available.

5. Handle Startup Time

• Move heavy initialization (e.g., database migrations) to a separate init container or to a background worker service.
• Use startupProbe to let Cloud Run know when the container is ready, preventing premature request routing.

Escalation Criteria

• Latency > 500 ms for 95th percentile over a 5‑minute window despite applying all fixes.
• Instance count consistently > 10 for a service that should only need 1–2 instances.
• Persistent OOM logs after memory increase (suggesting a memory leak).
• VPC connector health errors or connector IP exhaustion reports.

At this point, engage the Google Cloud Support with collected metrics, logs, and a clear description of the steps already taken.

Practical Verification Checklist

• After changing minInstances, confirm run.googleapis.com/container/startup_latencies drops to zero for the first request.
• After adjusting concurrency, observe a proportional drop in queue latency in the request_latencies metric.
• After increasing memory, ensure run.googleapis.com/container/exit_codes no longer shows 137 (OOM).
• After VPC connector fix, verify run.googleapis.com/vpc_connector_instance_count stays stable and request latency falls below 100 ms for VPC egress calls.