Health check failure: Cloud Run revision upgrade recovery

Question

Goal
Safely recover from a Cloud Run revision that fails health checks after an upgrade, ensuring minimal service disruption and preserving request integrity.
Constraints & Uncertainty
Cloud Run’s traffic‑splitting feature allows a new revision to receive a percentage of traffic, but the platform does not automatically revert traffic to the previous revision when health checks fail. Operators must manually adjust traffic weights via the console or gcloud, and there is no built‑in alert that forces a rollback. This lack of automatic rollback can expose services to prolonged failure periods if operators overlook manual intervention.
Unresolved Questions

Should we implement an automated traffic shift when a revision consistently fails health checks?
What monitoring thresholds or alerting patterns best indicate a need for manual rollback?
Does Cloud Deploy’s canary release workflow provide a more reliable rollback mechanism compared to native Cloud Run traffic splitting?

Wasi · Answer

p>To recover from a Cloud Run revision that fails health checks, you must manually reassign traffic back to the previous stable revision. Cloud Run does not currently perform an automatic rollback if a new revision fails its health checks; it will continue to route traffic to the failing instances until the configuration is updated.

Recovery Steps
If you deploy a revision that is returning 5xx errors or failing startup probes, follow these steps to restore service stability:

Identify the stable revision: List your revisions to find the name of the last known working version.
gcloud run revisions list --service [SERVICE_NAME]
Shift traffic back: Use the update-traffic command to point 100% of traffic to the stable revision.
gcloud run services update-traffic [SERVICE_NAME] --to revisions=[STABLE_REVISION_NAME]
Verify recovery: Check the metrics and logs to ensure error rates have dropped.

Why Automatic Rollback Failed
Cloud Run's traffic splitting allows for canary releases, but the platform treats the traffic assignment as the desired state. If a revision is assigned 10% of traffic and fails health checks, the platform maintains that 10% assignment while reporting errors. There is no native "circuit breaker" that triggers a traffic weight shift based on instance health check status alone.

Prevention and Better Strategies
To avoid manual intervention in the future, consider these architectural shifts:

Cloud Deploy: Implement Google Cloud Deploy with a Canary strategy. This allows for more granular control and automated rollbacks if deployment metrics exceed thresholds during the rollout.
Startup Probes: Ensure you have configured startup probes correctly. This prevents Cloud Run from sending traffic to a container until it is fully ready, reducing the impact of revisions that crash on boot.
Alerting: Set up Cloud Monitoring alerts on run.googleapis.com/request_count filtered by response_code >=500 to notify operators immediately when an upgrade fails.

Health check failure: Cloud Run revision upgrade recovery

Goal

Constraints & Uncertainty

Unresolved Questions

1 answer

Recovery Steps

Why Automatic Rollback Failed

Prevention and Better Strategies

Your perspective belongs here.

0 question comments