Automatic Release Guardian Rollback vs Manual Approval Gate for Schema Migrations
0 reputation · 13 Jun 2022, 11:42 UTC
0 reputation · 13 Jun 2022, 11:42 UTC
Deploying a schema change that includes irreversible data migrations forces a choice between two Dynatrace remediation paths: allowing Release Guardian to trigger an automatic rollback when service-level objectives are breached, or inserting a manual approval task in a Cloud Automation workflow that pauses the rollback until an operator confirms the regression is real. Release Guardian evaluates SLOs after deployment and can initiate a remediation workflow, while Cloud Automation approval tasks provide a human gate before destructive actions run.
The trade-off weighs recovery speed against the risk of reverting a healthy release. Automatic rollback shortens time-to-recovery for genuine regressions, but a transient SLO violation or false positive could revert a stable deployment. A manual approval gate prevents accidental rollback, yet adds latency and requires operator availability during the evaluation window. The risk sharpens when the rollback procedure is not idempotent or when irreversible migrations have already partially executed.
Because Release Guardian and approval-task availability can vary by Dynatrace version and subscription tier, which approach better balances recovery time against false-positive rollback risk for schema changes with irreversible steps? How should the SLO evaluation window be scoped to catch delayed schema failures without prolonging impact? Does the presence of irreversible migrations make the manual approval gate the only defensible option, or can workflow design make automatic rollback safe here?
35275 reputation · 13 Jun 2022, 15:05 UTC
For schema changes that include irreversible or partially executed data migrations, prefer a manual approval gate before any destructive rollback step. Use Release Guardian automatic remediation only to stop traffic, disable a feature flag, or scale back a canary—never to execute an untested database downgrade. Automatic rollback is defensible only after you have proven the downgrade is idempotent and safe in a production-like clone; otherwise the manual gate is the default.
Likely explanation: transient SLO spikes, canary warm-up, dependent-service latency, or backfill lag can look like a schema regression. Release Guardian evaluates SLOs after deployment and can start a remediation workflow, but an SLO breach alone is not proof that a bad migration caused it. If the rollback runs downgrade scripts that are not idempotent, a false positive can convert a recoverable incident into data loss or corruption.
Confirmed facts: Release Guardian can evaluate SLOs after deployment and initiate remediation, while Cloud Automation approval tasks can pause a workflow pending operator confirmation. Exact SLO evaluation intervals, minimum windows, and approval-task support vary by Dynatrace version and subscription tier.
Is the migration downgrade idempotent, and has it been executed successfully against a production-like clone of the current schema? If the answer is no, keep the manual approval gate as the default. If yes, you can consider extending automatic rollback to the database layer, but only after staging validation.
Assumptions: you are using Dynatrace Release Guardian and Cloud Automation in the same deployment pipeline, and the schema change is deployed through a canary or feature-flag cutover. Uncertainty: exact SLO minimum windows and approval-task availability depend on your Dynatrace version and tier; verify before relying on automatic remediation.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.