Full backup restore vs in-place retry after Azure DevOps Server schema upgrade failure
0 reputation · 06 May 2024, 15:38 UTC
0 reputation · 06 May 2024, 15:38 UTC
Azure DevOps Server on-premises upgrades proceed in two phases: application-tier binary update and collection database schema upgrade. Once the schema upgrade phase begins, the Tfs_Configuration schema version can become irreversible, and Microsoft documentation describes restoring a consistent pre-upgrade backup set of Tfs_Configuration and all Tfs_ databases as the supported recovery when phase 2 fails.
The upgrade wizard surfaces a Retry option for certain transient failures, but the boundary between a safe retry and a state requiring full restore is not clearly defined. The collection upgrade is single-threaded per collection and can leave the database in an intermediate state; an interrupted process may leave orphaned schema objects that are not cleaned up by retry logic.
For Azure DevOps Server 2019/2020/2022 on-premises, the decision point is whether to attempt an in-place retry or to revert the entire deployment to a point-in-time backup set, with implications for recovery time objective and data consistency.
Is there a documented checklist to determine post-failure whether the schema upgrade transaction has committed? Can the presence or absence of an entry in TfsConfig upgrade history reliably indicate a committed schema version bump? What conditions make an in-place retry unsafe for a partially completed collection schema upgrade?
35275 reputation · 07 May 2024, 03:31 UTC
1. Checklist for a committed upgrade: Query TfsConfigUpgradeHistory for the most recent row and confirm its Status is Success and the UpgradeVersion matches the expected post‑upgrade value. Also verify that all objects listed in the upgrade history exist in sys.objects for the Tfs_Configuration database. If both checks pass, the transaction was committed and an in‑place retry is safe.
2. Reliability of the history entry: A Success row is a strong indicator, but it does not guarantee that every schema object was applied correctly. Missing objects or orphaned schema items can still exist because some changes occur outside the transaction. Therefore the history entry should be paired with an object‑level consistency check.
3. Conditions that make retry unsafe:
Failed or missing.Tfs_Configuration and all collection databases.
sqlcmd -S <instance> -d Tfs_Configuration -E
SELECT UpgradeVersion, UpgradeDate, Status
FROM dbo.TfsConfigUpgradeHistory
ORDER BY UpgradeDate DESC;
Status equals Success and that UpgradeVersion equals the target version (e.g., 2019.2.0).SELECT name, type_desc
FROM sys.objects
WHERE name IN (/* list of objects from the upgrade */);
Success record and full object consistency.Follow Microsoft’s guidance for the 2019‑2022 releases: if the wizard reports a non‑transient error, or if the consistency checks above fail, restore the pre‑upgrade backup set for Tfs_Configuration and all collection databases. Perform the restore on a non‑production instance first to validate that the failure reproduces and that the same error code appears, confirming that the problem is not transient.
To fine‑tune the recommendation, could you share the exact error code or message that was displayed by the upgrade wizard when the failure occurred? This will help confirm whether the failure was classified as transient or schema‑related.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.