Checkpoint Restoration vs Metastore Rollback for Upgrade Recovery

When a Spark upgrade fails and the driver process crashes mid-transition, the platform must decide whether to restore from the last persisted checkpoint or roll back the Spark SQL metastore and configuration to a pre-upgrade snapshot.

Both approaches are documented recovery paths, but they address different data integrity concerns and operate under conflicting constraints. The choice hinges on whether recomputing lost partitions via RDD lineage is feasible given available disk space and whether the metastore snapshot predates critical schema changes introduced during the upgrade.

In practice, driver crash behavior varies across Spark releases: some versions re-queue pending stages, while others terminate with UncataloguedPlanException, leaving the optimal recovery route version-dependent and context-sensitive.

Under what conditions does checkpoint restoration succeed without excessive recomputation, and when does metastore rollback risk schema mismatch or lost commits? Does the Spark version's handling of mid-upgrade driver crashes reliably preserve stage state, or does it introduce nondeterminism in recovery outcomes? How should operations teams prioritize durable checkpoint storage versus metastore snapshot consistency when both recovery mechanisms are available?