Delta Lake rollback after a schema change: RESTORE to an old version or additive-only evolution?

Answer to the question

For a Delta Lake table that has already been read or written by downstream pipelines, the safest rollback strategy depends on how far downstream has consumed the changed schema and what retention settings are in place.

Which strategy gives a safer boundary for multiple downstream consumers?

• RESTORE rewrites the table’s metadata to a prior version. It removes the destructive change from the current schema view, but it does not delete the newer commits. Downstream jobs that have already cached the new schema (e.g., via checkpoints, structured‑streaming state, or hard‑coded column lists) will break or re‑process data unless they are paused and replayed.
• Additive‑only evolution (mergeSchema or autoMerge) accepts only safe changes: new columns, nullability relaxations, and type widening. It guarantees that no downstream job will see an incompatible schema after the change, but it cannot undo a wrong column type or a dropped column. The table remains usable, and downstream consumers can continue without coordination, but the bad schema is still present in the history and visible if someone time‑travels to the post‑change version.

Recommendation: If the destructive change can be fully excised and you can afford to pause or replay downstream pipelines, use RESTORE. If you need to keep the table live for all consumers without stopping them, prefer additive‑only evolution and handle the bad column with compensating logic (e.g., add a corrected column, deprecate the bad one).

Retention planning to keep a restored version readable

Delta Lake keeps data files for 30 days by default (controlled by delta.deletedFileRetentionDuration) and transaction‑log history for 30 days (via delta.logRetentionDuration). To guarantee that a RESTORE to a given version succeeds:

1. Run DESCRIBE HISTORY t and note the target version v and its timestamp.
2. Check the current retention settings: SHOW TBLPROPERTIES t (\"delta.logRetentionDuration\", \"delta.deletedFileRetentionDuration\"). Ensure that the horizon has not elapsed since v.
3. Verify that no VACUUM scheduled job has removed the data files needed for v by inspecting the _delta_log directory or using DESCRIBE HISTORY to confirm the commit still exists.
4. In a test environment, clone the table (SHALLOW or DEEP) and run RESTORE TABLE t TO VERSION AS OF v to confirm that downstream Spark jobs read the restored schema without errors.

Maintain the retention horizon for at least the longest expected rollback window. If you anticipate needing to roll back within 7 days, consider setting delta.logRetentionDuration = \"7 days\" and delta.deletedFileRetentionDuration = \"7 days\" during that period.

Determining the latest safe restore version after downstream consumption

Once downstream consumers have read or written data with the new schema, the safe restore boundary is the newest commit that has not yet been durably consumed by any downstream pipeline. Practically:

1. Audit checkpoints, streaming state, and materialized views that reference the table. The highest commit timestamp they point to marks the boundary.
2. Do not restore past that commit; doing so would orphan downstream state and force reprocessing.
3. If you need to roll back beyond the boundary, coordinate a coordinated re‑processing of downstream jobs.

In summary:

• Use RESTORE when you can control downstream consumption and need a clean table state.
• Use additive‑only evolution when you must keep the table live for all consumers and the bad change can be mitigated with compensating logic.
• Plan retention to cover the longest rollback window you foresee.
• Identify the safe restore boundary by inspecting downstream checkpoints and the table’s commit history.

Missing diagnostic detail that could change the recommendation

To finalize the plan, could you confirm which Delta Lake version your cluster is running? The default retention values and column‑mapping behavior differ between releases, and knowing the exact version ensures the steps above apply correctly.