Delta Lake + Spark: Unresolved Consistency Between OPTIMIZE Compaction and Concurrent Reads

Goal

We want to confirm that data recovered from a Delta Lake backup remains accurate after a subsequent OPTIMIZE compaction, while reads are performed concurrently.

Constraints

The Delta transaction log guarantees ACID semantics, but the compaction commit is appended asynchronously. When a reader starts after the compaction trigger but before the commit is fully propagated, it may see a mix of pre‑compaction and post‑compaction files, potentially violating snapshot isolation.

Unresolved Questions

• Does the compaction commit provide true snapshot isolation for readers that start during the optimization?
• Which Spark or Delta Lake configuration options can enforce consistent reads while OPTIMIZE is in progress?
• How does the underlying storage system (S3, HDFS, Azure Blob) influence this consistency gap?