Delta Lake + Spark: Unresolved Consistency Between OPTIMIZE Compaction and Concurrent Reads

Delta Lake provides true snapshot isolation during concurrent OPTIMIZE operations by utilizing the transaction log (the delta-log). When a reader starts, it identifies the latest version of the log and reads only the files explicitly listed in that specific snapshot. Even if an OPTIMIZE commit completes while the read is active, the reader continues to see the pre-compaction files until its session concludes.

How Snapshot Isolation is Maintained

The consistency 'gap' mentioned is mitigated by Delta's atomic-commit architecture. The process follows these rules:

• Atomic Commits: The OPTIMIZE command creates new compacted parquet files, but these are invisible to any reader until the new JSON commit file is written to the transaction log.
• Version Pinning: A Spark reader pins the log version at the start of the query. It ignores any new log entries created after its version was established.
• Immutability: Original data files are not deleted immediately upon compaction. They remain on storage until a VACUUM command is executed, ensuring active readers can still access them.

Storage System Influence

The underlying storage system (S3, HDFS, Azure Blob) affects performance but not the logical consistency of Delta Lake:

• S3/Azure Blob: These systems offer strong read-after-write consistency. Once the Delta log JSON is written, any subsequent reader is guaranteed to see it.
• HDFS: Provides atomic directory renames which Delta uses for certain operations, though Delta primarily relies on the log-level atomicity to bypass file-system-level consistency issues.

Configuration for Consistent Reads

To ensure the strictest behavior during high-concurrency environments, consider these configurations and strategies:

1. Avoid Frequent VACUUM: Ensure spark.vacuumRetentionDuration is set high enough (the default is 7 days) to prevent the deletion of files that concurrent readers are still using.
2. Isolation Level: Verify that spark.isolationLevel is set to WriteSerializable (the default) to prevent concurrent writes from corrupting the state.
3. Check Refresh Logic: If your application requires the absolute latest data immediately after an optimization, the reader must implement a retry or re-initialize the DataFrame after the OPTIMIZE commit is detected.