ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Detaspace’s native Delta Lake support lets you add columns on‑the‑fly with minimal overhead. Learn how to evolve schemas, the limits of the feature, and a step‑by‑step example that keeps your lake consistent and performant.
A Delta Lake table on Apache Spark receives frequent schema changes: new columns in most releases, and occasionally a column drop or type change that later has to be undone. The goal is to choose one documented rollback strategy that keeps the table recoverable without breaking downstream Spark jobs. Two documented approaches fit. Time travel with RESTORE re
Configuring Idempotent Writes with Delta Lake MERGE in foreachBatch The goal is to use Structured Streaming’s foreachBatch to write each micro‑batch to a Delta Lake table via a MERGE statement, relying on the merge’s upsert behavior to obtain exactly‑once semantics even when Spark retries a batch. Delta Lake’s MERGE requires a deterministic match condition t
Goal We want to confirm that data recovered from a Delta Lake backup remains accurate after a subsequent OPTIMIZE compaction, while reads are performed concurrently. Constraints The Delta transaction log guarantees ACID semantics, but the compaction commit is appended asynchronously. When a reader starts after the compaction trigger but before the commit is
detaspace