ReadMeFeed / Community knowledge

#structured-streaming

Real questions. Useful conversations. Find the people who know your stack.

Question0 answers2,535 followingPosted 30 Mar 2024, 07:08 UTC117.8K views3,386 votes

Configuring Idempotent Writes with Delta Lake MERGE in foreachBatch

Configuring Idempotent Writes with Delta Lake MERGE in foreachBatch The goal is to use Structured Streaming’s foreachBatch to write each micro‑batch to a Delta Lake table via a MERGE statement, relying on the merge’s upsert behavior to obtain exactly‑once semantics even when Spark retries a batch. Delta Lake’s MERGE requires a deterministic match condition t

Community & Careers · Editorial question

Community & Careers

Question0 answers1,960 followingPosted 15 Oct 2020, 10:40 UTC112.7K views4,197 votes

Will Spark Provide a Unified Exactly‑Once API for All Sinks?

Goal Maintain exactly‑once semantics across all Structured Streaming sinks during retries, avoiding duplicate writes when a task restarts. Current Landscape Apache Spark relies on checkpointing and write‑ahead logs to recover from failures. Exactly‑once guarantees are only available for sinks that implement idempotent writes (Kafka, Delta Lake, HDFS/Parquet)

Community & Careers · Editorial question

idempotence structured-streaming exactly-once