ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
An architecture note on building exactly-once Spark Structured Streaming ETL: the minimal design (Kafka, checkpointing, idempotent sink), trust boundaries, failure behavior, and how to verify recovery actually works.
When IoT devices send logs late, dashboards can become stale. Spark’s event‑time watermarks let you drop or keep late data automatically. This guide shows how to set up watermarks, a working example, and the trade‑offs you need to monitor.
Configure a reliable checkpoint directory to preserve state and offsets, enabling exactly‑once processing after driver or executor failures in Spark Structured Streaming.
Learn how to implement exactly-once ETL pipelines using Apache Spark Structured Streaming, focusing on micro-batch architecture, checkpointing, and operational monitoring.
Configuring Idempotent Writes with Delta Lake MERGE in foreachBatch The goal is to use Structured Streaming’s foreachBatch to write each micro‑batch to a Delta Lake table via a MERGE statement, relying on the merge’s upsert behavior to obtain exactly‑once semantics even when Spark retries a batch. Delta Lake’s MERGE requires a deterministic match condition t
I'm designing a Spark Structured Streaming pipeline (assume Spark 3.4/3.5) that writes micro-batches to an external store via foreachBatch , with checkpointing and the write-ahead log enabled. The hard constraint is that when a batch fails partway and Spark retries it, the target must not end up with duplicate records. Two documented patterns seem applicable
Goal: Determine whether Apache Spark Structured Streaming’s internal checkpointing and write‑ahead log (WAL) mechanism provides sufficient fault tolerance for recovery from HDFS‑based checkpoint storage, or whether augmenting it with external HDFS snapshots is necessary to guard against metadata corruption or accidental deletion. Constraints: The checkpoint
Migration goal Consider a small Structured Streaming job that reads from Kafka and writes to a relational sink. The plan is to move it to a newer Spark 3.x release, possibly with a revised aggregation, without pausing the pipeline. The open decision is whether the existing checkpoint directory can carry the query across that boundary, or whether cutover need
Goal Maintain exactly‑once semantics across all Structured Streaming sinks during retries, avoiding duplicate writes when a task restarts. Current Landscape Apache Spark relies on checkpointing and write‑ahead logs to recover from failures. Exactly‑once guarantees are only available for sinks that implement idempotent writes (Kafka, Delta Lake, HDFS/Parquet)