Spark Structured Streaming vs DStream: A Decision Guide
Decide between Spark Structured Streaming and the legacy DStream API for new real-time pipelines on Spark 3.x.
Loading…
ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Decide between Spark Structured Streaming and the legacy DStream API for new real-time pipelines on Spark 3.x.
Pick a watermark delay by measuring event lateness, then verify with state-store metrics and console output.
An architecture note on building exactly-once Spark Structured Streaming ETL: the minimal design (Kafka, checkpointing, idempotent sink), trust boundaries, failure behavior, and how to verify recovery actually works.
When IoT devices send logs late, dashboards can become stale. Spark’s event‑time watermarks let you drop or keep late data automatically. This guide shows how to set up watermarks, a working example, and the trade‑offs you need to monitor.
Configure a reliable checkpoint directory to preserve state and offsets, enabling exactly‑once processing after driver or executor failures in Spark Structured Streaming.
Learn how to implement exactly-once ETL pipelines using Apache Spark Structured Streaming, focusing on micro-batch architecture, checkpointing, and operational monitoring.
Configuring Idempotent Writes with Delta Lake MERGE in foreachBatch The goal is to use Structured Streaming’s foreachBatch to write each micro‑batch to a Delta Lake table via a MERGE statement, relying on the merge’s upsert behavior to obtain exactly‑once semantics even when Spark retries a batch. Delta Lake’s MERGE requires a deterministic match condition t
I'm designing a Spark Structured Streaming pipeline (assume Spark 3.4/3.5) that writes micro-batches to an external store via foreachBatch , with checkpointing and the write-ahead log enabled. The hard constraint is that when a batch fails partway and Spark retries it, the target must not end up with duplicate records. Two documented patterns seem applicable
Goal: Determine whether Apache Spark Structured Streaming’s internal checkpointing and write‑ahead log (WAL) mechanism provides sufficient fault tolerance for recovery from HDFS‑based checkpoint storage, or whether augmenting it with external HDFS snapshots is necessary to guard against metadata corruption or accidental deletion. Constraints: The checkpoint