Spark Structured Streaming vs DStream: A Decision Guide
Decide between Spark Structured Streaming and the legacy DStream API for new real-time pipelines on Spark 3.x.
Loading…
ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Decide between Spark Structured Streaming and the legacy DStream API for new real-time pipelines on Spark 3.x.
Pick a watermark delay by measuring event lateness, then verify with state-store metrics and console output.
An architecture note on building exactly-once Spark Structured Streaming ETL: the minimal design (Kafka, checkpointing, idempotent sink), trust boundaries, failure behavior, and how to verify recovery actually works.
When IoT devices send logs late, dashboards can become stale. Spark’s event‑time watermarks let you drop or keep late data automatically. This guide shows how to set up watermarks, a working example, and the trade‑offs you need to monitor.
I'm designing a Spark Structured Streaming pipeline (assume Spark 3.4/3.5) that writes micro-batches to an external store via foreachBatch , with checkpointing and the write-ahead log enabled. The hard constraint is that when a batch fails partway and Spark retries it, the target must not end up with duplicate records. Two documented patterns seem applicable