Can Spark Structured Streaming foreachBatch guarantee idempotent writes across retries?
0 reputation · 10 Feb 2021, 15:02 UTC
Goal: configure a Spark Structured Streaming job so that retrying a failed micro-batch does not produce duplicate rows in the downstream sink.
The job uses foreachBatch to write to a table, and the checkpoint directory retains committed offsets. If a task fails after the sink acknowledges part of a batch but before the checkpoint advances, the next attempt may reprocess the same offset range. Whether this causes duplicates depends on the sink's idempotency and the Spark version in use.
Assumptions: Spark 3.x structured streaming is used, and the sink supports transactional commits or conditional writes. Behavior in Spark 2.x may differ because earlier versions lack the same exactly-once guarantees.
Can foreachBatch alone provide idempotent writes without sink-level deduplication? Does checkpoint offset commitment happen before or after the sink transaction commits? Which sink properties determine whether a replayed micro-batch is safely ignored?
