#idempotent-writes

Can Spark Structured Streaming foreachBatch guarantee idempotent writes across retries?

Goal: configure a Spark Structured Streaming job so that retrying a failed micro-batch does not produce duplicate rows in the downstream sink. The job uses foreachBatch to write to a table, and the checkpoint directory retains committed offsets. If a task fails after the sink acknowledges part of a batch but before the checkpoint advances, the next attempt m