Can Spark Structured Streaming foreachBatch guarantee idempotent writes across retries?

Short answer

No. foreachBatch alone does not guarantee idempotent writes across retries in Spark 3.x Structured Streaming. Spark treats the function as opaque user code and cannot roll back partial writes to an external sink. Idempotency must come from the sink, the write logic, or both.

Confirmed facts

• Spark commits micro-batch offsets to the checkpoint directory only after foreachBatch returns successfully.
• If the function writes to the sink and then fails before the checkpoint advances, Spark replays the same offset range on the next attempt.
• Spark 3.x can provide exactly-once semantics only for sinks integrated with the streaming engine and supporting transactional commits. foreachBatch bypasses that coordination.
• Offset commitment ordering relative to the sink transaction is controlled by your code inside foreachBatch, not by Spark automatically.

Likely explanation

A failed micro-batch may have already written some rows to the downstream table. Because the checkpoint still holds the previous offset, the restarted job reprocesses that range and writes the same logical rows again. Without sink-level deduplication or conditional overwrite, the replay produces duplicate rows.

Sink properties that matter

• Transactional commit: the sink can stage all rows for a batch and commit them atomically.
• Conditional write or upsert: the sink can reject or overwrite rows carrying an already-seen batch identifier.
• Primary key or unique constraint: enables deduplication on replay when combined with deterministic keys.

Steps for this case

1. Pass a deterministic batch identifier into each micro-batch, such as the batch ID or a derived watermark-based key.
2. Write rows conditionally: insert only if the batch identifier is absent, or upsert so a replayed batch overwrites the previous attempt rather than appending.
3. Stage writes in a transaction where supported, and commit the sink transaction only after all rows for the batch are ready. Do not commit checkpoint offsets separately from the sink transaction unless your sink and code can reconcile the gap.
4. Prefer a native Structured Streaming sink with exactly-once support over manual foreachBatch when one exists for your target system.

Verification

• Inspect checkpoint metadata and query status to confirm offsets advance only after the batch function succeeds.
• Inject a failure after a partial sink write, restart the query, and check whether the replayed batch creates duplicate rows.
• Review sink documentation for transactional commit or idempotent write support, and verify batch-ID deduplication on your table.

Assumption: Spark 3.x Structured Streaming. Spark 2.x behavior differs and is not covered here.