Will Spark Provide a Unified Exactly‑Once API for All Sinks?

Goal

Maintain exactly‑once semantics across all Structured Streaming sinks during retries, avoiding duplicate writes when a task restarts.

Current Landscape

Apache Spark relies on checkpointing and write‑ahead logs to recover from failures. Exactly‑once guarantees are only available for sinks that implement idempotent writes (Kafka, Delta Lake, HDFS/Parquet). JDBC and custom sinks lack native idempotence, forcing developers to embed deduplication logic manually. The foreachBatch operator does not enforce any global guarantee, and the API exposes no single toggle to enable exactly‑once semantics across all sinks.

Unresolved Decision

There is no clear roadmap indicating whether Spark will introduce a unified exactly‑once API that abstracts idempotent writes for arbitrary external systems. This decision impacts how developers design fault‑tolerant streaming pipelines.

Specific Questions

• Will future Spark releases add a global configuration to enforce exactly‑once semantics on any sink, regardless of native support?
• If such a toggle is introduced, how will Spark reconcile conflicts between sink‑specific idempotence settings and the global flag?
• What mechanisms will Spark provide to detect and prevent duplicate writes for non‑idempotent sinks when the global toggle is enabled?