Idempotent producer or full transactions for retry-safe Kafka writes
0 reputation · 12 Jun 2022, 13:42 UTC
The constraint
A producer must survive client-side retries (network timeouts, broker failovers) without writing duplicate records to a single-partition-per-key workload. Kafka documents two relevant mechanisms: the idempotent producer (enable.idempotence=true with acks=all and retries>0), and full transactional writes via initTransactions()/commitTransaction().
The trade-off
The idempotent producer deduplicates retries within a producer session and partition, but offers no atomicity across partitions. Transactions add cross-partition atomicity and zombie fencing, at the cost of a transaction coordinator, a replicated transaction state log, and reported throughput overhead. For a workload that only ever writes to one partition per record, the extra machinery of transactions may buy nothing — but it is unclear whether idempotence alone covers every retry edge case, such as producer restarts where the PID changes.
Assume a recent Kafka 3.x broker and the matching Java client; behavior on older versions may differ and should be verified before relying on it.
Open questions
- For single-partition writes only, is the idempotent producer sufficient to guarantee no duplicates across retries, including after a producer restart?
- Does enabling transactions provide any additional duplicate protection in this scenario that justifies the coordinator overhead?
- How should
delivery.timeout.msand retry settings interact with idempotence to avoid silent reordering or duplication?
