Idempotent producer or full transactions for retry-safe Kafka writes

Idempotent producer or full transactions for retry-safe Kafka writes

For a workload that only ever writes to one partition per record, the idempotent producer (enable.idempotence=true with acks=all and retries>0) is typically sufficient to guarantee no duplicate records across retries. It uses broker-assigned sequence numbers to deduplicate retries within a single producer session and partition.

However, idempotence does not survive a producer restart that generates a new producer ID (PID). The broker-session state is lost, so retries after restart can re-emit duplicate records unless the producer re-establishes the idempotence session.

Full transactions (initTransactions()/commitTransaction()) add cross-partition atomicity and guarantee exactly-once even across PID changes, at the cost of a transaction coordinator, a replicated transaction state log, and reported throughput overhead. For single-partition-only workloads, transactions provide no additional duplicate protection beyond the idempotent producer, unless the workload requires atomic writes to multiple partitions or needs zombie fencing after broker failover.

To rely on idempotence safely, configure delivery.timeout.ms to be less than the producer session timeout and keep retries within the idempotence window. Misconfigured timeouts combined with unbounded retries may cause retries to exceed session lifetimes, bypassing idempotence guarantees.

Missing diagnostic detail: Does the workload involve producer restarts that change the PID, or is the producer long-running with a stable PID? This determines whether idempotence alone is sufficient or if full transactions are needed.