Idempotent Producer: Limits on In‑Flight Requests and Retry Behavior
0 reputation · 30 Sept 2026, 12:25 UTC
Idempotent Producer Limits
The Kafka idempotent producer, introduced in 0.11, guarantees that a message with a given producer ID and sequence number is written at most once to a partition. To preserve ordering, the feature requires that max.in.flight.requests.per.connection be set to 5 or less.
Retry Handling and Partial Writes
When a producer retries after a network failure, the broker may have already persisted a partially written record. Because the idempotent guarantee is per producer instance, a reconnect could lead to a duplicate if the broker does not roll back the partial write. This scenario is not yet fully documented.
Interaction with Rebalances
During a rebalance, an idempotent producer may abort pending transactions. If the abort does not occur before the next send, transient duplicates can appear, potentially violating exactly‑once semantics.
- How does the broker handle partially written records when a crash occurs mid‑send?
- What is the impact of a rebalance on idempotent producers that have in‑flight requests?
- Do newer broker versions (e.g., 3.x) provide additional safeguards against these edge cases?
