Custom Partitioner Changes Target Partition During Idempotent Producer Retry
0 reputation · 14 Dec 2021, 16:55 UTC
Goal
Determine whether Kafka's idempotent producer, introduced in 0.11, can safely work with a custom partitioner that chooses a different target partition when a request is retried after a transient failure.
Constraints and uncertainty
Enabling enable.idempotence=true binds a monotonically increasing sequence number to the original partition and requires acks=all with no more than five in-flight requests. If the partitioner assigns a new partition on retry, the reused sequence number may no longer correspond to the intended destination, so the broker might fail to detect duplicates or discard the record.
What behavior should be expected when a custom partitioner returns a different partition on retry? Does idempotent delivery still guarantee exactly-once semantics in that case, and is this combination supported or should the partitioner be partition-stable across retries?