Custom Partitioner Changes Target Partition During Idempotent Producer Retry

Expected behavior

Kafka's idempotent producer should not be used with a custom partitioner that returns a different partition for the same record on retry. This combination is not a supported stable configuration. Idempotency is scoped to each producer ID, epoch, and partition, so a retry targeting a new partition is validated against that partition's sequence state, not the original destination. The record may be accepted in the wrong partition, rejected with sequence or epoch errors, or missing from the original partition. Exactly-once semantics remain per partition per producer epoch; they do not become end-to-end exactly-once across partitions.

Confirmed facts and assumptions

Confirmed for Kafka 0.11+ idempotent producers: the broker tracks the last sequence per producer ID, epoch, and partition. Idempotent production requires acks=all and limits in-flight requests, commonly to five via max.in.flight.requests.per.connection. Sequence numbers are not portable between partitions. Assumption: the same logical record is retried within one producer epoch. Exact partitioner invocation and sticky batching on retry can differ by client and broker release, so verify against your target versions.

Steps for this case

1. Make the custom partitioner deterministic and partition-stable: for the same topic, key, value, and headers, return the same partition on every attempt, including retries.
2. Remove transient inputs from the partitioner, such as random values, timestamps, current load, or leader-dependent choices unless the leader mapping is stable for the record.
3. Keep enable.idempotence=true with acks=all and max.in.flight.requests.per.connection no more than five, or allow the client to enforce those settings.
4. If your application resubmits records after failed send callbacks, add application-level deduplication or use transactions; idempotent production does not deduplicate new ProducerRecord submissions.

Verification

Run a test producer with enable.idempotence=true, acks=all, and max.in.flight.requests.per.connection set to five. Use a logging partitioner that returns partition A first and partition B after a forced transient failure, then send one record and check producer callback errors and offsets in both partitions. Repeat with a stable partitioner that always returns the same partition; the stable case should show no sequence errors and single delivery. Confirm behavior for your target Kafka client and broker versions.

Missing diagnostic detail

Is the different partition chosen by the producer during an internal retry, or by your application resubmitting the record after a failed send? This determines whether the fix is partitioner stability alone or application-level deduplication for resubmitted records.