Kafka's Idempotent Producer: The Retry Safety Net You Already Paid For

Your producer times out on a produce request. The client library retries, because that's what retries are for. The problem: the broker actually wrote the batch the first time — the acknowledgment just got lost. Now the same records sit in the partition twice, and every downstream consumer sees them twice. Nothing crashed, nothing logged an error, and your data is quietly wrong.

Apache Kafka's idempotent producer exists to close exactly this hole. Setting enable.idempotence=true makes retries safe: the broker can recognize a resent batch and discard the duplicate. This post covers how it works, what it does not guarantee, and how to verify it's actually doing its job.

How the broker spots a duplicate

When idempotence is enabled, the producer is assigned a producer ID (PID) by the broker, and it attaches a monotonically increasing sequence number to every batch it sends, per partition. The broker keeps track of the last sequence number it accepted for each (PID, partition) pair. If a retried batch arrives with a sequence number the broker has already seen, the broker acknowledges it without writing the data again. If a batch arrives out of order — a gap in the sequence — the broker rejects it with an OutOfOrderSequenceException rather than silently reordering your stream.

Two consequences fall out of this design:

• Duplicates from retries are eliminated within a partition, for the lifetime of that producer session.
• Ordering is preserved even when multiple requests are in flight and one needs a retry, because the broker enforces sequence continuity.

The overhead is small: a few extra bytes of metadata per batch and some broker-side bookkeeping. For most workloads the throughput cost is negligible next to the correctness benefit of being able to retry without fear.

Configuration, and why Kafka 3.x made it the default

Idempotence imposes constraints on three related settings, and the producer enforces them at startup:

• acks must be all (the broker must confirm the write is replicated)
• retries must be greater than zero (in practice, effectively unlimited by default)
• max.in.flight.requests.per.connection must be 5 or fewer

If you set enable.idempotence=true explicitly and also set a conflicting value — say acks=1 — the producer throws a ConfigException at startup rather than running in a half-safe mode. That fail-fast behavior is a feature: it surfaces the contradiction before you're in production.

Since Kafka 3.0, if you don't explicitly override any of those three settings, the producer defaults to enable.idempotence=true. So on a modern client you may already be running idempotently without knowing it. The examples here are written against Kafka 3.x behavior; if you're on an older client, check your version's documentation, because the defaults differ.

An explicit configuration looks like this (Java producer properties):

props.put("enable.idempotence", "true");
props.put("acks", "all");              // required; explicit for clarity
props.put("retries", Integer.MAX_VALUE);
props.put("max.in.flight.requests.per.connection", 5);

The limits of the guarantee

This is the part that gets overclaimed, so it's worth being precise. Idempotence gives you exactly-once writes scoped to a single producer session and a single partition. It does not give you:

• Atomic writes across partitions or topics. If your application writes to two partitions and crashes between them, idempotence won't help you reason about that. You need Kafka transactions (transactional.id plus the transactions API) for multi-partition atomicity or read-process-write pipelines.
• Protection across producer restarts. A restarted producer gets a new PID. If your application re-reads its source and republishes the same events, those are brand-new writes as far as the broker is concerned. Application-level replay requires application-level deduplication or transactions.
• End-to-end exactly-once on its own. Consumers can still see duplicates from other sources — consumer rebalances with read_committed unset, offset commit failures, upstream replays. Idempotence is a building block, not the whole pipeline guarantee.

Also note the compatibility angle: idempotence requires broker support, so if you're running a very old broker cluster, verify support before relying on it.

Verifying it actually works

Don't just set the flag and trust it. Two practical checks:

1. Confirm the effective config. At startup, the producer logs its full configuration — grep for enable.idempotence, acks, and retries to confirm the values in effect, especially if you rely on the 3.x defaults. A ConfigException at startup would also tell you immediately if settings conflict.

2. Run a controlled retry experiment. Produce to a test topic with idempotence on, then inject a failure that forces a retry — for example, briefly disrupt network connectivity between the producer and broker so a produce request times out after the broker has written it. Then consume the partition from the beginning and count records:

# Run on a broker or a machine with the Kafka CLI tools; needs read access to the topic
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic idempotence-test --from-beginning | sort | uniq -d

If the uniq -d output is empty, retries produced no duplicates. For a lower-level view, kafka-dump-log on the partition's log segment will show the PID and sequence numbers recorded per batch — run it on the broker host with read access to the log directory. Neither check changes cluster state, so they're safe to run against a test cluster freely.

Closing

Idempotence turns Kafka producer retries from a correctness hazard into a free safety net, and on Kafka 3.x it's likely already on. The actionable steps: confirm your producer's effective config at startup, keep the required settings compatible, and be honest about the scope — the moment your pipeline spans partitions, restarts producers mid-stream, or does read-process-write, reach for transactions instead. Then run the duplicate-injection test once in a staging environment so you know the guarantee is real, not just configured.