Kafka's Idempotent Producer: Exactly-Once Per Partition Without the Transaction Overhead

You deploy a producer, a broker hiccups mid-request, the client retries — and now your topic has two copies of the same order event. Downstream, someone double-charges a customer or double-counts a metric. This is the default at-least-once behavior of Kafka producers, and it's the reason many teams reach for the full transactional API when they don't actually need it. Kafka's idempotent producer, stable since 0.11.0, solves the duplicate-on-retry problem with a single configuration flag — as long as you understand exactly what it does and doesn't cover.

How the broker spots a duplicate

When you enable idempotence, the broker assigns each producer instance a unique Producer ID (PID). The producer then attaches a monotonically increasing sequence number to every batch it sends, scoped per partition. The broker keeps track of the highest sequence number it has written for each (PID, partition) pair and rejects anything with a sequence number less than or equal to that watermark.

So when a retry fires because an acknowledgment got lost on the network, the broker sees the same PID and sequence number, recognizes it as a duplicate of a batch it already persisted, and acks it without appending again. The state lives in the log itself, so it survives broker restarts and leader failover. Ordering is preserved too — the broker won't accept sequence number 7 if it's still waiting on 6, which closes the reordering hole that used to force people to set max in-flight requests to 1 and eat the throughput hit.

The configuration contract

Idempotence isn't just one flag; it's a set of settings that must hold together. In modern clients (2.5+), setting enable.idempotence=true makes the rest the default, but it's worth knowing what the contract requires:

Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// The idempotence contract
props.put("enable.idempotence", "true");
props.put("acks", "all");                          // required
props.put("retries", Integer.MAX_VALUE);           // must be > 0; large is safe
props.put("max.in.flight.requests.per.connection", 5); // must be <= 5

KafkaProducer producer = new KafkaProducer<>(props);

Each constraint exists for a reason. acks=all means the leader waits for all in-sync replicas before acknowledging, so an acked write survives a leader failure. Generous retries let the client ride out transient errors instead of surfacing them to your code. And the in-flight cap of 5 is the window within which the broker can safely reorder retried batches back into sequence.

If you set conflicting values explicitly — say acks=1 with idempotence on — the client throws a ConfigException at startup rather than silently weakening the guarantee.

Verifying it actually suppresses duplicates

Don't take the guarantee on faith; it's easy to exercise on a local cluster (Docker Compose with a broker version 2.8 or newer works fine). The test: produce a batch of messages, force retries, and check for duplicates.

# Terminal 1: start consuming so you can watch the output
kafka-console-consumer --bootstrap-server localhost:9092 \
  --topic orders --from-beginning --property print.offset=true

Then run your idempotent producer against orders with a short request.timeout.ms (e.g., 5000) and, mid-batch, restart the broker container. The producer will retry against the recovered broker. What to check:

• The consumer output shows each message exactly once, with sequential offsets and no gaps.
• Broker logs may show duplicate-sequence rejections during the retry window — that's the mechanism working, not an error.
• Producer metrics: record-error-rate should stay near zero for retriable errors because retries absorb them.

If you see duplicated values in the consumer output, the usual suspects are a second producer instance writing to the same partition (each gets its own PID, which is fine — but your test harness may be double-sending) or idempotence silently disabled by an incompatible config.

The boundary: one partition, one producer session

This is the trade-off that matters most for design decisions. Idempotence guarantees exactly-once writes within a single partition, from a single producer session. It does not cover:

• Cross-partition or cross-topic atomicity. If your write to orders and your write to audit must succeed or fail together, you need the transactional API (transactional.id, initTransactions, commit/abort), which builds on the same PID/sequence machinery but adds a coordinator.
• Read-process-write pipelines. Consuming from topic A and producing to topic B with exactly-once end-to-end semantics requires transactions plus isolation.level=read_committed on the consumer.
• Application-level duplicates. If your code calls send() twice with the same payload, the broker sees two distinct sequence numbers and happily writes both. Idempotence deduplicates retries, not your business logic.

There's also an availability cost worth naming: because acks=all is mandatory, a produce request will block or fail when fewer than min.insync.replicas replicas are alive. During a partial outage, an idempotent producer is less available than a fire-and-forget acks=0 producer. For most pipelines that's the right trade — you're choosing correctness — but size your replica count and min.insync.replicas=2 (with replication factor 3) so a single broker loss doesn't stall writes.

When to reach for it

The decision rule is simple: if you're producing to Kafka at all, turn idempotence on. The metadata overhead is small, ordering is preserved under retries, and you eliminate the most common duplicate scenario — retry-after-lost-ack — for free. Reserve the transactional API for the cases that genuinely need atomic multi-partition writes or consume-transform-produce chains. Start with the config above, run the forced-retry test against a local broker, and check your consumer output for gaps and duplicates before you trust it in production.