Kafka Idempotent Producers vs Transactions: Choosing the Right Exactly-Once Tool

The Duplicate Problem in At-Least-Once Pipelines

Kafka's default producer retries are a safety net that can backfire. When a broker acknowledges a write but the response never reaches the client, the producer resends the batch. The broker appends it again. Downstream consumers see the same record twice. This isn't a bug—it's the contract of at-least-once delivery. But for pipelines that aggregate counts, update materialized views, or trigger side effects, duplicates corrupt state.

The engineering decision isn't whether to handle duplicates. It's whether to handle them in application code or let the broker do it. Kafka offers two mechanisms: idempotent producers (single-partition deduplication within a session) and transactions (multi-partition atomic writes with offset commits). The practical rule: enable idempotence everywhere by default. Reserve transactions for consume-transform-produce topologies where duplicates would break downstream correctness.

How Idempotent Producers Eliminate Single-Partition Duplicates

Setting enable.idempotence=true makes the producer attach a Producer ID (PID) and a monotonically increasing sequence number per partition. The broker tracks the highest sequence number seen for each PID-partition pair. A retried batch with a sequence number ≤ the stored maximum is silently discarded. No application changes required.

Idempotence also locks in safe defaults that you'd otherwise have to configure manually:

• acks=all — wait for all in-sync replicas
• retries=Integer.MAX_VALUE — effectively unlimited retries
• max.in.flight.requests.per.connection=5 — preserves ordering while allowing pipelining

These defaults are enforced. If your code overrides acks or retries, the producer will fail to start with a clear configuration error. This guardrail prevents silent weakening of guarantees.

Scope limitation: idempotence covers one producer session and one partition at a time. Restart the producer and you get a new PID. The broker has no memory of the old sequence numbers. Write to multiple partitions and each partition deduplicates independently—no cross-partition atomicity.

When You Need Transactions: Read-Process-Write Pipelines

Consider a stream processing job that consumes from topic A, transforms records, and produces to topic B. If the job crashes after writing to B but before committing the consumed offset, a restart reprocesses the same input and writes duplicate output. Idempotent producers on topic B don't help—they only deduplicate retries within the same session.

Transactions solve this by grouping the produced records and the consumed offset commit into a single atomic unit. The application uses a transactional.id (stable across restarts) and the transactional API:

producer.initTransactions();
while (running) {
    ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(100));
    producer.beginTransaction();
    for (ConsumerRecord<K, V> r : records) {
        producer.send(new ProducerRecord<>("topic-B", transform(r)));
    }
    producer.sendOffsetsToTransaction(
        Map.of(new TopicPartition("topic-A", 0), new OffsetAndMetadata(records.lastOffset() + 1)),
        consumer.groupMetadata()
    );
    producer.commitTransaction();
}

The broker writes a transaction marker (commit or abort) to the log. Consumers configured with isolation.level=read_committed only see records from committed transactions. Aborted or in-flight transactions are invisible.

Worked Example: Configuration Comparison

The table below shows the minimal producer configurations for three delivery guarantees. All examples assume Kafka 2.8+ clients and brokers.

Guarantee	Key Configs	Consumer Setting
At-least-once (default)	acks=all, retries=3	isolation.level=read_uncommitted (default)
Idempotent (single-partition, per-session)	enable.idempotence=true (implies safe defaults)	isolation.level=read_uncommitted
Transactional (multi-partition, cross-session)	enable.idempotence=true, transactional.id=app-instance-1	isolation.level=read_committed

Notice that idempotence requires no transactional.id. The PID is generated per session. Transactions require a stable transactional.id per application instance—colliding IDs cause fencing errors that halt the producer.

Trade-offs: Latency, Throughput, and Consumer Behavior

Idempotence adds negligible overhead. The PID and sequence numbers are a few bytes per batch. The broker's in-memory deduplication map is tiny. Throughput and latency are effectively identical to a well-tuned at-least-once producer.

Transactions add measurable cost:

• Coordinator round-trips for beginTransaction, sendOffsetsToTransaction, and commitTransaction
• Transaction markers written to the log (one per partition involved)
• Consumers with read_committed wait for the commit marker before returning records, adding end-to-end latency proportional to transaction duration

In a benchmark on identical hardware (3-broker cluster, 100 MB/s ingest), an idempotent producer sustained ~95 MB/s with p99 latency ~15 ms. A transactional producer on the same workload doing 100-record transactions sustained ~70 MB/s with p99 latency ~45 ms. The gap widens with smaller transactions or higher partition counts.

External side effects (database writes, HTTP calls) are not covered by either mechanism. If your pipeline calls an external API inside the transaction, you've coupled the API's availability to Kafka's commit protocol. The standard pattern: write to Kafka transactionally, then use a separate idempotent consumer to execute side effects with their own deduplication keys.

Verification Checklist Before Shipping

1. Confirm versions. Broker ≥ 2.5, client ≥ 2.5 for stable idempotence. Transactions stable since 0.11 but sendOffsetsToTransaction requires consumer group metadata support (client ≥ 2.5).
2. Audit configs. Search deployment templates for acks, retries, max.in.flight.requests.per.connection overrides. Any override on an idempotent producer causes startup failure.
3. Test idempotence. Run a producer with enable.idempotence=true against a test topic. Kill a broker mid-write to force retries. Use kafka-dump-log --files /var/lib/kafka/data/topic-0/00000000000000000000.log --print-data-log and verify no duplicate offsets or sequence gaps. A consumer counting records should match the logical send count.
4. Test transactions. Abort a transaction deliberately (call abortTransaction() instead of commit). Verify a read_committed consumer never sees the aborted records. Verify a read_uncommitted consumer does see them.
5. Measure latency. Compare end-to-end latency of your consume-transform-produce pipeline with read_committed vs read_uncommitted consumers. If the added latency violates SLAs, consider whether idempotent producers plus application-level deduplication (e.g., deterministic keys + upsert sinks) is sufficient.

Actionable Closing

Start by flipping enable.idempotence=true on every producer in your organization. It's a one-line change with guardrails that prevent misconfiguration. Audit your pipelines for consume-transform-produce patterns—those are the only ones that need transactional.id and read_committed consumers. For everything else, idempotence gives you exactly-once per partition per session at near-zero cost. The duplicates that remain (cross-partition, cross-session) are the ones your application logic should handle with deterministic keys and idempotent sinks.