When Exactly-Once Works: Idempotent vs Transactional Producers in Kafka

You've built a pipeline that writes orders to a database and an event stream. Duplicate records cause double-charges, broken analytics, or corrupted state. Kafka's exactly-once semantics promise to fix this, but the path isn't one-size-fits-all. This post breaks down when idempotent producers suffice and when you need the transactional API, plus the operational trade-offs that catch teams off guard.

Idempotent Producers: The Low-Friction EOS Path

Since Kafka 3.0, enable.idempotence=true is the default. Idempotent producers assign a producer ID (PID) and sequence numbers per partition. When a broker receives a retry, it deduplicates using the PID and sequence, so your application code doesn't need explicit dedup logic. This covers the common case of "at-least-once with no duplicates" for a single partition.

• Requires max.in.flight.requests.per.connection ≤ 5 (the default). Setting it higher disables idempotence silently in older versions.
• Works per-partition only; cross-partition atomicity still needs the transactional API.
• No transactional.id needed; the broker manages PID lifecycle automatically.

Transactional Producers: Crossing Partitions and Systems

When you need atomic writes across multiple partitions, topics, or external systems (e.g., a database insert plus a Kafka record), use the transactional API. A producer wraps multiple produce() calls in a transaction identified by a stable transactional.id. The broker's transaction coordinator tracks state in the internal __transaction_state topic. Either all records commit, or all abort.

# Example configuration (server.properties)
transactional.id=my-app-producer
enable.idempotence=true

To send a cross-partition transaction:

// Pseudo-code
 txn = producer.beginTransaction()
 producer.send(record, partition=A)
 producer.send(record, partition=B)
 producer.commitTransaction()

If any step fails, call producer.abortTransaction() to leave no partial data.

Consumer Side: read_committed and the Latency Cost

By default, consumers use isolation.level=read_uncommitted, which sees all records including those from in-flight transactions. Switch to isolation.level=read_committed to skip uncommitted records and only see committed data. This is essential for exactly-once guarantees, but it adds latency: records become visible only after the transaction commits, and the consumer must track the last stable offset (LSO) per partition.

• Aborted transaction records are never visible, even after commit.
• In-flight records are hidden until commit, which can increase end-to-end latency for long-running transactions.
• Ensure transaction.state.log.replication.factor (default 3) and min.insync.replicas (default 2) are set appropriately; losing the transaction coordinator partition makes all transactional producers unavailable.

Verification and Practical Checks

To validate EOS behavior in your cluster, follow this pattern (assume Kafka 3.6+ and a three-node cluster):

1. Create a topic with sufficient partitions and replication: kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 3
2. Run a Java producer with idempotence enabled and a stable transactional ID: java -jar producer.jar --enable-idempotence --transactional-id my-app-1
3. Send 100 records in a transaction, kill the producer mid-flow, then consume with isolation.level=read_committed. Verify only committed records appear.
4. Check broker logs for TransactionCoordinator handling InitProducerId, AddPartitionsToTxn, and CommitTransaction requests. Confirm the __transaction_state topic has correct replication.
5. Use kafka-transactions.sh --list to view active, aborted, and completed transactions; verify the producer epoch increments on restart with the same transactional.id.
6. Benchmark throughput with and without transactions using kafka-producer-perf-test --transactional-id test --record-size 1000 --num-records 1000000 to quantify the ~2-5 ms commit latency in your environment.

If records duplicate after a rebalance, confirm your consumer group uses read_committed and that the producer's transactional.id is unique per instance. Reusing the same ID across processes triggers a ProducerFencedException and fences the old producer.

Closing

Idempotent producers give you exactly-once per-partition reliability with a single config flag—ideal for most single-stream pipelines. Reach for the transactional API only when you need atomic writes across partitions or downstream systems, and always pair it with a stable transactional.id, read_committed consumers, and careful broker configuration. Verify the pattern in a staging cluster before promoting to production, and monitor transaction coordinator health to avoid unexpected unavailability.