Achieving Exactly-Once Delivery in Apache Kafka with Idempotent Producers and Transactions

Problem and Takeaway

Duplicate records appear when a producer retries after a network glitch or a broker fails mid‑write. Kafka solves this by combining idempotent producers (automatic deduplication per partition) with the transactional API (atomic multi‑partition writes). Enable both and you get exactly‑once semantics without application‑level deduplication.

How It Works

Producer ID and Sequence Numbers

When enable.idempotence=true (default since Kafka 3.0) the broker assigns each producer a unique PID. Every record sent to a partition carries a monotonically increasing sequence number. The broker stores the highest committed sequence number per PID in the partition log and discards any later record with a duplicate or lower sequence number.

Transactional Boundaries

A transactional.id ties a logical producer instance to its PID across restarts. The broker keeps the PID‑to‑sequence mapping for transactional.id.expiration.ms (default 7 days). Calls to initTransactions(), beginTransaction(), send(), and commitTransaction() write transaction markers (commit/abort) into the log. Consumers with isolation.level=read_committed only advance past committed markers.

Worked Java Configuration

Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092");
props.put("enable.idempotence", true);                 // default in 3.0+
props.put("transactional.id", "txn-" + appInstanceId); // unique per logical producer
props.put("acks", "all");                             // enforced by idempotence
props.put("max.in.flight.requests.per.connection", 5); // enforced by idempotence

KafkaProducer producer = new KafkaProducer<>(props);
producer.initTransactions();

try {
    producer.beginTransaction();
    producer.send(new ProducerRecord<>("orders", key, value));
    producer.send(new ProducerRecord<>("inventory", key, value));
    producer.commitTransaction();
} catch (Exception e) {
    producer.abortTransaction();
    throw e;
} finally {
    producer.close();
}

Each logical producer instance must use a distinct transactional.id. Reusing the same ID concurrently raises ProducerFencedException.

Limits and Common Mistakes

• Cross‑cluster exactly‑once is not provided; MirrorMaker 2 runs independent transactions per cluster.
• Downstream duplicates remain possible — consumers or external systems still need idempotent processing or deduplication.
• Transaction timeout (transaction.timeout.ms, default 1 min) must exceed the longest processing window; too short aborts valid work, too long delays broker cleanup.
• Consumer latency rises with read_committed because the consumer waits for commit markers.
• PID expiration: a producer idle longer than transactional.id.expiration.ms loses its PID and may emit duplicates on restart.
• Ordering across partitions is not guaranteed; use a single partition or keyed messages for per‑key order.

Verification Steps

1. Unit test: send 10 000 records in a transaction, kill the producer mid‑transaction, restart with the same transactional.id, and consume with isolation.level=read_committed. Expect zero duplicates.
2. Broker log check: grep for Completed transaction and Aborted transaction markers; confirm transactional.id.expiration.ms matches retention policy.
3. Metrics: monitor JMX producer-metrics:record-send-rate, request-latency-avg, transaction-commit-rate to quantify overhead (typically 3‑10 % throughput loss).
4. Consumer offset inspection: use kafka-consumer-groups.sh --describe --group my-group and verify fetched offsets never pass uncommitted transaction markers.
5. Performance benchmark: run kafka-producer-perf-test --topic test --num-records 1000000 --record-size 1000 --throughput -1 --producer.config producer.properties --transactional-id bench-txn with and without enable.idempotence to measure the difference.

Quick Comparison

Tune batch.size and linger.ms as usual; they still apply on top of the idempotence overhead.