Getting Exactly‑Once Guarantees in Kafka: How to Enable and Test EOS

The Problem: Duplicate Messages in a Distributed System

In a micro‑service architecture, a single event can trigger several downstream jobs. If a message is written twice to a topic, the consumer may process it twice, leading to duplicate records, double charges, or inconsistent state. Traditional at‑least‑once delivery guarantees in Kafka mean that duplicates are possible, especially after a crash or network hiccup. Engineers have long asked: can we make sure a record is processed exactly once?

Exactly‑Once Semantics – The Core Idea

Kafka’s Exactly‑Once Semantics (EOS) is a feature that combines two concepts:

• Idempotent producers – the producer can retry sending the same record without creating duplicates.
• Transactional consumers – the consumer can commit offsets atomically with the produced records.

When both sides participate, the broker can guarantee that each record is delivered and processed once, even if the producer or consumer crashes mid‑transaction.

How EOS Works Internally

EOS relies on a transaction log that brokers maintain. The producer marks a group of sends as a transaction and writes them to the log. The broker records the transaction’s start and end offsets. A consumer that reads with isolation.level=read_committed only sees records from fully committed transactions. If the producer aborts, the broker discards the records and no offset is committed for them.

Key points:

• The broker must support EOS (Kafka 2.4+).
• All brokers in the cluster must be upgraded; otherwise, a producer may fail to start a transaction.
• Transactions introduce an extra round‑trip: the producer must query the broker for the next transaction ID before sending.

Enabling EOS in Your Producer and Consumer

Below is a minimal Java‑style configuration that turns on EOS. Adapt the properties to your language or client library.

Producer Configuration

Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "true");          // Required
props.put("transactional.id", "my-unique-transactional-id"); // Required
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();

Key points:

• enable.idempotence must be true to guarantee per‑record idempotency.
• transactional.id uniquely identifies the producer instance; reuse the same ID across restarts to resume the transaction.
• Call initTransactions() once after construction.

Consumer Configuration

Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092");
props.put("group.id", "my-consumer-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("isolation.level", "read_committed"); // Required
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("my-topic"));

Only consumers with read_committed can see EOS‑protected data.

A Concrete Example: Sending a Transactional Message

Below is a step‑by‑step snippet that writes a single record inside a transaction and commits it. If the process crashes after beginTransaction() but before commitTransaction(), the broker will discard the record.

producer.beginTransaction();
try {
    producer.send(new ProducerRecord<>("my-topic", "key1", "payload"));
    // Simulate some work
    Thread.sleep(100);
    producer.commitTransaction();
} catch (Exception e) {
    producer.abortTransaction();
    throw e;
}

Verification steps:

1. Run the snippet and observe the consumer logs – you should see the record once.
2. Restart the producer with the same transactional.id and send a second record inside a new transaction. The consumer should see both records exactly once.
3. To test failure handling, start a transaction, then kill the JVM before commitTransaction(). Restart the producer and send another record. The consumer should only see the new record, confirming the aborted transaction was discarded.

Trade‑offs and Limitations

• Latency – every transactional send requires a round‑trip to obtain the next transaction ID and to commit the transaction. Expect ~10–20 ms extra latency per batch.
• Throughput – the broker’s transaction log consumes additional disk I/O and log compaction. Under heavy load, EOS may reduce overall throughput by 10–15 %.
• Cluster‑wide upgrade – EOS requires all brokers to be at least Kafka 2.4. Mixing versions can cause transaction failures.
• Consumer rebalances – EOS consumers must handle rebalances carefully. The consumer must not commit offsets until the transaction is fully committed; otherwise, duplicates can appear.
• Idempotent producer limits – the producer’s max.in.flight.requests.per.connection must stay below 5 to maintain idempotence. This may limit parallelism.

Actionable Checklist

1. Upgrade all brokers to a version that supports EOS (≥ 2.4, preferably 3.x for lower overhead).
2. Enable enable.idempotence=true and set transactional.id on every producer that needs EOS.
3. Configure consumers with isolation.level=read_committed and ensure they commit offsets only after the transaction is committed.
4. For Kafka Streams, set StreamsConfig.PROCESSING_GUARANTEE_CONFIG=exactly_once_v2 and verify the underlying producer/consumer settings.
5. Run a smoke test: send a message in a transaction, kill the producer mid‑transaction, restart, and confirm no duplicate appears.
6. Monitor broker metrics: kafka.server.ProducerTransactionCount, kafka.server.TransactionAbortCount, and consumer lag to spot anomalies.
7. Adjust max.in.flight.requests.per.connection and batch size to balance latency and throughput.

How to Verify EOS in Production

After deploying EOS, perform the following checks:

• Consumer logs: verify that every message appears exactly once. Use a checksum or hash of the payload and compare across consumer instances.
• Broker logs: look for TransactionCommit and TransactionAbort entries. A high abort count may indicate misconfiguration.
• Run a kafka-consumer-groups.sh --describe to confirm group offsets match the committed transaction offsets.
• Simulate a crash: send a batch, kill the producer, restart, and confirm the aborted batch is not visible to consumers.

By following this guide, you can confidently add exactly‑once guarantees to your Kafka pipelines, reduce downstream duplicate processing, and maintain high data integrity without sacrificing too much performance.