Guaranteeing One‑Time Delivery in Kafka: How Exactly‑Once Semantics Work in Practice

The Duplicate Message Problem

In a typical event‑driven architecture, a producer writes a record to a Kafka topic and a consumer processes it. If the producer retries after a transient failure, or if the consumer re‑reads a partition after a crash, the same record can be processed twice. The result is duplicate updates, inconsistent state, or double‑charged invoices. Traditional “at‑least‑once” delivery guarantees cannot prevent this without additional idempotence logic in the consumer.

Exactly‑Once Semantics (EOS) in a Nutshell

EOS is Kafka’s built‑in mechanism for eliminating duplicates. It relies on two pieces of configuration:

• Transactional producer – the producer groups records into a transaction that is either committed or aborted.
• Consumer isolation level – the consumer reads only records from committed transactions.

When both sides are configured correctly, every record that reaches a consumer has been committed exactly once, regardless of retries or re‑balances.

How the Transaction Flow Works

1. The producer calls initTransactions() and starts a transaction with beginTransaction().
2. It sends one or more records to one or more partitions.
3. It either calls commitTransaction() or abortTransaction().
4. The broker writes a special Transactional marker to the log. Only after the commit marker is visible to all in‑sync replicas does the transaction become visible.
5. A consumer with isolation.level=read_committed will skip any records from aborted transactions and read only committed ones.

Configuring EOS in a Real Project

Producer Settings

Below is a minimal Java‑style configuration. Replace placeholders with your environment values.

Properties props = new Properties();
props.put("bootstrap.servers", "");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "true");                // Guarantees idempotent writes
props.put("transactional.id", "user-updates-producer");  // Unique per producer instance
props.put("transaction.timeout.ms", "60000");           // 60‑second transaction timeout

After creating the KafkaProducer, call initTransactions() once before sending data. For each batch:

producer.beginTransaction();
producer.send(new ProducerRecord<>("user-updates", key, value));
// … send more records if needed
producer.commitTransaction();

Consumer Settings

The consumer must be configured to read only committed data.

Properties consumerProps = new Properties();
consumerProps.put("bootstrap.servers", "");
consumerProps.put("group.id", "user-updates-consumer-group");
consumerProps.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
consumerProps.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
consumerProps.put("isolation.level", "read_committed");   // Skip uncommitted records
consumerProps.put("enable.auto.commit", "false");

After processing a batch, commit offsets manually:

consumer.commitSync();

Verification Steps

• Run kafka-topics.sh --version to confirm broker version ≥ 0.11.
• Send a batch with commitTransaction() and verify that a consumer with read_committed reads each record exactly once.
• Use kafka-consumer-groups.sh --describe --group user-updates-consumer-group to check that offsets are updated only after the transaction commit.
• Measure latency before and after enabling EOS to quantify the overhead.

Trade‑offs and When to Use EOS

• Latency & Throughput – The transactional log adds metadata and requires replication of commit markers. A large transaction (many records) can increase round‑trip time and CPU usage on the broker.
• Resource Usage – EOS requires the broker to maintain a transaction state log. Set transaction.state.log.min.isr appropriately to balance durability and performance.
• Compatibility – Clients older than 0.11 cannot use transactional IDs. If you need to support legacy producers, keep EOS optional.
• Complexity – You must handle transaction timeouts and retries. A mis‑configured consumer isolation level (e.g., read_uncommitted) negates the benefit.

Actionable Checklist for Production Roll‑out

1. Confirm broker version and enable allow.auto.create.topics if you plan to auto‑create transactional topics.
2. Deploy a test producer/consumer pair with EOS enabled and run a load test that includes retries.
3. Measure the impact on latency and throughput; adjust transaction.timeout.ms and transaction.state.log.min.isr as needed.
4. Update deployment scripts to include enable.idempotence=true and transactional.id for all producers that require EOS.
5. Document the required consumer configuration (isolation.level=read_committed) and enforce it via code reviews.
6. Monitor broker metrics such as transaction_commit_latency and transaction_abort_rate to detect issues early.

Once the above steps are in place, you can confidently rely on Kafka’s Exactly‑Once Semantics to eliminate duplicate processing, simplify consumer logic, and build more resilient event‑driven systems.