Stopping Duplicate Messages with Kafka Idempotent Producers

The Cost of Retries

In a distributed system, the "network glitch" is a constant. When a Kafka producer sends a message and the broker fails to send an acknowledgment (ACK) due to a transient network timeout—even if the broker actually wrote the data to disk—the producer's default reaction is to retry. This creates a duplicate: the same message exists twice in the partition, leading to skewed analytics, double-billing, or corrupted state in your downstream consumers.

The solution is the Idempotent Producer. This feature ensures that exactly one copy of a message is written to the log, regardless of how many times the producer retries the request.

How Idempotence Works Under the Hood

Kafka achieves idempotence by assigning a unique identity to every producer session and tracking the order of messages. This involves two primary identifiers:

• Producer ID (PID): A unique ID assigned by the broker when the producer first initializes.
• Sequence Number: A monotonically increasing integer assigned to every message sent to a specific partition.

When the broker receives a message, it checks the PID and the sequence number. If the sequence number is exactly one greater than the last recorded number for that PID, the broker accepts it. If the sequence number is less than or equal to the last recorded value, the broker recognizes it as a duplicate and discards it, while still sending a successful ACK back to the producer so the producer can move on.

Implementing the Configuration

For modern Kafka clients (version 3.0+), idempotence is enabled by default. However, if you are managing legacy configurations or need to explicitly ensure the guarantee is active, you must align these three settings in your producer properties:

# Required configuration for idempotent delivery
enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5

Configuration Requirements and Risks

Setting	Requirement	Risk of Misconfiguration
acks	Must be all	Setting to 1 or 0 disables the safety guarantee.
max.in.flight...	Must be ≤ 5	Setting this above 5 will silently disable idempotence in older versions or cause errors.
retries	High value (e.g., Integer.MAX_VALUE)	Low retry counts may cause the producer to give up before the broker recovers.

Practical Verification

To verify that your producer is successfully filtering duplicates, you cannot simply look at the logs; you must monitor the broker's internal metrics. The most reliable indicator is the record-duplicates-total JMX metric.

Diagnostic Workflow:

1. Deploy a Kafka cluster (e.g., using confluentinc/cp-kafka:7.5.0).
2. Initialize a Java producer with enable.idempotence=true.
3. Start producing a high volume of messages.
4. Force a failure by stopping the leader broker for a specific partition while the producer is active.
5. Restart the broker and observe the record-duplicates-total metric via JMX or Prometheus. A non-zero value confirms the broker caught and discarded retried messages that had already been written.

The Boundary of Idempotence

It is critical to understand that idempotence is per-partition. It prevents duplicates caused by retries for a single producer sending to a single partition. It does not provide the following:

• Cross-Partition Atomicity: If you need to write to Topic A and Topic B as a single atomic unit, idempotence is insufficient. You must use the Transactional API.
• Application-Level Duplicates: If your application logic accidentally calls producer.send() twice for the same event, Kafka sees these as two different messages with two different sequence numbers and will write both.

Summary Checklist

When auditing your producer configuration for data integrity, ensure enable.idempotence=true is set and verify that max.in.flight.requests.per.connection does not exceed 5. If your pipeline requires absolute consistency across multiple topics, move beyond idempotence and implement Kafka Transactions.