Choosing a Kafka Producer Acknowledgment Strategy for Data Durability

The Trade-off Between Throughput and Durability

When configuring a Kafka producer, the acks setting determines how many broker replicas must receive a record before the producer considers the write successful. Choosing the wrong setting leads to either silent data loss during broker failures or unnecessary latency that throttles your ingestion pipeline.

The core decision depends on your recovery point objective (RPO): can your business tolerate the loss of a few seconds of data in exchange for higher performance, or is every single message critical?

Comparing Acknowledgment Levels

Setting	Wait Condition	Durability Risk	Latency
acks=0	None (Fire and forget)	High (Loss on broker crash)	Lowest
acks=1	Leader replica only	Medium (Loss if leader fails before replication)	Medium
acks=all (or -1)	Full In-Sync Replica (ISR) set	Lowest (Safe as long as one ISR survives)	Highest

The Critical Link: acks=all and min.insync.replicas

Setting acks=all does not guarantee durability on its own. It only ensures that the current In-Sync Replicas (ISR)—the set of replicas that are caught up with the leader—have acknowledged the write. If the ISR set shrinks to only the leader, acks=all behaves identically to acks=1.

To enforce a minimum level of redundancy, you must configure the broker-side setting min.insync.replicas. This defines the minimum number of replicas that must acknowledge a write for it to be successful when acks=all is used. If the number of active replicas falls below this threshold, the broker will reject the write with a NotEnoughReplicasException.

Decision Matrix

• Use acks=0 for high-volume telemetry or logging where individual missing data points do not impact the overall trend.
• Use acks=1 for general-purpose streams where performance is priority, but you want to know if the leader is completely unreachable.
• Use acks=all + min.insync.replicas=2 for financial transactions, order processing, or any system where data loss is unacceptable.

Implementation and Validation

To implement a high-durability configuration, apply changes to both the producer client and the broker configuration. This example assumes a cluster with a replication factor of 3.

1. Broker Configuration

Modify the server.properties file on all brokers or update the topic-specific configuration using the Kafka CLI. This ensures at least two replicas must be in sync for a write to succeed.

# Run on a Kafka broker or via admin client
kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name my-critical-topic --alter --add-config min.insync.replicas=2

2. Producer Configuration

Set the producer to wait for all synchronized replicas. In a Java client, this is configured in the properties map:

properties.put(ProducerConfig.ACKS_CONFIG, "all");
properties.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
properties.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");

Note: Enabling enable.idempotence=true is strongly recommended with acks=all to prevent duplicate messages caused by producer retries during transient network failures.

3. Verifying the Durability Guarantee

To verify that your configuration prevents data loss at the cost of availability, perform the following diagnostic test in a staging environment:

1. Start a 3-node cluster with min.insync.replicas=2.
2. Produce messages using acks=all. The producer should succeed.
3. Stop one broker. The ISR set drops to 2. The producer should still succeed because the minimum threshold (2) is met.
4. Stop a second broker. The ISR set drops to 1.
5. Attempt to produce another message. The producer should receive a NotEnoughReplicasException (or a timeout depending on retry settings).

Limitations and Risks

Increasing the acknowledgment level introduces specific operational risks:

• Availability Risk: With min.insync.replicas=2, you cannot write to the topic if two out of three brokers are down, even if one broker is perfectly healthy. You are trading availability for consistency.
• Latency Spikes: acks=all is sensitive to the slowest replica in the ISR set. If one follower experiences disk I/O contention, the producer's end-to-end latency will increase.
• Network Overhead: Higher acks settings increase the volume of acknowledgment traffic between brokers and the producer.