Choosing Kafka Producer Acks: Balancing Latency and Durability

Problem: How to decide the producer acks level?

When a Kafka producer sends a record it can wait for different numbers of broker replicas to confirm the write. The acks setting controls this. Choosing acks=0, acks=1, or acks=all changes end‑to‑end latency, throughput, and the likelihood that a record will survive a broker failure. The decision is usually driven by the application’s tolerance for lost messages versus the need for high write throughput.

Decision Matrix

Setting	What is acknowledged	Typical latency impact	Durability risk
acks=0	No broker response is awaited.	Lowest; no round‑trip.	Highest: a broker crash immediately after the request can lose the record.
acks=1	The leader broker writes the record to its local log.	Low; one network round‑trip.	Moderate: if the leader fails before the follower catches up, the record may be lost.
acks=all (or -1)	All replicas currently in the in‑sync replica set (ISR) acknowledge.	High; requires round‑trips to all ISR members.	Lowest, provided min.insync.replicas is set > 1; otherwise it behaves like acks=1 if only the leader remains in the ISR.

Key Constraint: min.insync.replicas

The broker‑side or topic‑level setting min.insync.replicas defines the minimum number of replicas that must be in the ISR for a write to succeed when acks=all is used. If the ISR shrinks below this threshold, the producer receives a NotEnoughReplicasException. The default value is 1, which means that without explicit configuration acks=all is effectively the same as acks=1.

Because min.insync.replicas is available in all actively maintained Kafka versions (≥ 0.10), the guidance applies across current releases, not just 2.8+.

Concrete Implementation

Assume a cluster with three brokers and a topic that has a replication factor of 3. The goal is to guarantee that at least two replicas store the record before the producer is informed of success.

1. Broker‑ or Topic‑level Configuration

Set min.insync.replicas=2 on the topic. The command can be run from any machine that can reach the cluster and has the kafka-configs tool. Run as a user with cluster‑admin rights.

kafka-configs \
  --bootstrap-server broker1:9092 \
  --entity-type topics \
  --entity-name my‑topic \
  --alter \
  --add-config min.insync.replicas=2

Check the change:

kafka-configs \
  --bootstrap-server broker1:9092 \
  --entity-type topics \
  --entity-name my‑topic \
  --describe

2. Producer Configuration

Configure the producer to wait for full ISR acknowledgment and enable retries so that transient failures are retried automatically.

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092,broker3:9092");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, 3);
props.put(ProducerConfig.LINGER_MS_CONFIG, 10);   // optional: batch small records
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 32 * 1024); // 32 KiB
KafkaProducer<String, String> producer = new KafkaProducer<>(props);

3. Validation Test

1. Start the cluster with the topic created as described above.
2. Send a batch of messages with the producer configured for acks=all.
3. Shut down two of the three brokers so that only one replica remains in the ISR.
4. Attempt to send another message. The producer should throw a NotEnoughReplicasException (or the equivalent error in the client library), indicating that the write would not meet the min.insync.replicas requirement.
5. Restart one of the stopped brokers so that the ISR contains at least two replicas again, and verify that subsequent sends succeed.

During the test you can capture the producer’s recordMetadata timestamps or use a benchmarking tool such as kafka-producer-perf-test to measure the latency difference between acks=1 and acks=all under identical batch sizes.

Trade‑Off Summary

• acks=0 is suitable for high‑volume telemetry where occasional loss is acceptable and latency must be minimal.
• acks=1 offers a good compromise for most applications that need reasonable durability without a large latency penalty.
• acks=all with min.insync.replicas>1 is required for mission‑critical data (e.g., financial transactions, audit logs) where you cannot tolerate a single‑broker loss.

Performance Mitigation

Because acks=all forces the producer to wait for all ISR members, throughput can drop sharply if batch sizes are small. Increase batch.size and linger.ms to let the producer send larger packets, thereby amortizing the round‑trip cost. Monitor the producer’s recordsSent and requestLatencyMs metrics to confirm that the trade‑off is acceptable for your workload.

Limitations and Caveats

• The configuration assumes the topic’s replication factor is at least equal to min.insync.replicas; otherwise the broker will reject the topic creation outright.
• If the cluster is partitioned and the ISR shrinks to a single broker, acks=all behaves like acks=1 unless min.insync.replicas is greater than 1.
• Setting min.insync.replicas too high can cause writes to fail during normal transient network glitches, so adjust the value with an understanding of your network reliability.

Practical Check

After configuring, run the validation test above. A successful test is indicated by:

• Messages sent while at least two replicas are available are acknowledged with a RecordMetadata object.
• Messages sent when only one replica remains result in a NotEnoughReplicasException.

These observations confirm that the broker‑side min.insync.replicas setting is active and that the producer’s acks level is behaving as expected.