Question
NotEnoughReplicasException during acks=all production deployment
Dev Moss
0 reputation · 24 Jan 2020, 22:38 UTC
72.7K views
Loading…
0 reputation · 24 Jan 2020, 22:38 UTC
In a production Kafka cluster (version 3.x), a producer is configured with acks=all to ensure maximum data durability. While this configuration functions without issue in a local single-node environment, it triggers a NotEnoughReplicasException when deployed to a multi-broker production cluster.
The cluster is configured with a specific min.insync.replicas threshold. There is uncertainty regarding the exact interaction between the available synchronized replicas and the producer's request when a subset of brokers becomes unavailable or lags behind the leader.
min.insync.replicas setting and the acks=all producer requirement?31275 reputation · 25 Jan 2020, 03:40 UTC
The NotEnoughReplicasException is thrown when the producer’s acks=all requirement cannot be satisfied because the current In‑Sync Replica (ISR) set size is smaller than the topic’s min.insync.replicas threshold.
Even if the leader broker is alive and accepting requests, enough followers may have fallen out of the ISR (due to lag, network issues, or being offline). When the ISR size drops below min.insync.replicas, the broker rejects the acks=all produce request with the exception.
apachekafkaacks=all (or acks=-1) requires the leader to wait until all replicas that are currently in the ISR have written the record.min.insync.replicas defines the minimum number of ISR replicas that must be present for the broker to consider the request successful.min.insync.replicas, the broker returns NotEnoughReplicasException regardless of the leader’s availability.min.insync.replicas value:
kafka-topics.sh --describe --topic <topic-name> --bootstrap-server <broker>:9092
min.insync.replicas.min.insync.replicas:
min.insync.replicas (only if your durability policy allows) or increasing the replication factor/adding brokers.min.insync.replicas, resume producing with acks=all; the exception should no longer appear.To confirm whether the ISR size is the cause, please provide the current ISR size for the topic as shown by the kafka-topics --describe output.
Use comments to ask for clarification. Post a solution as an answer.
31,275 reputation · 25 Jan 2020, 10:09 UTC
One detail worth adding: the ISR doesn't shrink instantly when a follower stalls. A replica is only expelled after it falls outside replica.lag.time.max.ms (default 30s in Kafka 3.x), so a brief broker restart may not trigger NotEnoughReplicasException at all, while a slow-but-alive follower eventually will. This explains why the exception can appear intermittently during rolling deployments rather than immediately.
Also note the producer's retry behavior. NotEnoughReplicasException is retriable, so with default retries the client may silently absorb short ISR dips and only surface the exception after delivery.timeout.ms expires. If you're seeing it in logs but not as a fatal send failure, that's likely why.
For verification, watch the JMX metric kafka.cluster:type=Partition,name=InSyncReplicasCount per partition alongside UnderMinIsrPartitionCount on the controller — the latter directly counts partitions below min.insync.replicas and confirms the condition before you tune anything.