Can acks=all cause production‑cluster failures when ISR shrinks?

Answer

Yes. With acks=all the producer waits for acknowledgment from every broker that is currently in the ISR for a partition. If a follower falls out of the ISR, the ISR size drops. When the ISR size becomes smaller than min.insync.replicas, the broker returns NotEnoughReplicasException and the producer times out.

Confirmed facts

• acks=all requires acknowledgment from all in‑sync replicas.
• The broker only considers a replica in‑sync if it has not lagged beyond replica.lag.time.max.ms.
• If the ISR size ≥ min.insync.replicas, the write succeeds; otherwise the broker rejects it with NotEnoughReplicasException.

Steps to diagnose and mitigate for this case

1. Check the current ISR size for the affected topic partitions during a failure (e.g., via kafka-topics --describe or JMX metric kafka.server:type=ReplicaManager,name=IsrShrinkRate).
2. Compare that ISR size to the configured min.insync.replicas (2 in your setup).
3. If ISR size < 2, the follower is lagging or unavailable; improve follower performance (increase replica.lag.time.max.ms, fix network/disk issues, or add brokers) so the follower can stay in sync.
4. If ISR size is consistently 2 but you still see transient exceptions, consider raising replica.lag.time.max.ms to give followers more time to catch up, monitoring that the value does not hide real lag.
5. As a last resort, lower the producer’s acks to 1 to avoid waiting for followers, accepting that a leader crash could lose the last un‑replicated message.

Missing diagnostic detail

To decide whether adjusting min.insync.replicas or follower lag tuning is appropriate, we need to know the observed ISR size when the NotEnoughReplicasException occurs. Please provide that value.