Diagnosing NotEnoughReplicas Errors and ISR Churn in Apache Kafka

The Problem: Partition Unavailability Despite Active Brokers

When a Kafka producer uses acks=all, it requires all replicas currently in the In-Sync Replica (ISR) list to acknowledge the write. If the ISR shrinks below the min.insync.replicas threshold, Kafka returns a NotEnoughReplicasException. This often happens not because brokers are down, but because replicas are being evicted from the ISR due to transient lag, leading to a state of "ISR churn."

Identifying the Root Cause

ISR churn occurs when replicas repeatedly drop out of and rejoin the ISR list. This is typically driven by a mismatch between network throughput and the replica.lag.time.max.ms setting (the time a follower can be behind the leader before being evicted).

Symptom	Likely Cause	Diagnostic Indicator
Intermittent NotEnoughReplicas	Transient Network Spikes	Frequent Begin ISR / End ISR logs
Consistent Producer Failures	Under-provisioned I/O	High replica lag in kafka-topics.sh
Controller CPU Spikes	High Partition Count + Churn	Increased metadata request latency

Step-by-Step Diagnostic Workflow

Perform these checks on a cluster running Kafka 2.8 through 3.3 to determine why replicas are falling out of sync.

1. Verify Current ISR State

   Run the following command on a client machine with access to the cluster to check the current ISR membership and leader for the affected topic:

   bin/kafka-topics.sh --bootstrap-server <broker-address> --describe --topic <topic-name>

   Check: Compare the Replicas list with the Isr list. If the Isr list is significantly shorter than the Replicas list, you have a synchronization problem.

2. Analyze Broker Logs for Eviction Events

   Search the server.log on the brokers acting as followers for the affected partition. Use a grep command to find ISR transitions:

   grep -E "Begin ISR|End ISR" /var/log/kafka/server.log

   Check: If you see a rapid sequence of End ISR followed by Begin ISR for the same partition, the replica is flapping. This indicates the follower is catching up, but is evicted again before it can stabilize.

3. Measure Replica Lag

   Check the high-water mark (the last offset replicated to all ISR members) versus the log end offset (the last message written to the leader). If the gap is widening despite available bandwidth, check for disk I/O saturation on the follower.

Remediation Strategies

Depending on the findings above, apply the following fixes:

Scenario A: Frequent Flapping (Transient Lag)

If logs show constant ISR membership changes but the brokers are healthy, increase the grace period for followers to catch up. This prevents premature eviction during minor network jitters.

• Action: Increase replica.lag.time.max.ms in server.properties.
• Example: Change from the default 10000 (10s) to 30000 (30s).
• Risk: Increasing this value increases the window where a leader failure could result in data loss if the new leader is slightly behind the old one.

Scenario B: Consistent Under-Replication

If the ISR list remains small and NotEnoughReplicas persists, the followers cannot keep up with the producer's write volume.

• Action: Scale the broker hardware (specifically disk IOPS) or reduce the min.insync.replicas setting for that specific topic to allow for higher availability at the cost of durability.
• Command (Topic Level):

  bin/kafka-configs.sh --bootstrap-server <broker> --alter --entity-type topics --entity-name <topic> --add-config min.insync.replicas=2

Rollback Procedure

If changing replica.lag.time.max.ms causes unacceptable recovery times during leader elections, revert the value in server.properties to the previous setting and perform a rolling restart of the brokers.

Escalation Criteria

Escalate to infrastructure or network teams if:

• ISR churn persists after increasing replica.lag.time.max.ms to 60 seconds.
• Broker logs show NetworkException or TimeoutException during fetch requests between brokers.
• The Controller broker shows sustained CPU usage above 80% specifically during ISR transition events.