Diagnosing Intermittent Connection Exhaustion in Kafka Clients with connections.max.idle.ms

Goal

Identify why a Kafka client intermittently exhausts its broker connections under sustained load.

Constraints and Uncertainty

The client keeps a single TCP socket per broker node. The connections.max.idle.ms setting (default 30 000 ms in Kafka 2.8+) dictates when an idle socket is closed. When many concurrent producers or consumers issue requests, the socket may be closed before a new request re‑establishes it, leading to repeated connect/disconnect cycles. The broker imposes a max.connections.per.ip limit that can be hit if many clients share the same IP. It is unclear whether the client‑side idle timeout or the broker‑side limit is the primary cause of the observed exhaustion, and how to distinguish the two in production.

Specific Questions

• Under what conditions does the Kafka client close an idle socket defined by connections.max.idle.ms, and how does that affect high‑rate request streams?
• When the broker reaches its max.connections.per.ip threshold, what observable behavior appears on the client side (e.g., connection failures, retries, latency spikes)?
• Which JMX metrics or debug logs can reliably show the count of open versus idle connections on the client, and how can they be correlated with broker connection limits?