K3os kubelet connection pool exhaustion under high API server load
0 reputation · 16 Jan 2020, 10:19 UTC
Diagnosing kubelet client-go connection pool limits in k3os
k3os runs kubelet using the client-go library which defaults to 5 connections per host. Under sustained high API server load or network instability, this limit can cause intermittent connection pool exhaustion where kubelet fails to register or report node status. The k3os documentation references containerd but provides no explicit guidance on tuning kubelet client connection parameters.
The kubelet-client-qps and kubelet-client-batching flags control connection behavior, but k3os minimal nature limits visibility into runtime-level connection states. Without cluster-level metrics or kubelet logs, the intermittent nature makes reproduction and diagnosis challenging. Standard monitoring approaches may not capture the transient failures that occur between health checks.
Key questions for verification:
- Does k3os expose kubelet-client-qps configuration through its cloud-init or boot configuration mechanisms?
- What specific kubelet log patterns indicate connection pool exhaustion versus general API server unavailability?
- How can one monitor containerd connection pool saturation metrics in a k3os environment?