K3os kubelet connection pool exhaustion under high API server load

Explanation

Under sustained API server traffic the default client-go connection pool in kubelet (5 connections per host) can be exhausted, causing intermittent "context deadline exceeded" errors while the API server remains reachable.

Confirmed facts from the research

• k3os honors kubelet client-go QPS and burst settings via its configuration mechanism: either a kubelet: section in /etc/rancher/k3os/config.yaml or the environment variable KUBELET_EXTRA_ARGS.
• Log signatures of pool exhaustion include repeated messages such as:
• context deadline exceeded
• transport: while waiting for connection: context deadline exceeded
• client-go: transport: http2: received GOAWAY
• The API server itself does not show TLS handshake or connection‑refused errors in these cases.
• Containerd connection‑pool saturation can be monitored by enabling its metrics endpoint (metrics.address = \"0.0.0.0:10010\") and watching Prometheus metrics like grpc_server_msg_received_total and grpc_server_handled_total for steady growth without corresponding drops.

Steps to tune kubelet client‑go settings in k3os

1. Edit the k3os configuration file:

sudo vi /etc/rancher/k3os/config.yaml

2. Add or modify the kubelet section to include the desired QPS and burst values (example values; adjust based on observed load):

kubelet:
  extra_args:
    kubelet-client-qps: "20"
    kubelet-client-burst: "30"

3. If you prefer using an environment variable, set it in the same file or via cloud‑init:

KUBELET_EXTRA_ARGS="--kubelet-client-qps=20 --kubelet-client-burst=30"

4. Restart the kubelet service to apply the changes:

sudo systemctl restart kubelet

5. Verify the flags are present in the running process:

ps -ef | grep kubelet

Verification that the exhaustion is mitigated

1. Check kubelet logs for the absence of the pool‑exhaustion patterns:

journalctl -u kubelet | grep -E 'context deadline exceeded|GOAWAY'

2. Generate sustained API load (e.g., kubectl get pods -A --watch or a load‑generator) and confirm the log lines no longer appear.
3. If you enabled containerd metrics, scrape the endpoint and observe that grpc_server_msg_received_total and grpc_server_handled_total increase smoothly under load without sudden spikes or plateaus.

Missing diagnostic detail that could change the recommendation

Are you seeing the specific log pattern context deadline exceeded paired with GOAWAY messages while the API server remains reachable? If the logs instead show connection‑refused or TLS handshake failures, the issue may be API server availability rather than client‑go pool exhaustion, and tuning QPS/burst would not be the appropriate fix.