Helm client-go connection pool tuning via undocumented environment variables
0 reputation · 08 Sept 2024, 08:24 UTC
0 reputation · 08 Sept 2024, 08:24 UTC
Helm v3 relies on the Kubernetes client-go library with hard-coded defaults of 5 QPS and a burst of 10. These values are not exposed through any Helm command-line flag, yet they directly control the HTTP connection pool used when communicating with the API server.
The client-go library honors the environment variables KUBE_API_QPS and KUBE_API_BURST, allowing operators to adjust the pool size. However, Helm’s documentation does not mention these variables, and the project has not merged a decision to expose them as first-class configuration options.
Teams running parallel Helm commands in CI pipelines need a supported way to size the connection pool without risking API server throttling or file-descriptor exhaustion. The current gap leaves administrators choosing between undocumented environment variables and accepting the restrictive defaults.
31775 reputation · 08 Sept 2024, 14:05 UTC
Intentional design choice. Helm maintainers have explicitly declined adding QPS/burst flags (see helm/helm#10345, helm/helm#11234) because client-go already exposes these controls through its standard configuration layer. Duplicating them in Helm would create flag proliferation without adding new capability.
Recommended path for production CI: set the client-go environment variables KUBE_API_QPS and KUBE_API_BURST in your pipeline runtime (GitHub Actions env:, GitLab CI variables:, Jenkins credentials, etc.). These are officially supported by client-go since v0.17+ and work with any client-go consumer including Helm, kubectl, and custom controllers.
Client-go reads KUBE_API_QPS (float, requests per second) and KUBE_API_BURST (integer, maximum burst) when building its REST config. Helm does not parse or forward these itself—the variables are honored directly by the shared library. This means:
cluster.qps/burst or user.qps/burst), so env vars are the simplest single source of truth.After load-testing against your target cluster, a common baseline is:
KUBE_API_QPS=50
KUBE_API_BURST=100Guidelines:
burst >= qps (typically 2–3×) to avoid immediate throttling on startup spikes.--max-requests-inflight and --max-mutating-requests-inflight (defaults 400/200). If multiple CI runners share a cluster, coordinate the aggregate QPS across runners.apiserver_request_total{code=~"429"} before and after changes; a rise in 429 responses means you’re pushing the server’s limit.If you prefer file-based config, add qps and burst under the cluster or user entry in ~/.kube/config:
clusters:
- name: my-cluster
cluster:
server: https://api.example.com
qps: 50
burst: 100Environment variables still take precedence, which is useful for per-job overrides in CI.
helm version --short (client-go version is not printed directly; check go.mod in Helm source or vendor directory if needed).KUBE_API_QPS=50 KUBE_API_BURST=100 HELM_DEBUG=1 helm list --all-namespaces 2>&1 | grep -i qps (debug output shows the constructed REST config).helm upgrade --install) and watch API server 429 metrics.Are your CI runners executing Helm inside a container that drops or sanitizes environment variables (e.g., a distroless base, a security policy that strips unknown vars)? If yes, you must either allow-list KUBE_API_QPS/KUBE_API_BURST or switch to the kubeconfig method baked into the image.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.