Question
How does Helm handle latency that appears only under concurrent install or upgrade commands?
Nia Moss
0 reputation · 04 Jul 2022, 02:35 UTC
97.1K views1
Loading…
0 reputation · 04 Jul 2022, 02:35 UTC
Goal: Determine whether Helm v3 can reduce latency spikes observed when multiple helm install or helm upgrade commands run concurrently without relying on external serialization.
Constraints/uncertainty: The observed latency stems from Kubernetes API server queuing, etcd lock contention, and overlapping exponential backoff retries; it is unclear if adding a built‑in concurrency limit would conflict with Helm’s stateless design or affect existing automation workflows.
Specific questions: Does the Helm community consider adding a configurable concurrency‑limit flag to helm install/helm upgrade? What trade‑offs (e.g., reduced parallelism, API‑server load) would such a flag introduce? How could users validate the effectiveness of internal throttling versus external tooling in their own clusters?
34275 reputation · 04 Jul 2022, 07:03 UTC
Helm v3 does not include any built‑in throttling or concurrency‑limit mechanism for helm install or helm upgrade. Latency that appears only when several commands run at the same time originates from the Kubernetes API server (and its etcd backend), not from Helm itself.
helm
kubernetes--wait flag is used, Helm polls the status of resources; concurrent polls increase load on the API server and can lengthen the observed wait time.Under concurrent helm install/upgrade invocations, many API requests arrive at the API server almost simultaneously. The server (and etcd) must serialize writes, leading to increased request processing time, possible throttling, and exponential back‑off retries. Helm surfaces this as longer command duration because it waits for each request to complete.
helm install --debug (or upgrade) and note the round‑trip time of API calls from the debug output.apiserver_request_latencies_seconds and apiserver_request_total (throttling sub‑metrics) during the parallel run.--wait or when targeting different releases, the cause is API server load rather than Helm internal processing.--wait when immediate status is not required.Use comments to ask for clarification. Post a solution as an answer.
34,275 reputation · 04 Jul 2022, 13:36 UTC
To clarify the relationship between latency and concurrency, it is important to note that Helm v3 relies on optimistic concurrency control. When Helm updates a release, it reads the current state (stored as a Secret or ConfigMap), modifies it, and sends it back with a specific resourceVersion.
If multiple concurrent processes attempt to update the same release, the Kubernetes API server will reject any request with an outdated version via a 409 Conflict error. While the previous response noted that this causes failure, the perceived "latency" often occurs when automation wrappers or CI/CD pipelines implement their own retry loops around these failures. This creates a "thundering herd" effect where multiple clients repeatedly poll and retry against the API server, compounding etcd contention.
To verify if 409 Conflict errors are driving your latency, monitor the API server logs for HTTP 409 responses during your concurrent runs. If these are frequent, the bottleneck is the state lock on the release itself, not general API server load.