Question
Global Parallelism Caps vs Provider Retry Tuning for Terraform Connection Exhaustion
Ira Dey
0 reputation · 16 Jun 2024, 21:09 UTC
115.5K views1
Loading…
0 reputation · 16 Jun 2024, 21:09 UTC
Terraform core drives applies with a default concurrency of 10, adjustable only through the global -parallelism flag. Each provider process owns its HTTP connection pool, and Terraform exposes no settings for that pool's sizing. In large multi-resource configurations, worker concurrency can outpace what a provider's HTTP client handles cleanly, producing intermittent connection failures that often succeed on a later retry.
That leaves two documented levers, and they work at different layers:
-parallelism (for example to 3–5) caps concurrency globally, but slows every provider equally, including lightweight ones such as null or local that sit far from their own limits.max_retries and retry_mode absorbs transient failures after they occur, but does nothing to prevent the initial connection spike that causes them, and can mask contention in pipelines that pass on re-run.For a configuration dominated by one cloud provider, which lever should anchor the stability strategy: a low global parallelism cap, or default parallelism with aggressive provider retries? Does any current Terraform release offer per-provider concurrency control? And how should the choice account for Terraform Cloud or Enterprise agents that enforce their own parallelism defaults?
For a configuration dominated by a single cloud provider, anchor the stability strategy on the global -parallelism flag: tune it so the number of concurrent provider processes stays within the provider's HTTP connection-pool capacity. Keep provider-specific retry settings at their defaults or only modestly increased to absorb occasional transient errors. High retries alone do not prevent the initial connection spike that creates the failures.
terraform-parallelism flag (or the TF_PARALLELISM environment variable). No stable release up to Terraform 1.5.x offers a per-provider concurrency setting.max_retries and retry_mode) affect only how a plugin process recovers from transient HTTP errors; they do not reduce the number of simultaneous connections opened.When many resources are managed by the same provider, each concurrent Terraform worker drives a provider plugin process with its own HTTP connection pool. If global parallelism exceeds the provider's pool capacity, excess workers contend for sockets, producing intermittent connection-reset or timeout errors that often succeed on retry. Lowering -parallelism directly reduces simultaneous pools and prevents the contention. Raising retries only masks the symptom; the spike remains and can hide capacity problems that would surface under sustained load.
HTTPClient.ConnectionCount, or the provider's equivalent). Without that metric, run a short test at the default parallelism of 10 and note the error rate.-parallelism=4 or -parallelism=6, via the flag or TF_PARALLELISM.max_retries = 3, retry_mode = "standard") or raise them only slightly (e.g., to 5) if you see occasional transient spikes unrelated to parallelism.TF_LOG=TRACE terraform apply -parallelism=4 and count simultaneous plugin invocations in the log, or watch the provider's connection metric during the apply.agent.conf) for the parallelism setting. The agent's value supersedes both the workspace setting and the CLI flag when the agent runs the job.If you cannot observe the provider's connection-pool metric, provide the approximate error rate at the default parallelism of 10 (e.g., what percentage of resources fail with a connection timeout on the first attempt). That determines whether a lower cap is needed or the default is already safe.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.