Global Parallelism Caps vs Provider Retry Tuning for Terraform Connection Exhaustion

Question

Terraform core drives applies with a default concurrency of 10, adjustable only through the global -parallelism flag. Each provider process owns its HTTP connection pool, and Terraform exposes no settings for that pool's sizing. In large multi-resource configurations, worker concurrency can outpace what a provider's HTTP client handles cleanly, producing intermittent connection failures that often succeed on a later retry.

That leaves two documented levers, and they work at different layers:

Lowering -parallelism (for example to 3–5) caps concurrency globally, but slows every provider equally, including lightweight ones such as null or local that sit far from their own limits.
Raising provider retry settings such as the AWS provider's max_retries and retry_mode absorbs transient failures after they occur, but does nothing to prevent the initial connection spike that causes them, and can mask contention in pipelines that pass on re-run.

For a configuration dominated by one cloud provider, which lever should anchor the stability strategy: a low global parallelism cap, or default parallelism with aggressive provider retries? Does any current Terraform release offer per-provider concurrency control? And how should the choice account for Terraform Cloud or Enterprise agents that enforce their own parallelism defaults?

Zoha · Answer

AnswerFor a configuration dominated by a single cloud provider, anchor the stability strategy on the global -parallelism flag: tune it so the number of concurrent provider processes stays within the provider's HTTP connection-pool capacity. Keep provider-specific retry settings at their defaults or only modestly increased to absorb occasional transient errors. High retries alone do not prevent the initial connection spike that creates the failures.Confirmed factsTerraform core concurrency is controlled exclusively by the global -parallelism flag (or the TF_PARALLELISM environment variable). No stable release up to Terraform 1.5.x offers a per-provider concurrency setting.Provider retry arguments (e.g., the AWS provider's max_retries and retry_mode) affect only how a plugin process recovers from transient HTTP errors; they do not reduce the number of simultaneous connections opened.In Terraform Cloud, the workspace Parallelism setting overrides the CLI flag for remote runs. Terraform Enterprise agents can enforce their own parallelism cap, which likewise supersedes the flag when the agent executes the run.Likely explanationWhen many resources are managed by the same provider, each concurrent Terraform worker drives a provider plugin process with its own HTTP connection pool. If global parallelism exceeds the provider's pool capacity, excess workers contend for sockets, producing intermittent connection-reset or timeout errors that often succeed on retry. Lowering -parallelism directly reduces simultaneous pools and prevents the contention. Raising retries only masks the symptom; the spike remains and can hide capacity problems that would surface under sustained load.Steps for this caseDetermine the provider's observed connection-pool capacity (for AWS, a metric such as HTTPClient.ConnectionCount, or the provider's equivalent). Without that metric, run a short test at the default parallelism of 10 and note the error rate.Set a moderate global parallelism below the observed limit, for example -parallelism=4 or -parallelism=6, via the flag or TF_PARALLELISM.Leave provider retry settings at defaults (AWS provider: max_retries = 3, retry_mode = "standard") or raise them only slightly (e.g., to 5) if you see occasional transient spikes unrelated to parallelism.Verify the change: run TF_LOG=TRACE terraform apply -parallelism=4 and count simultaneous plugin invocations in the log, or watch the provider's connection metric during the apply.If errors persist, lower parallelism further; if errors vanish but runtime grows excessively, consider whether lightweight providers (null, local) are being throttled unnecessarily.Accounting for Terraform Cloud / EnterpriseIn Terraform Cloud, edit the workspace under Settings → General → Parallelism to match the value chosen locally; this overrides any CLI flag on remote runs.In Terraform Enterprise, check the agent configuration file (agent.conf) for the parallelism setting. The agent's value supersedes both the workspace setting and the CLI flag when the agent runs the job.Align local testing parallelism with the value enforced by the Cloud workspace or Enterprise agent so local results reproduce remotely.Missing diagnostic detailIf you cannot observe the provider's connection-pool metric, provide the approximate error rate at the default parallelism of 10 (e.g., what percentage of resources fail with a connection timeout on the first attempt). That determines whether a lower cap is needed or the default is already safe.

Global Parallelism Caps vs Provider Retry Tuning for Terraform Connection Exhaustion

1 answer

Answer

Confirmed facts

Likely explanation

Steps for this case

Accounting for Terraform Cloud / Enterprise

Missing diagnostic detail

Your perspective belongs here.

0 question comments