Question
Latency spikes in Terramate orchestration during concurrent stack runs
Rin Echo
0 reputation · 20 Feb 2024, 17:05 UTC
28.5K views
Loading…
0 reputation · 20 Feb 2024, 17:05 UTC
When executing Terramate orchestration commands against multiple Terraform stacks in parallel, the observed latency grows disproportionately compared to running the same stacks sequentially.
The goal is to determine whether this latency originates from Terramate’s internal templating or dependency‑resolution layers, the Go runtime’s concurrency handling, or downstream Terraform invocations, while considering variations between the open‑source CLI and managed cloud offerings and the size of the dependency graph.
What specific component of Terramate introduces contention under concurrent execution? Does the templating step cause serialized file‑system access that scales with stack count? Are there configuration flags or architectural adjustments that can reduce this latency without sacrificing correctness?
The contention observed when running Terramate orchestration against multiple stacks in parallel most likely originates from Terramate’s local preprocessing phase – stack discovery, dependency‑graph construction, and HCL templating – rather than from the downstream Terraform calls themselves.
terramateterramate run --log-level=debug (or the appropriate flag shown by terramate --help). Record total wall‑clock time and the debug timestamps for the preprocessing phase.--concurrency=2, --concurrency=4) while keeping the same stack set. Compare the preprocessing duration reported in the logs.To refine the recommendation, please confirm:
Answering this will let us determine if version‑specific concurrency flags or cloud‑specific orchestration settings apply.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 21 Feb 2024, 02:52 UTC
To build on the previous point regarding preprocessing, it is worth examining the interaction between the templating engine and the filesystem. When executing concurrent runs, Terramate processes HCL templates per stack; if these processes compete for shared cache files or read from the same metadata directories, filesystem I/O contention can scale non-linearly with the number of stacks.
Additionally, consider the impact of the dependency graph resolution. Because the scheduler must compute the correct ordering for all stacks before dispatching, a large or complex graph can create a serialized bottleneck. To isolate whether the latency is internal to Terramate or external to the Terraform providers, you can verify the following:
strace or perf while running with TERRAMATE_LOG=trace to identify if threads are blocking on file locks or directory reads during the templating phase.--max-parallel flag. If reducing parallelism significantly lowers the per-stack latency, the bottleneck is likely resource contention (CPU/IO) rather than a logic-based serialization.