Reducing Terraform Pipeline Latency with Terramate Stack Orchestration

The problem: Sequential bottlenecks in large Terraform estates

As infrastructure grows, the common pattern is to split a monolithic state file into multiple smaller stacks to reduce blast radius. However, this introduces a new problem: execution latency. When you have dozens of stacks, running terraform apply sequentially across every directory in a CI pipeline creates a massive bottleneck. Even if the 'database' stack and the 'compute' stack are entirely independent, a standard shell script or basic CI loop will process them one after another, wasting available runner resources and increasing the time to deploy.

Thesis: Parallel execution via dependency graphing

Terramate solves this by treating your directory structure as a Directed Acyclic Graph (DAG). Instead of a linear list of folders, the terramate run command analyzes dependencies declared in terramate.config.hcl files. It identifies which stacks can be executed simultaneously and which must wait for a prerequisite, allowing you to maximize your CI runner's CPU and network throughput without risking race conditions in your infrastructure.

Defining Stack Relationships

In Terramate, a stack is defined by a directory containing Terraform configurations and a terramate.config.hcl file. To manage the order of operations, you use the depends_on attribute. This tells the orchestrator that the current stack requires the outputs or the existence of another stack before it can safely begin its own execution.

For example, a compute stack cannot be provisioned until the network stack has created the VPC and subnets. By declaring this relationship, Terramate ensures the network stack finishes successfully before the compute stack is even initialized.

Worked Example: Orchestrating a Three-Tier App

Consider a project with three stacks: network, compute, and db. Both compute and db require the network to be present, but they do not depend on each other.

Configuration:

# infra/network/terramate.config.hcl
# No depends_on: this is a root stack

# infra/compute/terramate.config.hcl
depends_on = ["../network"]

# infra/db/terramate.config.hcl
depends_on = ["../network"]

Execution: Run the following command from the root of your project (requires Terramate CLI installed and configured):

terramate run terraform apply --parallel=4

Expected Behavior:

1. Terramate analyzes the graph and identifies that network has no dependencies. It executes terraform apply in the infra/network/ directory first.
2. Once the network apply completes, both compute and db become eligible.
3. Because --parallel=4 is set, Terramate launches the apply commands for both compute and db concurrently.

In a real-world scenario, if the network takes 3 minutes and the other two take 4 minutes each, a sequential run would take 11 minutes. With terramate run, the total time drops to approximately 7 minutes.

Trade-offs and Constraints

While parallelization speeds up deployments, it introduces specific engineering risks:

• API Rate Limiting: Launching too many concurrent Terraform processes can trigger rate limits from your cloud provider (e.g., AWS or Azure API throttling). You must tune the --parallel flag based on your provider's limits and the size of your estate.
• Backend Contention: If multiple stacks attempt to access the same remote state backend or lock file simultaneously, you may encounter locking errors. To avoid this, ensure each stack has a unique state key, which can be managed via Terramate's generate feature.
• Circular Dependencies: If Stack A depends on B, and B depends on A, Terramate will detect the cycle and fail immediately with an error before any infrastructure is touched, requiring a manual refactor of the architecture.

Verification and Results

To verify the orchestration logic without actually modifying infrastructure, run a plan across the estate:

terramate run terraform plan

You can inspect the computed execution order by checking the .terramate/orchestration.json file. This file contains the resolved DAG and confirms which stacks are grouped together for parallel execution. If a stack is running earlier than expected, review its terramate.config.hcl to ensure the depends_on path is correct.