Cutting Terraform Monorepo CI Time with Terramate Change Detection

The Problem: Every Plan Runs Everywhere

Terraform monorepos promise shared modules and consistent environments, but they punish you at CI time. A one-line change in a networking module triggers terraform plan across fifty root modules. Most of those plans return "no changes" — yet each consumes minutes of runner time, API quota, and review attention. The blast radius is equally painful: a broken plan in an unrelated stack blocks the entire pipeline.

Terramate solves this by treating each directory with a terramate.stack.hcl as an independent stack, building a directed acyclic graph from filesystem nesting and explicit depends_on declarations, then running commands only on stacks that actually changed and their downstream dependents.

How Change Detection Works

Terramate uses git diff (or filesystem timestamps as a fallback) to mark stacks as changed. When you run terramate run terraform plan, it computes the affected subgraph: the changed stacks plus any stacks that depend on them, transitively. Stacks outside that subgraph are skipped entirely.

This relies on full git history. Shallow clones (--depth=1) or CI systems that fetch only the latest commit break diff logic, causing false positives where every stack appears changed. The fix is simple: ensure git fetch --unshallow runs before Terramate, or rely on the filesystem timestamp fallback by setting TM_CHANGE_DETECTION=fs.

DRY Configuration Without Copy-Paste Drift

The generate block renders shared files — backend configuration, provider blocks, common variables — into every stack from a single source. Each stack remains a valid standalone Terraform root module, so you can still cd stacks/prod/network && terraform plan locally without Terramate.

Globals and data blocks provide typed, validated input that flows into generate templates. Instead of stringly HCL interpolation, you define globals { environment = "prod" } once and reference global.environment in templates, with validation catching typos early.

Worked Example: Two-Stack Monorepo

Create a temporary repository to see the behavior:

mkdir -p stacks/{network,compute}
cat > terramate.hcl <<'EOF'
generate "backend" {
  path      = "backend.tf"
  content   = <<-EOT
    terraform {
      backend "s3" {
        bucket         = "tf-state-${global.environment}"
        key            = "${terramate.stack.path}/terraform.tfstate"
        region         = "us-east-1"
        dynamodb_table = "tf-locks"
      }
    }
  EOT
}
globals {
  environment = "dev"
}
EOF
cat > stacks/network/terramate.stack.hcl <<'EOF'
stack {
  name        = "network"
  description = "VPC and subnets"
}
EOF
cat > stacks/compute/terramate.stack.hcl <<'EOF'
stack {
  name        = "compute"
  description = "EC2 instances"
  depends_on  = ["network"]
}
EOF
cat > stacks/network/main.tf <<'EOF'
resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
}
EOF
cat > stacks/compute/main.tf <<'EOF'
resource "aws_instance" "app" {
  ami           = "ami-0abcdef1234567890"
  instance_type = "t3.micro"
}
EOF

Run generation and verify change detection:

terramate generate
# Edit stacks/network/main.tf (add a tag)
terramate list --changed
# Output: stacks/network
terramate run terraform plan
# Only network stack plans; compute is skipped because it depends on network but network changed

After terramate generate, run terraform fmt -check -diff in each stack directory. Generated files must be formatted before commit; forgetting this step leaves unformatted backend.tf files that break downstream tooling.

Parallel Execution and Race Conditions

terramate run --parallel=4 --continue-on-error terraform apply executes independent stacks concurrently. The concurrency limit prevents API throttling. However, parallel order is non-deterministic for independent stacks. If two stacks mutate the same remote resource (e.g., both write to the same S3 bucket), races occur. Model this as an explicit depends_on in the affected stack's terramate.stack.hcl to serialize them.

Drift Detection Aggregates Exit Codes

terramate drift runs terraform plan -detailed-exitcode per stack and aggregates results. Exit code 0 = no drift, 2 = drift detected, other = error. Only stacks with real drift surface; clean stacks stay silent. This requires valid credentials for every stack's backend and providers. A single misconfigured stack stalls the entire run unless you add --continue-on-error.

Limitations to Watch

• Globals evaluate before generate: Circular references between globals and generate templates are not caught early; they surface as confusing template errors.
• Shallow clones break diff: Ensure full history in CI or use TM_CHANGE_DETECTION=fs.
• Generated files need formatting: Add terraform fmt -check as a post-generate gate in CI.
• Parallel races on shared resources: Declare explicit dependencies when stacks touch the same remote object.

Verify in Your Pipeline

1. Add git fetch --unshallow (or fetch-depth: 0 in GitHub Actions) before Terramate steps.
2. Run terramate generate && terraform fmt -check -diff -recursive as a lint job.
3. Replace your full-repo terraform plan with terramate run terraform plan --parallel=4 --continue-on-error.
4. Schedule nightly terramate drift --continue-on-error to catch manual changes.

Start with a two-stack subset of your monorepo. Measure CI minutes before and after. The reduction is usually immediate and measurable.