Terramate Stack-Based Change Detection: Architecture Note

Requirements

To use Terramate’s selective execution feature you need:

• A Git repository that contains the Terramate project.
• Each infrastructure component represented as a stack with its own terramate.hcl file.
• Terramate version 0.10.0 or newer (the changed filter was introduced in this release).
• Access to the local file system and Git metadata; no network calls are made unless you enable remote caching or drift checks explicitly.

If any of these conditions are not met, Terramate falls back to a full‑run execution or may miss changes.

Smallest Suitable Design

The minimal design that enables change detection consists of three parts:

1. Stack definition – each stack declares its Terraform files and optionally a depends_on block to express ordering.
2. Change detection layer – Terramate scans the Git history between the current HEAD and the last successful run (stored in a hidden .terramate directory) to produce a list of stacks whose terramate.hcl or .tf files have changed.
3. Selective executor – the terramate run command receives the filtered list and invokes Terraform only for those stacks, respecting depends_on for ordering while parallelizing independent stacks.

This design keeps the blast radius limited to the stacks that actually changed and reduces runtime proportionally to the number of affected stacks.

Trust/Data Boundaries

Terramate operates entirely on the local workstation:

• File system – reads terramate.hcl, Terraform files, and any local variable files.
• Git metadata – reads commit objects and tree objects to compute diffs; it does not push or pull from remotes unless you configure a remote backend for Terraform state.
• External services – no code is sent to Terramate’s servers or third‑party APIs. If you enable the optional remote cache (terramate cloud) or drift checks, then hashed file contents are transmitted, but the original HCL remains local.

Thus the trust boundary is the developer’s machine; any compromise of the local Git repository could affect detection, but Terramate itself does not introduce additional network exposure.

Operational Checks

Terramate provides several built‑in checks that run before or during execution:

• Formatting validation – terramate fmt --check verifies that all HCL files are correctly formatted.
• Linting – integrates with tflint and custom HCL linters to catch syntax issues early.
• Drift detection – when enabled, Terramate runs terraform plan in a read‑only mode and compares the output against the stored state to flag manual changes.
• Dependency validation – the depends_on block is verified for circular dependencies before the run starts.

These checks can be invoked manually or configured to run automatically in CI pipelines.

Failure Modes

Because execution is scoped to changed stacks, failures are isolated:

• If a stack’s terraform apply returns a non‑zero exit code, Terramate halts the run, marks the stack as failed, and skips any downstream stacks that declared a depends_on relationship.
• The error message includes the stack name and the underlying Terraform output, making troubleshooting straightforward.
• Upstream stacks (those that do not depend on the failed stack) remain unaffected and are not rolled back automatically; you must decide whether to re‑run them after fixing the failure.

Terramate does not attempt to roll back Terraform state; state rollback must be handled by your Terraform workflow or external tooling.

Conditions That Would Change the Design

Certain circumstances would require revisiting the minimal design:

1. Non‑Git workflows – if stacks are managed outside Git (e.g., tarballs or external APIs), the change detection layer would need a different mechanism, such as file‑system watchers or explicit manifest inputs.
2. Uncommitted changes – Terramate only considers committed history; local uncommitted edits are ignored unless you stage them. To detect uncommitted work, you would need to add a pre‑run git diff --quiet check or commit a temporary snapshot.
3. Large monorepos with frequent unrelated changes – if the Git history scan becomes a performance bottleneck, you might introduce a caching layer that records the last‑seen commit per stack, trading a bit of freshness for speed.
4. Regulatory constraints on metadata export – if transmitting hashed file contents to a remote cache is prohibited, you would disable the remote cache feature and rely solely on local execution.

In each case, the core concept of scoping Terraform runs to affected stacks remains valid, but the implementation of the change detection layer would need adjustment.

Practical Verification

To confirm that selective execution is working as expected:

1. Clone a Terramate‑enabled repository:
2. git clone https://example.com/org/infra.git
3. cd infra
4. Make a change to a single stack, e.g., edit stacks/network/terramate.hcl.
5. Run the detection command:
6. terramate list --changed
7. You should see only the network stack listed.
8. Execute a plan for the changed stacks:
9. terramate run terraform plan
10. Observe that Terraform runs only for the network stack; other stacks are skipped and reported as “unchanged”.

If you instead edit a file in an unrelated stack, the list will reflect that stack, and the plan will be limited to it. This behavior validates the change detection layer and the selective executor.

Example Configuration with Dependencies

The following snippet shows two stacks where web depends on network:

# stacks/network/terramate.hcl
terramate {
  stack {
    name = "network"
  }
}

# stacks/web/terramate.hcl
terramate {
  stack {
    name = "web"
  }
  depends_on = ["network"]
}

When you run terramate run terraform apply after modifying only the network stack, Terramate will:

1. Detect the network stack as changed.
2. Run terraform apply for network.
3. Wait for network to finish.
4. Then run terraform apply for web, because the dependency is satisfied.
5. If network fails, web is skipped and the error includes the stack name.

This demonstrates how Terramate respects ordering while still parallelizing independent stacks (e.g., a third monitoring stack with no dependencies would run concurrently with network).

Limitations

• Relies on a clean Git history; rebasing or force‑pushing after a Terramate run can cause the internal .terramate checkpoint to become stale, leading to false‑positive or false‑negative change detection.
• Only files tracked by Git are considered; ignored files (per .gitignore) will not trigger a run even if they affect Terraform behavior.
• The selective execution feature is unavailable in versions prior to 0.10.0; older releases always execute all stacks.

To mitigate the first limitation, you can reset the checkpoint after a rebasing operation:

terramate purge-cache   # removes .terramate directory, forcing a full detection on next run

Running this command is safe because it only deletes Terramate’s internal metadata; it does not affect your Terraform state or source code.

Summary

Terramate’s stack‑based change detection provides a lightweight, Git‑driven mechanism to limit Terraform execution to the stacks that actually changed. The smallest workable design requires a Git‑backed repository, per‑stack terramate.hcl files, and version 0.10.0 or newer. Trust boundaries stay local, operational checks catch formatting, linting, and drift issues, and failures are confined to the affected stack with clear error messages. Understanding the conditions that would alter the design—such as non‑Git workflows or uncommitted changes—helps you decide when the built‑in feature is sufficient and when you need to augment it with custom checks or external tooling.