Architecting with Terramate: Stack Orchestration, Code Generation, and Operational Safeguards

Problem

Teams managing large Terraform codebases often struggle with duplicated modules, scattered variable definitions, and brittle pipelines that re‑run the entire infrastructure on every change. Terramate introduces a layer that organizes Terraform into logical stacks, generates code from reusable templates, and orchestrates Terraform execution based on dependency graphs.

Requirements

• Git‑based repository containing Terraform modules and Terramate configuration.
• Terramate CLI (v0.1+), Terraform CLI, and a CI system that can run shell commands.
• All stack directories must contain a terraform.hcl (optional) and a main.tf.tmpl (template).
• Local cache directory .terramate/cache should be writable; it is used for change detection.

Smallest Suitable Design

At its core, a Terramate repository needs:

# Directory layout
├── .terramate
│   └── cache
├── stacks
│   ├── infra
│   │   ├── terraform.hcl
│   │   └── main.tf.tmpl
│   └── services
│       ├── api
│       │   ├── terraform.hcl
│       │   └── main.tf.tmpl
│       └── worker
│           ├── terraform.hcl
│           └── main.tf.tmpl
└── terramate.hcl

The terramate.hcl file defines global variables and default settings. Each stack can override or extend these variables. The template file uses Terramate’s templating syntax (similar to Go templates) to reference shared variables, ensuring a single source of truth.

Trust & Data Boundaries

• Source of Truth: All Terraform code that should persist is kept in templates (.tmpl) and stack configuration files (.hcl). Generated .tf files are considered transient and should never be edited manually.
• Cache Isolation: The .terramate/cache directory holds metadata about stack dependencies and file timestamps. It must not be committed to source control; it is regenerated on every run.
• Variable Visibility: Variables defined in terramate.hcl are scoped to the stack hierarchy. Child stacks inherit parent variables unless overridden.
• Dependency Graph: Terramate automatically builds a directed acyclic graph (DAG) of stack dependencies based on depends_on directives. This graph is used for incremental runs.

Operational Checks

1. Generate Verification

   After any change to a template or terraform.hcl, run:

   terramate generate
   

   Check that the expected .tf files appear under each stack directory. Use git diff to confirm only the generated files changed.

2. Change Detection

   To see which stacks Terramate thinks are affected by a file change:

   terramate changes --json
   

   Review the JSON output; it lists stack paths and the reason for inclusion (e.g., "template updated"). This is useful for CI to decide which stacks to run.

3. Incremental Run

   Execute Terraform only for affected stacks:

   terramate run --parallel 4
   

   Terramate will invoke terraform init, plan, and apply in dependency order. Verify that the output shows only the targeted stacks.

4. Cache Health Check

   If Terramate re‑runs all stacks unexpectedly, delete .terramate/cache and re‑run terramate generate to rebuild the cache. A corrupted cache is a common cause of full rebuilds.

Failure Modes & Mitigation

• Overwritten Manual Edits – Any manual changes to generated .tf files will be lost on the next generate. Mitigation: keep all custom logic in templates or in separate modules referenced from templates.
• Cache Corruption – Leads to unnecessary full runs and longer pipeline times. Mitigation: treat .terramate/cache as a transient artifact; exclude it from version control and let CI rebuild it.
• Dependency Graph Cycles – Introducing a circular dependency in depends_on will cause Terramate to error out. Mitigation: run terramate validate locally before committing changes.
• Insufficient Permissions – Terramate requires write access to stack directories and the cache. In CI, ensure the runner has the correct filesystem permissions.

When to Change the Design

Consider revisiting the Terramate architecture if:

• Stacks grow beyond a few dozen and the cache size becomes a bottleneck.
• Teams need to share templates across organizations; you may need to externalize the template repository and use Terramate’s include feature.
• Your CI system cannot handle the parallelism Terramate offers, requiring a custom wrapper.
• You introduce a new Terraform provider that requires a different backend configuration for certain stacks; this might necessitate a more granular terraform.hcl hierarchy.

In such cases, evaluate whether adding a separate common stack for shared modules or migrating to a monorepo with a dedicated Terramate workspace is more efficient.