Using Nomad Templates to Inject Dynamic Configs from Consul and Vault

Why Use Nomad Templates?

When a service needs runtime configuration that can change without rebuilding the image, Nomad’s template stanza is the most efficient solution. It renders files at allocation start (or on change) by pulling data from Consul KV, Vault secrets, or environment variables, and can trigger a restart or custom action when the rendered content changes.

How the Template Stanza Works

The template block lives inside a task definition. At allocation time:

• Nomad evaluates the Go template expression.
• It fetches any referenced data from Consul or Vault.
• The resulting text is written atomically to the destination file.
• If change_mode is set, the task is notified when the rendered content changes.

Key functions:

• key("path") – reads a Consul KV key.
• secret("path") – reads a Vault secret.
• env("VAR") – reads an environment variable.
• file("/path/to/file") – reads a file on the client host.

Example: Injecting a Database URL from Consul KV

Below is a minimal job that pulls the database URL from Consul and restarts the service whenever it changes.

job "db-app" {
  datacenters = ["dc1"]
  type = "service"

  group "app" {
    task "web" {
      driver = "docker"
      config {
        image = "myorg/webapp:latest"
        port_map {
          http = 8080
        }
      }

      template {
        data = "{{ key \"config/db/url\" }}"
        destination = "local/config/db.conf"
        change_mode = "restart"
      }

      resources {
        cpu    = 500
        memory = 256
        network {
          port "http" {}
        }
      }
    }
  }
}

What happens when the Consul key config/db/url changes?

• Nomad re‑renders db.conf.
• The file is updated atomically.
• Because change_mode = "restart", the task is stopped and started again, ensuring the new URL is used.

Advanced Change Modes

Restart is common, but sometimes a full restart is too disruptive. Nomad offers three alternatives:

• signal – sends a configurable signal (default SIGHUP) to the task’s main process.
• noop – does nothing; the task must handle hot‑reload on its own.
• script – runs a custom script defined in change_script.

Example using signal:

template {
  data = "{{ key \"config/app/feature_flag\" }}"
  destination = "local/feature.flag"
  change_mode = "signal"
  change_signal = "SIGUSR2"
}

Or a custom script:

template {
  data = "{{ key \"config/app/feature_flag\" }}"
  destination = "local/feature.flag"
  change_mode = "script"
  change_script = "{{ env \"HOME\" }}/reload.sh"
}

Common Pitfalls and How to Avoid Them

1. Unreachable Consul/Vault at Start – If the key or secret cannot be fetched, the job fails to start. Use change_mode = "noop" for non‑critical files or add conditional logic in the template.
2. Frequent Restarts – A template that changes often and uses restart will churn the allocation. Prefer signal or script for hot‑reloadable services.
3. Permission Errors – The destination path defaults to the task’s local directory. Ensure the Nomad client user has write permission or use an absolute path with appropriate ACLs.
4. Missing Keys or Secrets – The key or secret function will block until the value exists. Validate with nomad job plan and test in a staging environment.
5. Atomicity Misunderstanding – Nomad writes the file atomically, but the application must read the file after the template is rendered. Verify by checking the file contents in the allocation’s local directory.

Verification Checklist

• Run nomad job plan <job.hcl> to see the template diff before deployment.
• After nomad job run, use nomad alloc fs <alloc-id> local/config/db.conf to confirm the file content.
• Change the Consul key: consul kv put config/db/url "postgres://user:pass@db:5432/app" and watch nomad alloc status show a restart.
• Check Nomad client logs (nomad monitor) for rendering errors or permission issues.

When to Use Templates vs. Environment Variables

Templates are ideal for:

• Large configuration files that are easier to maintain outside the image.
• Injecting secrets that need to be stored in a file rather than an env var.
• Providing dynamic values that may change over time (e.g., feature flags).

Environment variables are simpler for one‑off values that never change, or when the application expects them directly.