Guide: Zero‑Downtime Rolling Deployments in HashiCorp Nomad Using the update Stanza

Desired outcome

Publish a new version of a Nomad job without downtime and with automatic rollback if the new version fails health checks. The rolling behavior is controlled by the update stanza inside a job group, which manages canary allocations, health evaluation, promotion, and reversion.

Prerequisites

• A running Nomad cluster (≥1 server, ≥1 client).
• The Nomad CLI installed and configured to talk to the cluster.
• A job file written in HCL that includes a service block (so Consul health checks can be used).
• A Consul cluster reachable from Nomad clients (default health_check = checks mode).
• If ACLs are enabled, set NOMAD_TOKEN to a token with submit-job, read-job, and deployment capabilities.
• Nomad version ≥1.0 (field names and defaults are stable in the 1.x series; verify against your exact release).

Procedure

1. Add an update stanza to the job group. Example snippet (place inside the group block):

   group "example" {
     update {
       max_parallel        = 1          # how many allocations can be updated simultaneously
       canary              = 1          # run one canary before expanding
       auto_promote        = true       # promote canary automatically when healthy
       auto_revert         = true       # revert to last stable version on failure
       min_healthy_time    = "30s"      # time an allocation must stay healthy before promotion
       healthy_deadline    = "5m"       # max time to wait for health before marking failed
       progress_deadline   = "10m"      # overall time limit for the whole rollout
       health_check        = "checks"   # read Consul health checks (alternatives: task_states, manual)
     }
     # ... task and service definitions ...
   }
   

2. Preview the planned changes. Run:

   nomad job plan -var-file=vars.hcl example.nomad

   Verify that the output shows a rolling update with the specified canary and parallelism values.
3. Submit or update the job. Use:

   nomad job run example.nomad

   If the job already exists, this command creates a new deployment; otherwise it creates the job and starts the first deployment.
4. Monitor the rollout. After submission, track progress with:

   nomad deployment status <deployment-id>
   nomad job status example
   nomad alloc status <allocation-id>

   Look for fields such as Canary, Healthy, Progress Deadline, and Auto Revert in the deployment status.
5. Optional manual intervention.
   • To force promotion of a healthy canary: nomad deployment promote <deployment-id>
   • To mark a canary as unhealthy and trigger rollback: nomad deployment fail <deployment-id>
   • To abort a stuck rollout: same deployment fail command.

Expected checks during rollout

• nomad deployment status shows counts for Canary, Healthy, Unhealthy, and Progress Deadline remaining.
• nomad job status lists allocations grouped by version (e.g., v1 stable, v2 canary).
• Individual allocation health can be inspected with nomad alloc status <id>; watch for Restarts or Failed task states.
• When using health_check = checks, Consul should display the service with a canary tag for canary allocations and passing checks for promoted instances.

Recovery options

If health fails within healthy_deadline or the overall progress_deadline expires, Nomad marks the deployment as failed. With auto_revert = true, Nomad automatically places the last stable version back onto the nodes.

Manual recovery paths:

• Roll back to a specific prior version: nomad job revert example <version>
• Force a failed deployment to stop: nomad deployment fail <deployment-id>
• Remove the job entirely (if needed): nomad job stop example

Limitations and practical verification

• The default values for update fields have changed across Nomad releases; always consult the update‑stanza reference matching your nomad version output.
• health_check = checks requires Consul to be reachable; if Consul is unavailable, health will never be satisfied and the rollout will stall until deadlines expire. In Consul‑less environments, use health_check = task_states.
• auto_revert only rolls back to a previously stable version. On the very first deployment (no prior stable version), a failed rollout leaves allocations stopped rather than reverting to a non‑existent version.
• Nomad does not shift traffic percentages; the canary tag is set on Consul services so that an external router or proxy (e.g., Envoy, Kong) can decide to send traffic only to non‑canary instances.
• Nomad’s license changed to Business Source for newer releases; verify that your use case complies with the license of the exact binary you run.

To verify the setup in a safe environment:

1. Start a dev agent: nomad agent -dev
2. Submit the job with the update stanza above.
3. Watch nomad deployment status as the canary runs, becomes healthy, and is promoted.
4. Break health intentionally (e.g., stop the task or deregister the Consul service) and confirm that auto_revert