Zero‑Downtime Rolling Updates for Nomad Jobs: A Step‑by‑Step Guide

Desired Outcome

Deploy a new version of a service without any downtime by using Nomad’s built‑in rolling update mechanism. Healthy allocations stay running while new ones are started, and old allocations are only drained after the replacements pass their health checks.

Prerequisites

• A Nomad cluster running v1.0 or later (earlier versions lack canary and advanced update stanzas).
• Nomad CLI installed and authenticated (ACL token with job:write and node:read permissions).
• A job specification that includes an update stanza defining max_parallel, health_check, and min_healthy_time.
• Enough free resources on the cluster to run both the old and new allocations simultaneously (at least max_parallel extra task slots).
• Optional: Consul integration for service‑discovery health checks.

Job Configuration Example

The following snippet shows a minimal Docker job with a rolling‑update stanza. Adjust the image tag, resource limits, and health‑check endpoint to match your service.

job "api" {
  datacenters = ["dc1"]
  type        = "service"

  update {
    max_parallel     = 2          # number of new allocations started at once
    health_check     = "checks"   # wait for Consul/Nomad checks to pass
    min_healthy_time = "30s"      # allocation must stay healthy this long
    healthy_deadline = "5m"       # give up after this if health never passes
    progress_deadline = "10m"    # overall rollout timeout
    canary           = 1          # start a single canary before full rollout
  }

  group "api" {
    count = 6

    task "api" {
      driver = "docker"
      config {
        image = "myorg/api:v2.3.1"   # <-- change this tag for a new release
        port_map {
          http = 8080
        }
      }
      resources {
        cpu    = 500
        memory = 256
      }
      service {
        name = "api"
        port = "http"
        check {
          type     = "http"
          path     = "/health"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

Procedure

1. Edit the job file – change the Docker image tag (or binary version) to the new release.
2. Preview the changes – run the plan command from a workstation that has access to the Nomad API:

   nomad job plan api.nomad.hcl

   Verify that the plan shows the expected number of allocations to be created and destroyed, and that max_parallel does not exceed available capacity.
3. Apply the update – use the -check-index flag to avoid race conditions if the job was modified concurrently:

   nomad job run -check-index $(nomad job inspect -json api | jq -r '.JobModifyIndex') api.nomad.hcl

   Required permission: job:write on the target namespace.
4. Monitor the rollout – watch the deployment status:

   nomad job status api

   For a specific allocation, use:

   nomad alloc status -short <alloc-id>

   The UI (typically http://:4646) also shows a live deployment graph.
5. Adjust canary if needed – if the canary allocation fails health checks, Nomad pauses the deployment. Inspect logs with nomad alloc logs -stderr <alloc-id>, fix the issue, then resume:

   nomad job resume api

Validation Checks

• All new allocations running and healthy – nomad job status api should list Desired Status = run and the Deployed version matching the new image tag.
• Old allocations drained only after new ones are healthy – verify that the allocation count never drops below the min_healthy_time threshold.
• Service discovery reflects the new version – if using Consul:

  consul catalog service -service=api

  The output should show only instances registered with the new image tag (or the updated meta.version if you set it).
• Application endpoints respond correctly – send a request to the service endpoint (e.g., curl -s http://api.service.consul/health) and confirm the expected JSON payload or HTTP 200.
• No failed allocations in UI/logs – the Nomad UI deployment page should show a green “Successful” status; allocation logs should contain no error traces.

Rollback & Recovery

If the new version fails health checks, Nomad automatically pauses the deployment. You have two practical recovery paths:

• Fix and resume – after correcting the image, configuration, or health‑check definition, run nomad job resume api to continue the rollout.
• Revert to the previous version – Nomad keeps a history of job versions. Revert with:

  nomad job revert api

  This command restores the last successfully deployed version and starts a new rolling update back to the known‑good state. You can also specify a specific version index: nomad job revert -job-version=3 api.

Risk note: Reverting also follows the update stanza constraints, so capacity must still allow the old allocations to be placed.

Limitations & Practical Tips

• max_parallel must be ≤ the number of nodes with spare capacity; otherwise the deployment stalls with “failed to place allocation” errors.
• Overly aggressive health checks (short intervals, low timeouts) can mark healthy tasks as unhealthy, causing unnecessary rollbacks. Tune interval and timeout to realistic service start‑up times.
• Nomad versions before 0.10 lack the canary and progress_deadline fields; verify cluster version with nomad version before relying on them.
• When using Consul, ensure the service registration TTL/health check matches Nomad’s health_check = "checks" setting; mismatches can cause Nomad to consider an allocation healthy while Consul still marks it critical.

Quick Verification Checklist

Check	Command	Expected Result
Job deployment status	nomad job status api	Desired Status = run, Deployed version = new tag
Allocation health	nomad alloc status -short <alloc-id>	State = running, Checks = healthy
Consul service catalog	consul catalog service -service=api	Only new version entries present
Endpoint response	curl -s http://api.service.consul/health	HTTP 200 with expected payload

Follow this guide each time you ship a new container image or binary. The rolling‑update stanza gives you deterministic, zero‑downtime deployments as long as capacity and health‑check settings are sized for your workload.