Safe Nomad Rollouts: Update Blocks, Health Checks, and Auto-Revert

The problem: a bad release can replace healthy instances before you notice

You submit a new version of a Nomad job. Nomad starts placing replacement allocations, and if the new code crashes on startup, the old allocations may already be drained. Without update pacing and health checks, you can end up with zero healthy instances and no clear signal about which allocation failed.

The useful takeaway is to define an update stanza, attach a task health check, and set auto_revert = true. Nomad then waits for new allocations to prove themselves before removing old ones, and rolls the job back automatically if checks fail or progress stalls.

How the update stanza controls rollout speed

Nomad uses an allocation as the unit of work for a task group. A deployment is Nomad's tracked effort to move from one job version to the next. The update stanza controls that movement:

• max_parallel: how many allocations may be updated at once.
• min_healthy_time: how long an allocation must stay healthy before it counts.
• healthy_threshold: how many healthy allocations are required before continuing.
• progress_threshold: the percentage of allocations that must be healthy before Nomad considers the deployment stuck.

A conservative starting point for a three-instance service is updating one allocation at a time and requiring it to stay healthy for ten seconds.

Health checks decide whether an allocation is ready

Nomad supports HTTP, TCP, script, and shell checks. An HTTP check is common for web services: Nomad hits a path on the allocation's network port and treats a successful response as healthy. If the check fails, the allocation is marked unhealthy and the deployment can stall.

Place the check inside the task's service block so Nomad associates it with the allocation's address and port.

Auto-revert rolls the job spec back, not the application bug

When auto_revert is true, Nomad reverts the job to the previous version if the deployment fails health checks or stops making progress. This restores the last known-good task configuration. It does not undo database migrations, fix application logic, or recover data written by the bad version.

Worked example: a three-instance API rollout

Create a job file, for example api.nomad, on a machine with the Nomad CLI installed and authenticated to your cluster:

job "api" {
  datacenters = ["dc1"]
  type        = "service"

  group "api" {
    count = 3

    update {
      max_parallel             = 1
      min_healthy_time         = "10s"
      healthy_threshold        = 1
      progress_threshold       = 1
      auto_revert              = true
      auto_revert_healthy_threshold = 1
    }

    task "server" {
      driver = "docker"

      config {
        image = "myapp:1.2.0"
      }

      service {
        name = "api"
        port = "http"

        check {
          name     = "http-alive"
          type     = "http"
          path     = "/health"
          interval = "10s"
          timeout  = "2s"
        }
      }

      resources {
        network {
          port "http" {}
        }
      }
    }
  }
}

Run the job from your workstation or CI runner with permission to submit jobs:

nomad job run api.nomad

This changes cluster state by creating or updating the api job.

Watch the deployment with the deployment ID returned by the previous command, or look it up with nomad job status api:

nomad deployment status <deployment-id>

Look for Status: running while allocations are replaced, then Status: complete if all new allocations pass their checks. If a check fails, the deployment stays Status: running until it stalls, then Nomad triggers auto-revert and returns the job to the previous spec.

To verify the safety net without waiting for a real failure, temporarily break the /health endpoint in the new image, start the update, and confirm that nomad deployment status shows stalled progress followed by a revert. Do this first in a local dev agent:

nomad agent -dev

Then run nomad job run api.nomad against the local agent before touching production.

Verify behavior against your installed release with nomad --version and the HashiCorp Nomad update and health check reference before changing production jobs.

Trade-offs and limitations

• Tight values for min_healthy_time or flaky health checks can slow rollouts and cause repeated deploy-revert cycles.
• Client-to-server network partitions can delay health check reports and stall deployments until checks time out.
• Auto-revert restores the previous job spec but does not roll back data changes made by the new version.

Check the result before you declare success

After the deployment finishes, run:

nomad job status api
nomad deployment status <deployment-id>

Confirm that the desired count matches the running allocation count and that the latest deployment shows Status: complete. If you see repeated Status: failed deployments, inspect the failing allocation with nomad alloc status <allocation-id> and review its task events and check output.

Start with one allocation at a time, generous health timeouts, and auto-revert enabled. Tighten the timing only after you have observed stable rollouts in a non-production environment.