Affinity-aware allocation replacement in Nomad rolling updates

Answer

Nomad evaluates affinity and placement constraints when it creates a replacement allocation during a rolling update. This happens before the new allocation is marked healthy; health checks are only run after the allocation is placed and starts. The old allocation is terminated only after the replacement passes its health check.

Confirmed facts

• Hard affinity (affinity { rule = "hard" ... }) must be satisfied for the replacement to be placed; if no node matches, the update stalls until a suitable node appears.
• Soft affinity influences scoring but does not guarantee placement on the same logical node set.
• The placement stanza can combine hard/soft affinity with attribute expressions; guarantees of landing on the exact same node set depend on the node attributes remaining unchanged and the affinity being hard.
• For a three‑node job with max_unscheduled = 1, setting min_healthy_percentage = 66 (or 67) ensures at least two allocations stay healthy during the update, allowing one node to be taken out for replacement while preserving service availability.

Steps for this case

1. Define the job with hard affinity to the desired node class or label, e.g.:

   job "example" {
     type = "service"
     dispatch = { rolling = {} }
     update {
       max_unscheduled = 1
       min_healthy_percentage = 66
     }
     group "api" {
       count = 3
       affinity {
         rule = "hard"
         attribute = "${attr.unique.hostname}"
         value   = "node-group‑alpha"
       }
       network { port "http" {}}
       service {
         name = "api"
         port = "http"
         check { type = "http", path = "/health", interval = "10s" }
       }
       task "server" {
         driver = "docker"
         config { image = "myapp:latest" }
       }
     }
   }

2. Run nomad job run example.nomad to place the initial three allocations.
3. Trigger an update (e.g., change a meta field) with nomad job update -meta version=2 example.
4. Monitor with nomad status -verbose example and nomad alloc status <alloc-id> to verify that a new allocation appears on a node satisfying the hard affinity before the oldest allocation is marked unhealthy.

Missing diagnostic detail

If you cannot guarantee that node attributes (such as labels or custom attributes) remain static during the update, please confirm whether any node‑level metadata changes are expected; this determines whether the hard affinity guarantee holds.