Nomad Spread Constraint for High Availability Task Distribution

Distribute tasks across node attributes with Nomad's spread constraint to achieve fault tolerance and automatic load balancing.

Desired Outcome

Distribute a set of tasks across multiple node attributes (e.g., region or availability zone) so that a failure in one attribute does not affect the whole service, while also balancing load automatically.

Prerequisites

Nomad cluster running version 0.10 or later.
Nodes must have the spread attribute correctly labeled (e.g., `region=us-east-1` or `az=us-west-2`).
Operator permission to register the job and modify node attributes.
Network connectivity between client and server for job submission.

Step‑by‑Step Procedure

Define the node attributes on each server. Example using the Nomad client UI or `nomad server` config:
```
# /etc/nomad/attributes.hcl
node {
  attributes = {
    "region" = "us-east-1"
    "az"     = "us-east-1a"
  }
}
```
Ensure the attribute values match the names you will reference in the spread block.

Create a job specification that uses a spread constraint. Place the block under a group or the top‑level job definition:

job "my‑app" {
  datacenters = ["dc1"]
  group "frontend" {
    count = 6
    task "web" {
      driver = "docker"
      config {
        image = "myorg/web:latest"
      }
      resources {
        cpu    = 500
        memory = 256
      }
    }
    spread {
      attribute = "region"
      group_size = 2
      conservative = false
    }
  }
}

Here `group_size = 2` creates two tasks per distinct region value, spreading the six tasks across at least three regions.

Submit the job to the cluster:
```
nomad job run /path/to/job.hcl
```
Note: the command must be run on a client with operator rights.
Monitor allocation progress:
```
nomad status 
```
The output lists each task allocation. Verify that tasks appear on nodes with differing `region` values.
If needed, inspect detailed allocation status:
```
nomad alloc-status 
```
Check that the `attributes` field includes the varied region identifiers.

Expected Verification Checks

Run `nomad status ` and confirm that the `allocations` section shows tasks distributed across at least two distinct attribute values.
Execute `nomad alloc-status `; the `node` field should list nodes whose attributes match the spread criteria (e.g., different `region`).
Scale the job (e.g., `nomad scaling up 3`) and re‑run the status command. The new allocations must continue to respect the spread pattern, not collapse onto a single region.

Recovery and Rollback

If a misconfiguration causes tasks to land on nodes lacking sufficient resources, the allocation will stay pending. To recover:

Update the job definition to correct the spread attribute or `group_size` and re‑apply:
```
nomad job modify  /path/to/updated.job.hcl
```
Alternatively, delete the offending allocation and let the scheduler re‑evaluate:
```
nomad alloc kill 
```
Rollback to a previous job version is possible by re‑submitting the earlier `.hcl` file; this changes the cluster state, so treat it as a state‑changing operation rather than a simple undo.

Limitations and Practical Checks

Incorrect attribute names or misspelled values will cause the spread constraint to be ignored, resulting in all tasks landing on a single node type.
Very small `group_size` (e.g., 1) can fragment the allocation, reducing scheduling efficiency and increasing latency; test with a moderate value such as 2‑4.
When nodes are re‑labeled (e.g., after a region migration), existing allocations may become invalid; re‑run `nomad status` and, if needed, update the job to reflect new attribute values.

Summary

Using Nomad’s spread constraint provides a declarative way to achieve high‑availability task distribution without manual placement logic. By correctly labeling nodes, defining a `spread` block with an appropriate `attribute` and `group_size`, and verifying allocations through `nomad status` and `nomad alloc-status`, operators can ensure fault‑tolerant deployments that automatically balance across diverse infrastructure.

Using Nomad's Spread Constraint to Distribute Tasks Across Regions for High Availability