Using Nomad's Spread Stanza to Balance Allocations Across Racks

The problem: uneven allocation creates hot spots

When you run a Nomad job that scales to dozens of allocations, the scheduler may place many of them on the same rack or host if those nodes happen to have the best scores for CPU, memory, or affinity. This concentrates failure risk and can cause uneven load on network switches or power circuits.

Thesis: the spread stanza lets you influence placement without overriding hard constraints

Nomad’s spread stanza adds a scoring factor that favors under‑represented values of node attributes (e.g., node.rack, node.datacenter). It does not replace CPU/memory constraints or hard affinity rules; it merely nudges the scheduler toward a more even distribution.

How spread works

When a job includes a spread stanza, Nomad evaluates each candidate node by looking at how many allocations already use each attribute value. The fewer allocations that share a value, the higher the score for that node. The final placement score is a weighted combination of the spread score and other factors such as resource availability.

You can specify multiple attributes; Nomad will try to balance each independently. The feature is available in Nomad 0.9+ and works alongside affinity, constraint, and preempt stanzas.

Worked example: spreading a web service across racks

Assume a cluster where each node has a node‑level metadata label rack (e.g., rack=a, rack=b, rack=c). We want a three‑replica web service to be placed on different racks whenever possible.

job "web" {
  datacenters = ["dc1"]
  type = "service"

  group "api" {
    count = 3

    network {
      port "http" {}
    }

    service {
      name = "web"
      port = "http"
    }

    spread {
      attribute = "${node.rack}"
    }

    task "server" {
      driver = "docker"
      config {
        image = "nginx:latest"
        ports = ["http"]
      }
      resources {
        cpu    = 200
        memory = 64
      }
    }
  }
}

Save the above as web.hcl and run:

# Run from a workstation with Nomad CLI access and a token that has job submission rights
nomad job run web.hcl

After the job placements settle, inspect the allocations:

nomad alloc status -job <jobid>

In the output, note the Node column and compare it with node metadata:

nomad node status -self -json | jq '.[] | {ID, Name, Attributes: .Attributes}'

You should see each allocation on a node whose node.rack attribute differs from the others, assuming at least three distinct rack values exist and sufficient resources.

Trade‑off and limitation

Spread only influences the scoring phase; it cannot override hard constraints. If no nodes carry a particular attribute value (for example, a mislabeled rack), the scheduler may still place allocations elsewhere, but the spread score will be zero for that attribute, potentially leading to longer evaluation times as the scheduler weighs many spread factors.

Using many attributes in a single spread stanza increases the complexity of the scoring function. In large clusters with hundreds of nodes, this can add noticeable latency to the scheduling loop. A practical approach is to start with a single, high‑impact attribute (such as rack or availability zone) and add more only if you observe persistent imbalance.

Actionable closing

To verify that spread is having the desired effect, compare allocation distribution before and after adding the stanza:

1. Run the job without the spread block and record the rack values of each allocation (nomad alloc status -job <jobid> -json).
2. Add the spread stanza, rerun the job, and repeat the inspection.
3. Look for a more even spread of rack values across the allocations.

If the distribution remains uneven, check that the attribute is correctly set on all nodes (nomad node status -json) and that the job’s resource demands are not so high that only a subset of nodes can satisfy them.

By using the spread stanza deliberately, you can reduce the chance of correlated failures and improve load distribution without sacrificing the guarantees provided by Nomad’s constraint and affinity system.