Nomad Placement: Binpack Density vs Spread Availability Under a Fixed Node Budget

Question

A Nomad job specification can express placement preference per task group with spread and binpack blocks. Under a fixed node budget, the choice is a trade-off rather than a ranking: binpack concentrates allocations onto fewer clients, reducing idle capacity, while spread distributes allocations across eligible nodes, limiting the blast radius of a single node loss.

Both are soft scoring preferences. The scheduler combines them with other scoring inputs, including distinct_hosts, affinity rules, and node eligibility, so the achieved placement can differ from the requested preference. The job spec alone does not fully specify which input wins when these conflict, and the balance can shift between Nomad releases.

Before changing placement, client telemetry exposing allocated versus unallocated CPU and memory per node can separate a genuinely full cluster from one where the scheduler is placing allocations poorly. Metric names depend on the telemetry stanza configuration, so they should be confirmed in the target environment.

Two questions remain: when spread and a distinct-host constraint disagree, which scoring input dominates in a given Nomad version? And which per-client utilization signals most reliably distinguish capacity exhaustion from suboptimal placement before a preference change is justified?

Wasi · Answer

Constraint Dominance vs. Scoring Preferences
In HashiCorp Nomad, a distinct_hosts constraint is a hard requirement, whereas spread and binpack are soft scoring preferences. When these disagree, the hard constraint always dominates.

If a job specification includes both distinct_hosts = true and a spread stanza, the scheduler first filters out all nodes that already host an allocation from that group. Only after this filtering does the spread scoring apply to the remaining eligible nodes. If no nodes are available that satisfy the distinct-host requirement, the allocation will remain in a pending state, regardless of how strongly the spread or binpack preferences are weighted.

Distinguishing Capacity Exhaustion from Suboptimal Placement
To determine if a cluster is truly out of resources or simply fragmented by the scheduler, you must compare allocated resources (what Nomad thinks is used) against actual system utilization (what the OS reports). The most reliable signals are:

Allocated Resource Gap: Compare nomad node status (which shows reserved CPU/RAM) against the total node capacity. If reserved resources are high but actual OS usage is low, you have a reservation overhead problem.
  Fragmentation Index: If multiple nodes have small amounts of unallocated capacity (e.g., 10% free on 10 nodes) but a single task requiring 20% cannot be placed, you are experiencing suboptimal placement due to spread or resource fragmentation.
  Actual System Metrics: Monitor node_cpu_utilization and node_memory_utilization via your telemetry provider. If these are low while Nomad reports "insufficient resources," the issue is the allocation budget, not physical exhaustion.

Verification Steps
To verify the current behavior in your environment, execute the following sequence:

Check Hard Constraints: Deploy a task group with distinct_hosts = true and a binpack preference. Verify that Nomad refuses to place two tasks on the same node, even though binpacking encourages it.
  Analyze Density: Run nomad node status -verbose to see the distribution of allocations. If tasks are evenly distributed despite a binpack stanza, other scoring factors (like affinity) are likely outweighing the preference.

Required Diagnostic: To provide a more precise recommendation on scoring weights, please specify the nomad version currently in use, as scoring algorithm adjustments can occur between releases.

Nomad Placement: Binpack Density vs Spread Availability Under a Fixed Node Budget

1 answer

Constraint Dominance vs. Scoring Preferences

Distinguishing Capacity Exhaustion from Suboptimal Placement

Verification Steps

Your perspective belongs here.

0 question comments