Protecting Production in Nomad with Priority, Preemption, and Node Pools

Production services and best‑effort batch work end up on the same Nomad cluster. When capacity tightens, a low‑priority batch job can delay a high‑priority service, or a burst of batch work can push a service off a node. The useful takeaway is that Nomad priority controls scheduling order, preemption can evict lower‑priority allocations when configured, and node pools provide a hard boundary. Used together they let critical services be protected while batch backfills idle capacity.

Priority sets order, not reservation

Every Nomad job has a priority field from 1 to 100. Higher priority jobs are considered first by the scheduler when resources are constrained. Priority does not reserve capacity. A priority 90 job can still wait if no node satisfies its constraints, and without isolation it can be placed on a node shared with noisy neighbors.

Priority is useful for ordering, but it is not a guarantee. Placement constraints, affinity, and resource requests still dominate whether a job lands at all.

Preemption as a controlled eviction

Preemption is a scheduler behavior that can evict lower‑priority allocations to make room for a higher‑priority job. It is configured via the preempt stanza in the job specification. Preemption is opt‑in per job and is evaluated during plan and apply.

Preemption can cause allocation churn and restarts for evicted jobs. It should be used with careful resource requests and restart policies. Preemption is not a substitute for hard resource quotas or capacity planning and does not guarantee zero evictions under sustained overload.

Node pools give hard boundaries

Node pools and node classes provide a practical engineering decision point for physical or logical isolation. Critical services can be pinned to a dedicated pool, while best‑effort work uses a shared pool. This prevents batch jobs from ever competing for the same machines as production, even if priorities overlap.

Node pool membership is set on clients and visible via system APIs. Combining pool constraints with priority and preempt settings creates a common pattern: protect critical services and allow batch work to backfill idle capacity without starving production.

Worked example

The following illustrates a pattern for mixed‑criticality workloads. It is an example configuration, not a tested deployment.

job \"api-service\" {\n  type = \"service\"\n  priority = 90\n  datacenters = [\"dc1\"]\n  node_pool = \"critical\"\n\n  group \"api\" {\n    count = 3\n    preempt {\n      enabled = false\n    }\n    constraint {\n      attribute = \"${node.class}\"\n      value = \"critical\"\n    }\n    task \"api\" {\n      resources {\n        cpu = 500\n        memory = 512\n      }\n    }\n  }\n}\n\njob \"nightly-batch\" {\n  type = \"batch\"\n  priority = 30\n  datacenters = [\"dc1\"]\n  node_pool = \"besteffort\"\n\n  group \"batch\" {\n    count = 10\n    preempt {\n      enabled = true\n    }\n    constraint {\n      attribute = \"${node.class}\"\n      value = \"besteffort\"\n    }\n    task \"batch\" {\n      resources {\n        cpu = 200\n        memory = 256\n      }\n    }\n  }\n}\n

The service runs at priority 90 on the critical node pool with preemption disabled, so it cannot be evicted. The batch job runs at priority 30 on the best‑effort pool with preemption enabled, so it can be evicted if a higher‑priority job needs its resources within the same pool.

Trade‑offs and limits

Preemption introduces churn. Evicted allocations restart according to their restart policy, which can increase load during recovery. Behavior of preemption and node pool enforcement is version sensitive and requires compatible Nomad server and client versions.

Priority alone does not reserve capacity. Without node pools, a high‑priority job can still be delayed by placement constraints. Node pools add operational overhead for labeling nodes and maintaining pools.

Check it works

Verify configuration from a workstation with the Nomad CLI and read access to jobs and nodes.

Inspect a job to confirm priority and preempt fields are present:

nomad job inspect <job-name>\n

Run a plan for a higher priority job against a constrained cluster to see scheduler preemption messages in the plan output:

nomad job plan <job-name>\n

List nodes and pools to verify membership and class metadata:

nomad node status\nnomad system nodes\n

Check allocation status to observe evictions and restarts after a preemption event:

nomad alloc status <alloc-id>\n

Use these checks after any change to priority, preempt, or node pool constraints. Changes to node pools and preempt settings affect scheduling and can cause restarts. Plan changes in a non‑production environment first.