Nomad Preemption with Priority Classes to Share Production and Batch on One Fleet

Problem

Running production services and opportunistic batch on the same Nomad fleet without starving services or paying for idle capacity. The useful takeaway is to use cluster-wide priority classes with resource reservations and explicit preemption, not just node pools.

Requirements

Run critical services and best-effort batch on a shared Nomad fleet. Guarantee CPU and memory for services during load spikes. Allow batch to consume spare capacity and be evicted when needed. Keep operational overhead low and avoid separate clusters for cost reasons.

Design constraints: scheduling must be deterministic, preemption must be observable, and isolation must be logical with clear trust boundaries.

Smallest suitable design

Control plane: 3 Nomad servers for quorum, ACL enabled, namespaces per team. Clients run on worker nodes grouped into node pools by instance type.

Priority bands. Service jobs use priority 80-100 with resource reservations. Batch jobs use priority 10-40 with preempt=true.

Service job pattern. Priority is high and resources are reserved so the scheduler will not place lower priority work into the reserved slice.

job "payments-api" {
  datacenters = ["dc1"]
  namespace   = "prod"
  type        = "service"
  priority    = 90

  group "api" {
    count = 3
    ephemeral_disk { size = 500 }
    resources {
      cpu    = 1000
      memory = 2048
      reserved {
        cpu    = 1000
        memory = 2048
      }
    }
    network { port "http" {} }
    task "api" { driver = "docker" ... }
  }
}

Batch job pattern. Low priority and opt-in to preemption. The preempt block defines which priority classes it can evict.

job "etl-nightly" {
  datacenters = ["dc1"]
  namespace   = "batch"
  type        = "batch"
  priority    = 20
  preempt {
    system = false
  }
  group "etl" {
    count = 10
    restart { attempts = 0 }
    resources { cpu = 500 memory = 1024 }
    task "etl" { driver = "docker" ... }
  }
}

Node pools separate instance types and allow constraints to keep noisy workloads off latency-sensitive nodes. Placement constraints reference node_pool meta.

Validate placement before submit from a workstation with an ACL token that has job:write in the target namespace.

nomad job plan -namespace=prod job.hcl

Expected check: plan shows eligible nodes and no placement conflicts with reservations. Risk: a misconfigured high priority batch job can preempt production services if reservations are missing.

Trust and data boundaries

Nomad servers are trusted control plane holding Raft state and ACLs. Clients execute untrusted workloads inside cgroup and network namespaces. Isolation is OS-level, not hardware.

Identity and traffic: Consul Connect provides mTLS between services. Secrets are delivered via Vault and should not be written to persistent volumes. Namespace and ACL isolation is logical; co-tenant workloads share kernel and network stack on the same client.

Do not rely on namespaces for strong security boundaries. For hard compliance isolation use separate clusters.

Operational checks

Monitor scheduler evaluations per second, allocation pending time, preemption count, node drain status, and server leader changes.

Check server metrics for evaluations and preemptions, and review client system logs for cgroup limits and OOM events. On a client node with sudo access:

journalctl -u nomad --since "1 hour ago" | grep -i oom

Simulate load by scaling a critical service and observe low priority batch allocations being preempted and rescheduled via:

nomad job status etl-nightly

Expected behavior: allocations move to pending and are rescheduled when capacity returns. Validate ACL enforcement by attempting job submit with a restricted token and confirming namespace access denial in audit logs.

Failure modes

Server network partition stalls scheduling because the scheduler requires a leader. Client crash or OOM kills allocations without restart guarantee for batch jobs with attempts = 0.

Aggressive preemption causes thrashing if batch jobs restart quickly and reclaim resources repeatedly. Missing constraints leads to noisy neighbor on shared nodes.

Preemption reclaims resources but does not guarantee rapid restart. Batch jobs must be idempotent and checkpointed.

When the design changes

Multi-region requires Nomad federation and Consul WAN gossip. GPU or high I/O workloads need device plugins, topology constraints, and dedicated node pools.

Hard compliance isolation or untrusted multi-tenant workloads require separate clusters rather than namespaces.

Limitations: priority is cluster-wide. Reservations are required to protect services. Logical isolation is insufficient for strong security boundaries.