Choosing Nomad Job Types: Service vs Batch for Predictable Scheduling and Resilience

Problem

Teams deploying workloads on HashiCorp Nomad often pick a job type by habit rather than by the scheduling guarantees they need. A service job that restarts on every node failure can waste capacity for a short‑lived batch task, while a batch job without resilient: true will disappear silently when a host is drained. The result is either unnecessary churn or lost work.

Takeaway: Match the type stanza to the workload’s lifecycle—service for long‑running, health‑checked processes that must survive node loss; batch for finite tasks that should run once unless you explicitly opt‑in to rescheduling.

Requirements Driving the Decision

• Availability: Does the workload need to stay reachable after a host failure?
• Idempotency: Can the task be safely re‑executed from the start?
• Resource retention: Should allocations hold their reservations during a drain?
• Health visibility: Do you need built‑in HTTP/TCP/script checks that drive restarts?

Smallest Suitable Design

Nomad’s type field is a single‑word switch that determines the default scheduler behavior. No extra stanzas are required for the baseline semantics.

Service Jobs

job "api" {
  datacenters = ["dc1"]
  type        = "service"
  group "web" {
    count = 3
    task "server" {
      driver = "docker"
      config {
        image = "myorg/api:v1.2"
        port_map {
          http = 8080
        }
      }
      service {
        name = "api"
        port = "http"
        check {
          type     = "http"
          path     = "/health"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

Key defaults for type = "service":

• Allocations are resilient by default – they are rescheduled on node failure.
• Health checks defined in the service stanza drive automatic restarts.
• Leader election (via leader stanza) is available for singleton workloads.
• During nomad node drain the allocation is migrated, preserving its reservation.

Batch Jobs

job "render" {
  datacenters = ["dc1"]
  type        = "batch"
  group "worker" {
    count = 1
    task "renderer" {
      driver = "docker"
      config {
        image = "myorg/renderer:latest"
      }
      # No service stanza – no health checks, no leader election
    }
  }
}

Defaults for type = "batch":

• Allocations run to completion and then transition to dead.
• No automatic restart on node loss unless resilient = true is set on the group or task.
• Drain behavior: the allocation is stopped and not rescheduled (unless resilient).
• Suitable for CI pipelines, report generation, or any idempotent one‑shot work.

Trust and Data Boundaries

The scheduler treats the type as a contract between the operator and the cluster. Changing it after deployment requires a job update, which Nomad treats as a new version—allocations are replaced according to the new type’s rules. Because the contract is explicit, you can audit a job’s intent by reading the type field alone.

Interaction with resume, affinity, and resilient

Feature	Service (default)	Batch (default)	Override
resume (client restart)	Allocations resume on client restart	Allocations do not resume	Set resume = true on batch group to keep allocation
affinity (placement)	Works identically	Works identically	N/A
resilient (node failure)	Implicit true	Implicit false	Explicit resilient = true on batch group enables rescheduling

Example: a batch job that must survive a host loss but should not restart on client reboot:

group "worker" {
  resilient = true
  resume    = false
  task "renderer" { … }
}

Operational Checks

1. Verify the effective type – run on any machine with the Nomad CLI and a valid ACL token:

   nomad job status -json  | jq '.Job.Type'
   

   Expected output: "service" or "batch". If the value diverges from the spec, a stale plan may be applied.
2. Dry‑run a new job to see scheduler decisions without creating allocations:

   nomad job run -dry-run -output=json api.nomad | jq '.EvalCreateIndex, .FailedTGAllocs'
   

   Check that Type in the evaluation matches the stanza and that no unexpected resilient flags appear.
3. Observe health‑check driven restarts for service jobs:

   nomad alloc status -verbose 
   

   Look for Restart Count increments and Health Check results.
4. Simulate node drain (requires node:write ACL):

   nomad node drain -enable -self
   # watch allocations move
   nomad node status -verbose
   

   Service allocations should migrate; batch allocations without resilient should terminate.

Failure Modes

• Mis‑typed service as batch – long‑running API disappears on host loss, causing downtime.
• Batch job with resilient: true but non‑idempotent work – duplicate execution leads to data corruption.
• Health check interval too aggressive (default 10 s) – flapping restarts under transient network blips. Adjust interval and timeout in the service check.
• Version drift – Nomad 1.5+ changed the default check_interval for script checks from 30 s to 10 s. Verify the version (nomad version) and test after upgrades.

When to Re‑evaluate the Design

• Workload transitions from periodic to continuous (e.g., a nightly report becomes a streaming API). Switch to service and add health checks.
• Cluster adopts autopilot or scheduler plugins that alter placement; re‑run dry‑run to confirm behavior.
• New compliance requirement mandates that all allocations survive node drain – set resilient = true on batch groups or migrate to service type.

Limitations & Verification

Health‑check defaults (interval, timeout, grace period) are version‑dependent. The article assumes Nomad ≥ 0.9 where the type semantics are stable, but exact defaults should be confirmed on the target cluster (nomad version). The dry‑run and status commands above are the practical way to verify that the scheduler honors the intended type before promoting a job to production.

Diagram Labels

• Job Type Decision Flow
• Service Job Lifecycle
• Batch Job Lifecycle
• Resilience Interaction Matrix