Allocating GPUs in HashiCorp Nomad with the NVIDIA Device Plugin

Problem: You need to run a GPU‑dependent workload but Nomad shows no GPU resources

Your team wants to train a model or run a CUDA‑based tool inside Nomad, but when you submit a job it stays pending because the client reports zero GPUs. The missing piece is usually the Nomad device plugin that exposes NVIDIA GPUs as consumable resources.

Thesis: Enabling the Nomad device plugin and using the device stanza lets you request, isolate, and monitor GPUs just like CPU or memory

Once the plugin runs on each client, Nomad treats GPUs as a finite resource set. You can request a specific count or list of UUIDs, combine the request with constraints, and rely on Nomad’s scheduler to place the allocation only when the GPU is free.

Section 1: Install and enable the NVIDIA device plugin

On every Nomad client that should offer GPUs:

1. Install the NVIDIA driver (≥384) and verify with nvidia-smi.
2. Download the Nomad device plugin binary that matches your Nomad version (e.g., nomad-device-plugin-v0.10.0).
3. Place the binary in the plugin directory, typically /opt/nomad/plugins/device.
4. Enable the plugin in the client config:

plugin "device" {
  config {}
}

Restart the Nomad agent. After restart, run nomad node status -self -verbose and look for a devices section listing GPU UUIDs and a GPU attribute under Available Resources.

Section 2: Define a job that requests a GPU

Use the device stanza inside a task. The simplest form requests one GPU without caring about its UUID:

job "gpu-demo" {
  datacenters = ["dc1"]
  type = "service"

  group "gpu" {
    count = 1

    task "compute" {
      driver = "docker"

      config {
        image = "nvidia/cuda:11.8-base"
        command = "nvidia-smi"
      }

      resources {
        cpu    = 500
        memory = 256
      }

      device {
        gpu {
          # Request exactly one GPU; you could also list specific UUIDs
          count = 1
        }
      }
    }
  }
}

Submit the job with nomad job run gpu-demo.nomad. Nomad will schedule the allocation only when a client reports a free GPU.

Section 3: Verify the allocation and observe metrics

After submission, check the allocation logs:

nomad alloc logs 

You should see the output of nvidia-smi, confirming the container could access the GPU.

Verify resource consumption on the node:

nomad node status -self -verbose

The Available Resources GPU count should have decreased by one while the allocation is running.

If you have telemetry enabled, you can scrape GPU metrics such as nomad.client.devices.gpu.utilization from the Nomad metrics endpoint and visualize them in Prometheus/Grafana.

Trade‑off and limitation

• No oversubscription: Nomad treats GPUs as a hard‑capped resource. If a job asks for more GPUs than are physically present, it stays pending until enough devices are free.
• Preemption: Graceful preemption of GPU allocations is only available in Nomad 1.2+ via the preempt stanza. Older versions will kill the task when preempted.
• Driver and runtime compatibility: GPU isolation depends on the NVIDIA driver version and the container runtime’s support for device isolation. Using drivers older than 384 or a misconfigured runtime can break isolation.

Actionable closing

To start using GPUs in Nomad today:

1. Confirm the NVIDIA driver works (nvidia-smi).
2. Install the matching Nomad device plugin and enable it in the client config.
3. Restart the Nomad agent and verify GPUs appear in nomad node status.
4. Write a job with a device { gpu { count = 1 } } stanza (adjust count or UUID list as needed).
5. Run the job, inspect allocation logs, and check that the node’s available GPU count drops.
6. Optionally enable telemetry to monitor GPU utilization over time.

Once these steps are verified, you can treat GPUs like any other Nomad resource, combining them with constraints, affinities, and (on Nomad 1.2+) preemption policies to fit your workload’s scheduling needs.