Choosing Nomad’s Native Docker/OCI Driver vs Alternative Execution Methods for Container Workloads

Decision and Constraints

When running containerized workloads on HashiCorp Nomad, you must decide how Nomad will launch and manage those containers. The primary constraint is to use Nomad’s built‑in scheduling and resource isolation while avoiding unnecessary complexity or compatibility gaps with the host’s container runtime. The decision hinges on three factors:

1. Desired level of isolation and device‑plugin support (e.g., GPU, FPGA).
2. Compatibility with the host’s container runtime version (containerd, Docker Engine).
3. Operational overhead for image pulling, logging, and debugging.

Comparison Table

h>Device plugin support

Option	How Nomad launches containers	Resource isolation	Image handling	Operational notes
Native Docker/OCI driver	Nomad talks directly to the host’s containerd (or Docker Engine) via the CRI‑compatible interface.	Full Linux namespaces and cgroups as provided by containerd.	Yes – Nomad can attach GPUs, FPGAs, or other devices through the device plugin system.	Nomad pulls the image using the configured registry auth; layers are cached by containerd.	Minimal extra processes; relies on host runtime version.
Raw exec driver with containerd commands	Nomad runs a raw_exec task that invokes ctr or docker CLI to start a container.	Depends on the correctness of the invoked CLI; nominally same as native driver.	Limited – device plugins must be passed manually via CLI flags; no automatic Nomad‑device integration.	Image pull performed by the CLI; caching depends on CLI behavior.	More moving parts; requires careful quoting and error handling in the task script.
Exec driver running docker run	Nomad executes a simple exec task that runs docker run ….	Isolation provided by Docker Engine; Nomad does not enforce its own cgroup limits.	Possible via Docker’s --gpus flag, but Nomad cannot track device allocation.	Docker pulls the image; Nomad has no visibility into pull failures beyond the task exit code.	Simple to write, but bypasses Nomad’s resource enforcement and device‑plugin tracking.

Trade‑offs

The native Docker/OCI driver gives Nomad full visibility into container lifecycle, allowing it to enforce CPU/memory quotas, report accurate usage, and automatically attach Nomad‑managed devices. It also benefits from containerd’s efficient image layer caching and integrates with Nomad’s device‑plugin framework for GPUs or FPGAs.

The raw exec approach provides flexibility if you need to invoke a custom container runtime binary or pass exotic flags that the Docker driver does not expose. However, you lose Nomad’s automatic resource enforcement and must manually handle device allocation, increasing the chance of mis‑configuration.

Using the exec driver with docker run is the quickest to prototype but should be avoided for production workloads because Nomad cannot guarantee that the requested CPU/memory limits are honored, and device‑plugin accounting is absent.

Implementation Example: Native Docker/OCI Driver with GPU

The following Nomad job runs a TensorFlow container, requests one GPU, and enforces 2 CPU cores and 4 GiB of memory.

job "tensorflow-gpu" {
  datacenters = ["dc1"]

  group "tf" {
    count = 1

    task "train" {
      driver = "docker"

      config {
        image = "tensorflow/tensorflow:latest-gpu"
        # Optional: pull policy if you want to always fetch latest
        pull_policy = "always"
      }

      resources {
        cpu    = 2000   # 2 CPU cores
        memory = 4096   # 4 MiB
        # Nomad‑managed GPU device
        device {
          name = "gpu"
          # The device plugin must expose a device labeled "gpu"
          # on the Nomad client (e.g., NVIDIA GPU device plugin)
        }
      }

      service {
        name = "tf-train"
        port = "http"
        checks {
          type = "http"
          path = "/health"
          interval = "10s"
          timeout = "2s"
        }
      }
    }
  }
}

Place this file as tensorflow-gpu.nomad and run:

nomad job run tensorflow-gpu.nomad

Validation Steps

1. Verify that the job placed the GPU device as expected:

nomad alloc status <allocation-id>
nomad inspect task <allocation-id>.train

In the output, look for a Resources section that includes Device with the label gpu and confirm that CPU and Memory match the values defined in the job.

1. Check that the container is running and the image was pulled:

nomad job status tensorflow-gpu
# Look for "running" status and "latest" image digest

You can also enter the container to inspect logs:

nomad alloc exec <allocation-id> train ps aux
# or, if the container runtime is Docker:
docker exec -it <container-id> cat /var/log/tensorflow.log

1. Confirm resource enforcement from the host:

# On the Nomad client node, find the container’s cgroup path
cat /sys/fs/cgroup/cpu//cpu.max
cat /sys/fs/cgroup/memory//memory.max

The values should reflect the 2 CPU core quota (expressed as 200000 microseconds per 100 ms period) and the 4 MiB memory limit.

Limitations and Practical Checks

• The native driver requires a recent containerd (≥ 1.3) or Docker Engine that supports the CRI‑compatible image store. Older versions may lack certain OCI image features (e.g., manifest lists) or device‑plugin integration.
• Device plugin support depends on the Nomad client having a compatible device plugin installed (e.g., NVIDIA GPU device plugin). Without it, the device block will cause the placement to fail.
• If you need to run privileged containers or custom seccomp profiles, you must specify them in the config block; the driver does not automatically inherit host defaults.
• To check that the driver is being used, examine the allocation’s Driver field via nomad alloc status -json; it should show "docker".

By following the example and validation steps above, you can confidently decide that Nomad’s native Docker/OCI driver is the appropriate choice for workloads that require strong resource isolation, device‑plugin integration, and observable container lifecycle management.