Nomad Dispatch: Architecture Note for One‑Off Jobs

Requirements

Nomad’s dispatch feature lets a client trigger a one‑off execution of a parameterized job. To use it you need:

• Nomad server version ≥ 0.9 (the examples assume 1.2.x).
• A job defined with a dispatch stanza that declares the parameters it accepts.
• An ACL token that includes the job:dispatch capability for the target job’s namespace.
• Network access from the caller to the Nomad HTTP API (default port 4646).

Smallest Suitable Design

The minimal viable topology consists of:

• A three‑node Nomad server cluster for HA (or a single‑node dev server).
• A pool of Nomad client nodes that can run the job’s driver (e.g., exec or docker).
• A job specification that contains only:
  • The dispatch stanza with one or more parameter definitions.
  • A single task that runs a simple command (e.g., /bin/echo).
  • No affinity, spread, or template stanzas.

This design keeps the control plane isolated from the workload and avoids extra operational overhead.

Trust and Data Boundaries

The dispatch API is part of Nomad’s HTTP interface. Calls must be authenticated with an ACL token; the token’s policies govern whether the request is allowed. When Nomad accepts a dispatch request:

• The server creates an allocation and hands it off to a client node.
• The task runs under the Nomad client’s OS user (usually nomad) unless the driver overrides it.
• No direct access to the Nomad server’s data stores is granted; isolation relies on the client’s OS sandbox (namespaces, cgroups).
• If the job mounts host volumes or uses a CSI plugin, data leaves the client’s sandbox and introduces an additional trust boundary that must be secured separately.

Operational Checks

To verify that dispatch is working as expected:

1. ACL validation – Run nomad token capabilities <token> and confirm job:dispatch appears for the target job.
2. Request acceptance – After issuing nomad job dispatch <job-id> -<param>=<value>, check the server log for a line like [INFO] dispatcher: job dispatch received.
3. Allocation creation – Run nomad alloc status <alloc-id> (the ID is shown in the dispatch output) and ensure the allocation is in running or complete state.
4. Resource usage – On the client node, inspect nomad node alloc <node-id> or use nomad alloc status -json to verify CPU/memory stay within the job’s limits.
5. Alerting – Set up a metric alert on nomad.job.dispatch.failed (if using Prometheus integration) or on allocation failed events.

Failure Modes and Design Triggers

Certain conditions require moving beyond the minimal design:

• Privileged access needed – If the dispatched task requires capabilities like CAP_SYS_ADMIN or access to host devices, switch to a driver that supports privileged mode (e.g., Docker with privileged = true) and adjust the host’s security profile (SELinux/AppArmor, seccomp). This widens the trust boundary because the task gains more host privileges.
• Persistent storage required – Add a host volume or CSI volume binding in the task stanza. The volume introduces a data trust boundary; ensure the volume’s access policies align with the job’s sensitivity.
• High‑frequency dispatch traffic – Repeated API calls can saturate the Nomad HTTP endpoint. Introduce a rate‑limiting proxy (e.g., Envoy) in front of the Nomad servers or batch multiple dispatches into a single parameterized job that internally loops.
• Multi‑region or federation needs – If the job must run in a specific region, add a dispatch stanza with region constraint or use Nomad Federation to forward the request to the appropriate cluster.

Example Configuration

The following job file defines a simple echo task that accepts a message parameter via dispatch.

job "echo-dispatch" {
  datacenters = ["dc1"]

  dispatch {
    payload = {
      message = { description = "Message to print" type = "string" default = "hello" }
    }
  }

  group "example" {
    task "printer" {
      driver = "exec"
      config {
        command = "/bin/echo"
        args    = ["${dispatch.payload.message}"]
      }
      resources {
        cpu    = 100
        memory = 64
      }
    }
  }
}

To dispatch the job with a custom message:

# Run on a machine with the Nomad CLI installed
# Ensure you have an ACL token with job:dispatch capability exported as NOMAD_TOKEN
nomad job dispatch echo-dispatch -message="Nomad dispatch works"

The command returns an allocation ID, e.g., Allocation ID: a1b2c3d4. Verify with:

nomad alloc status a1b2c3d4

You should see the task’s stdout containing the supplied message in the allocation’s event log.

Limitations and Practical Verification

Even with a correct setup, be aware of these limits:

• Dispatch does not support passing complex structured payloads beyond key‑value strings; for richer data, encode JSON in a string and parse it inside the task.
• The dispatch stanza cannot be combined with parameterized job stanzas that use meta for dynamic values; rely solely on the payload.
• Audit logging of dispatch requests is an Enterprise feature; in OSS you must rely on server logs.

To confirm that authentication and authorization are enforced:

1. Attempt a dispatch with a token lacking job:dispatch capability; the CLI returns permission denied and the server logs show an ACL deny entry.
2. If audit logging is enabled (Enterprise), query the audit store for a dispatch event matching the token’s accessor ID.

These checks give confidence that the design respects the intended trust boundaries before moving to production workloads.