Ensuring Global Coverage: Using Nomad System Jobs for Infrastructure Agents

The 'Missing Agent' Problem

When managing a cluster, you often need a tool like a log forwarder, monitoring agent, or security scanner to run on every node. With a standard service job you must manually update the count each time you scale. Forget to increment it after adding nodes and you get blind spots in observability or security.

The solution is the System Job. Unlike service jobs that target a specific number of instances, a system job targets the infrastructure itself and ensures every node meeting your criteria runs the workload automatically.

How System Jobs Differ from Service Jobs

In Nomad the type parameter changes the scheduling logic.

• Service Jobs: The default. You request N instances and Nomad places them on the best nodes.
• System Jobs: You declare a system-level requirement. Nomad ensures one instance runs on every eligible client node in the cluster.

When a new node joins and registers with the Nomad server, the scheduler immediately deploys pending system jobs to the new node without manual intervention.

Applying Constraints to Global Workloads

You rarely want a system job on literally every node. A kernel agent may only work on Linux, or a GPU monitor should only run on nodes with NVIDIA hardware.

Nomad handles this with constraints. A system job only deploys to nodes that satisfy the requirements. Nodes that fail the check are skipped as incompatible rather than treated as a scheduling failure.

Worked Example: Deploying a Log Collector

The following job is configured to run on every Linux node. Run it from a machine with the Nomad CLI configured, using an ACL token with write permissions for jobs:

# Run with: nomad job run log-collector.hcl
# Requires a Nomad ACL token with job write access
job "log-collector" {
  type = "system"

  constraint {
    attribute = "${attr.kernel.name}"
    value     = "linux"
  }

  group "agents" {
    task "fluent-bit" {
      driver = "docker"

      config {
        image = "fluent/fluent-bit:latest"
      }

      resources {
        cpu    = 100
        memory = 128
      }
    }
  }

  update {
    max_parallel = 1
    health_check = "checks"
  }
}

Verification

1. Initial deployment: run the job and check the Nomad UI for one allocation per Linux node.
2. Scaling test: join a new Linux client and confirm the task starts automatically.
3. Constraint test: join a Windows node and confirm the job is not scheduled there.

The Risk of Global Failure

System jobs can cause correlated failure. A bad configuration propagates to every node at once.

Mitigate this with a careful update block. Setting max_parallel = 1 forces Nomad to update one node at a time. If the first update fails its health check, the rollout stops before causing a cluster-wide outage.

Resource Guardrails

Infrastructure agents can be noisy neighbors. Define strict resource limits per node. Since the job runs everywhere, a small leak aggregates into significant waste across a large cluster.