Ensuring Total Cluster Coverage with Nomad System Jobs

The Infrastructure Agent Dilemma

When managing a cluster, there are two types of workloads. Most are Service Jobs: applications that need to run X number of copies, and the scheduler decides where they fit. Then there are infrastructure agents—log forwarders, monitoring exporters, and security scanners—that must run on every single node to be effective.

If you try to manage these as Service Jobs, you face a manual scaling nightmare. Every time you add a new server to your pool, you have to increment the count. If a node fails and is replaced, you have a blind spot in your monitoring until the count is updated. The solution is the System Job, a specific Nomad job type designed to maintain 100% coverage automatically.

How System Jobs Differ from Service Jobs

A standard Service Job asks: "Run 5 instances of this app anywhere they fit." A System Job asks: "Run one instance of this app on every node that meets my criteria."

The primary technical shift is in how the scheduler handles scaling. With a System Job, the node itself becomes the trigger. When a new node joins the cluster and registers with the Nomad server, the scheduler immediately identifies that the node is missing the required system task and deploys it. Conversely, when a node leaves, Nomad simply removes that specific instance from the tracking list without attempting to "reschedule" it onto another node, as that would violate the one-per-node requirement.

Targeting Specific Hardware with Constraints

Rarely is a cluster perfectly homogeneous. You might have some nodes running Ubuntu and others running RHEL, or some with NVMe drives and others with standard SSDs. Running a kernel-level monitoring agent on an incompatible OS will lead to a crash loop.

System Jobs use constraints to filter the target list. A constraint is a requirement that a node attribute must match a specific value. For example, you can ensure a log-shipper only deploys to nodes where attr.kernel.name == "linux". This allows you to maintain a single job definition that intelligently ignores incompatible nodes while still ensuring total coverage across the supported fleet.

Example: Deploying a Cluster-Wide Monitoring Agent

Below is a configuration for a hypothetical monitoring agent. Note the type = "system" declaration, which tells Nomad to ignore the count parameter and target all eligible nodes.

# Run this via: nomad job run monitoring-agent.hcl
job "cluster-monitor" {
  type = "system"

  group "agent" {
    # Ensure we only run on Linux nodes to avoid crashes on Windows nodes
    constraint {
      attribute = "${attr.kernel.name}"
      value     = "linux"
    }

    restart {
      attempts = 3
      delay    = "15s"
    }

    task "prometheus-node-exporter" {
      driver = "docker"

      config {
        image = "prom/node-exporter:latest"
        ports = ["9100/tcp"]
      }

      resources {
        cpu    = 100 # MHz
        memory = 64  # MB
      }
    }
  }
}

Verification Steps

• Initial Deployment: Run the job and check the Nomad UI. You should see one allocation per Linux node.
• Scaling Test: Join a new Linux node to the cluster. Observe the "Allocations" tab; the agent should deploy automatically without updating the job file.
• Constraint Test: If you have a Windows node in the cluster, verify that no allocation is attempted for that node.

The Resource Exhaustion Risk

The most significant risk with System Jobs is resource amplification. In a Service Job, if you set a memory limit of 1GB for 3 instances, you use 3GB of cluster RAM. In a System Job, that 1GB is multiplied by every node in your cluster.

If you deploy a heavy agent to a 100-node cluster, you are committing 100GB of RAM across the fleet. If the agent has a memory leak, it won't just crash one app; it can trigger Out-Of-Memory (OOM) kills across your entire infrastructure, potentially knocking out the very services the agent was meant to monitor.

Managing Updates

Updating a System Job triggers a rolling update across the entire cluster. To avoid losing visibility into your infrastructure during an upgrade, define an update block with a max_parallel value. This ensures that only a small percentage of your agents are offline at any given time, maintaining a baseline of monitoring data while the new version rolls out.

Rollback Procedure

Because System Jobs change the state of every node, a bad update can be disruptive. To revert to the previous stable version, use the following command on the Nomad CLI:

# Reverts the job to the previous stable version
nomad job revert cluster-monitor