Diagnosing Nomad Job Placement Failures: Pending Jobs and No Eligible Nodes

Recognizable condition

A Nomad job stays in the pending state with the status "waiting for scheduler". Node evaluation logs (visible via nomad node status -self or the agent logs) repeatedly show messages such as "no eligible nodes" or "failed constraint evaluation".

Cause / diagnostic table

Symptom	Likely cause	Where to look
Node evaluation logs: "no eligible nodes"	Insufficient CPU/Memory on all nodes	Run nomad node status and inspect Used Resources vs Total Resources
Node evaluation logs: "failed constraint evaluation"	Job constraints or affinity/anti‑affinity rules cannot be satisfied	Examine the job’s constraint stanza and node attributes
Allocation logs show driver errors	Missing or incompatible task driver (e.g., docker, raw_exec, java)	Check nomad alloc logs <alloc> for driver‑specific messages
Node status shows drain flag	Node is drained or under maintenance	Look for drain in nomad node status <node> output
Placement warnings mention Consul service mesh	Consul agent unreachable or mis‑configured	Verify Consul connectivity from the Nomad agent

Ordered checks

1. Run nomad node status to confirm all nodes are ready and not marked drain. Look for any nodes with status == down or high utilization.

2. Run nomad node status -self (or nomad node alloc <node>) on a suspect node to see current CPU/Memory usage and compare with the job’s resource requests.

3. Examine the job placement plan: nomad job plan <job>. Look for warnings under the "Placement" section such as "Insufficient resources" or "Constraint evaluation failed".

4. If the plan shows no placement, fetch a recent allocation (even if failed) with nomad alloc status -pending and review its logs: nomad alloc logs <alloc>. Search for driver‑specific error strings (e.g., "docker: cannot connect to daemon").

5. If the job uses Consul Connect, test connectivity from the Nomad agent: consul members and consul catalog nodes. Nomad agent logs will also contain "failed to connect to Consul" messages when this is the issue.

Fixes tied to findings

Insufficient resources

• Scale the node pool: add new nodes or resize existing ones to increase CPU/Memory.
• If scaling is not immediate, reduce the job’s resources stanza (CPU/Memory) to fit current capacity.

Constraint or affinity mismatches

• Review the job’s constraint, affinity, and anti_affinity stanzas.
• Remove or relax conflicting constraints, or add matching node attributes (e.g., via node.meta or node.class).
• After changes, run nomad job plan <job> again to confirm the warning disappears.

Missing or incompatible task driver

• Verify the required driver plugin is installed on all nodes: nomad node status -self -json | grep -i Drivers.
• Install or upgrade the driver (e.g., for Docker: ensure docker daemon is running and the Nomad docker plugin version matches the Nomad version).
• Restart the Nomad agent on the affected nodes to load the new plugin.

Drained or maintenance nodes

• Undrain the node: nomad node drain -disable <node>.
• If the node is under scheduled maintenance, wait for the maintenance window to complete or reschedule the job.

Consul Connect issues

• Check Consul agent health: consul agent -info.
• Ensure the Nomad agent’s consul block points to the correct Consul address and token.
• Restart the Nomad agent after correcting the Consul configuration.

Verification

1. Run nomad job plan <job> and confirm that the "Placement" section shows no warnings and ends with "placement successful".
2. Submit the job (if not already) with nomad job run <job> or wait for the scheduler to pick it up.
3. Monitor the allocation: nomad alloc status <alloc>. The allocation should transition from pending to running within a minute on a healthy cluster.
4. Check node utilization again with nomad node status -self to ensure resources are now being consumed as expected and no node is over‑ or under‑utilized.

Limitations

• This guide assumes a Nomad version ≥ 1.2 where the nomad job plan command provides detailed placement warnings. Older versions may require inspecting agent logs directly.
• Resource adjustments should be followed by load testing; over‑provisioning wastes capacity and increases cost.
• Changing affinity/anti‑affinity rules can affect application availability; verify that the new rules still meet your redundancy requirements.

Escalation criteria

If after completing the checks and applying the corresponding fixes the job remains pending:

1. Collect Nomad agent logs (/var/log/nomad.log or journal) from all nodes for the time period when the job was evaluated.
2. Capture the output of nomad node status -json and nomad job plan <job> -json for further analysis.
3. Escalate to the platform team with the gathered logs, the job specification, and a summary of the steps already taken.