Diagnosing Nomad Job Pending Issues with the Autoscaler Feature

Recognizable Condition

Jobs stay in the pending state longer than expected while Nomad client nodes report sufficient CPU, memory, and disk resources and are not drained.

Cause/Diagnostic Table

Possible Cause	What to Look For
Autoscaler plugin not evaluating	No autoscaler events in server logs; policy not triggered.
Misconfigured scaling policy (min/max limits)	Policy file shows min/max values higher than current node count or target metric thresholds too high.
Autoscaler plugin disabled	Plugin stanza missing or set to disabled in agent config.
Quota limits preventing new allocations	Quota usage shows allocated resources at or near limit for the job’s namespace.
Insufficient nominal node count for trigger	Number of ready nodes is below the autoscaler’s min_ready_nodes or evaluation window requires more nodes.

Ordered Checks

1. Check Nomad server logs for autoscaler events: journalctl -u nomad | grep -i autoscaler (requires read access to system logs or Nomad agent log file).
2. Verify client node attributes and drain status: nomad node status -self or nomad node status (Nomad token with node read permission).
3. Inspect the autoscaler policy file: look for policy.hcl in the plugin directory; confirm target_metric, threshold, min, max values.
4. Confirm Nomad client registration and health: nomad node status (requires node read token).
5. Review quota usage: nomad quota usage -namespace (requires quota read permission).

Fixes Tied to Findings

• If autoscaler logs show no evaluation: ensure the plugin is enabled in the agent config (plugin "nomad-autoscaler" { config = { ... } }) and restart the Nomad agent (sudo systemctl restart nomad).
• If policy thresholds are too high: lower the threshold or adjust min/max in the policy file, then reload the autoscaler (nomad autoscaler policy apply -policy policy.hcl) which requires a plugin restart or agent reload.
• If client nodes are drained: mark them ready with nomad node eligibility -enable (node write permission).
• If quota exceeded: increase quota limits via nomad quota apply -namespace quota.hcl or request an infrastructure increase from the platform team.
• If node count insufficient: add healthy client nodes to the cluster or adjust the autoscaler’s min_ready_nodes to reflect the current capacity.

Escalation Criteria

• The autoscaler plugin fails to start after agent restart (check logs for repeated plugin load errors).
• Node health checks continue to fail despite correct eligibility and resource availability.
• Quota adjustments require changes to underlying infrastructure (e.g., requesting new host machines) that are beyond the operator’s scope.

Verification Steps

1. Run nomad node status and confirm the expected number of nodes show ready and available.
2. List active autoscaler policies: nomad autoscaler policy list and verify the updated policy appears.
3. Submit a test job with known resource requests (e.g., a simple redis container) and watch nomad job status transition from pending to running within the expected timeframe.
4. Enable debug logging temporarily (nomad agent -log-level=debug) and check for autoscaler evaluation messages matching the policy thresholds.

Limitations

• The Nomad Autoscaler only works with drivers that expose the required task metrics (e.g., docker, exec, java).
• Metric collection must be enabled (telemetry block) and the autoscaler must have access to the metric source (Prometheus, StatsD, etc.).
• Changes to the autoscaler plugin or policy require a Nomad agent restart; test modifications in a staging cluster before applying to production.
• Avoid setting min/max values that could cause thrashing; monitor the evaluation window to prevent rapid scale‑up/down cycles.