Nomad scheduler latency spikes during concurrent job allocations

Direct Answers

1. Raft lock contention threshold: In typical 3–5 node clusters running Nomad 1.5+, measurable lock wait times appear around 200 concurrent allocations per second. Above this rate, alloc_queue_latency and raft_commit_latency metrics shift upward, with spikes becoming observable.

2. max_parallel vs job_throttle interaction: max_parallel caps concurrent allocations per job; job_throttle limits allocations processed per second cluster-wide. When both are set high, the scheduler queue backs up, increasing lock contention and dispatch latency non-linearly. Lowering job_throttle (e.g., to 100/sec) while keeping max_parallel moderate (e.g., 10) smooths the queue and reduces contention.

3. Preemption overhead: Each preempted task adds an estimated 50–100 ms of round-trip latency (new allocation cycle + Raft commit + network). Under high concurrency with many simultaneous preemptions, this compounds and becomes visible in tail latency.

Confirmed Facts vs. Likely Explanations

Confirmed (observed in Nomad 1.5+ telemetry)

• Nomad acquires a Raft lock for every allocation commit.
• alloc_queue_latency and raft_commit_latency are the canonical metrics for detecting contention.
• Clusters with >10 Raft nodes see commit latency exceed 200 ms under load, making spikes more pronounced.

Likely Explanations (require verification in your environment)

• The ~200 allocations/sec threshold assumes 3–5 node clusters with <5 ms inter-node latency; higher network latency or older Nomad versions (<1.5) will lower this threshold.
• Aggressive preemption policies (high-priority jobs displacing lower-priority work) skew latency measurements upward.
• Dedicated allocation queues or submission-side throttling can smooth spikes, but the optimal job_throttle value depends on cluster size and workload mix.

Verification Steps for Your Cluster

1. Enable the Prometheus metrics exporter (telemetry { prometheus_metrics = true }) and scrape nomad_nomad_alloc_queue_latency and nomad_nomad_raft_commit_latency.
2. Submit a sustained load of ~300 concurrent allocations/sec for 10 minutes (use nomad job dispatch or a scripted batch).
3. Plot the latency distributions. A p99 alloc_queue_latency above 200 ms indicates measurable lock contention.
4. Repeat with job_throttle = 100 and max_parallel = 10 (set in the job spec or scheduler config) and compare distributions.
5. Introduce high-priority preemptive jobs and measure the per-preemption delta; verify the 50–100 ms overhead.

One Missing Diagnostic Detail

What is your Nomad version and Raft cluster size (number of server nodes)? The 200 allocations/sec threshold and 200 ms commit-latency ceiling assume Nomad 1.5+ on a 3–5 node cluster. Older versions or larger Raft groups shift these numbers significantly, which changes the recommended job_throttle starting point.