Nomad scheduler latency spikes during concurrent job allocations

Diagnosing Scheduler Latency Under Concurrent Allocations

Nomad's scheduler uses a best-fit decreasing bin-packing algorithm that maps jobs to available resources. When multiple jobs are submitted simultaneously, the scheduler must acquire locks on the Raft consensus group to update the global allocation state. This lock contention can introduce request-level delays that are not observable during sequential job submissions.

The interaction between max parallel and job throttle parameters affects how many jobs enter the allocation pipeline concurrently, but the measurable impact on end-to-end dispatch latency is not well documented. Additionally, preemption behavior under high concurrency may trigger re-scheduling of running tasks, adding further round-trip latency.

Latency measurements will vary based on cluster size, network topology, and client version, making it challenging to establish consistent thresholds.

Key Questions

• At what concurrent allocation rate does the Raft lock contention become a measurable bottleneck in typical clusters?
• How do max parallel and job throttle settings interact to affect dispatch latency under sustained load?
• Does preemption of running jobs during high concurrency introduce predictable latency overhead compared to non-preemptive scheduling?