Nomad scheduler tick interval limits task dispatch latency

Determining Scheduler Tick Bottlenecks

The Nomad scheduler tick interval does not act as a hard limit on dispatch latency for all jobs, but it defines the minimum window for the scheduler to evaluate pending jobs and trigger new allocations. While the default is typically 100ms, the perceived latency is often a composite of the tick interval, the evaluation cycle, and client-side task startup (such as image pulls).

Likely Explanation vs. Confirmed Behavior

Confirmed Behavior: The scheduler operates on a periodic cycle. If a job is submitted immediately after a tick, it must wait for the next cycle to be evaluated. Reducing this interval decreases the theoretical minimum wait time before an evaluation is triggered.

Likely Explanation: In most production environments, a 100ms tick is rarely the primary bottleneck. Dispatch latency is more frequently caused by the evaluation process (calculating the best-fit bin-packing) or client-side overhead. If you observe latency in the seconds rather than milliseconds, the tick interval is likely not the root cause.

Verification and Diagnostic Steps

To determine if the tick interval is the bottleneck, follow these steps:

1. Measure Evaluation Delta: Compare the timestamp of the job submission with the Created time of the resulting allocation. If this delta consistently aligns with your tick interval (e.g., always ~100ms), the tick is a factor.
2. Monitor Server CPU: Observe the CPU utilization of the Nomad server leader. If you reduce the tick interval, monitor for a proportional increase in nomad.scheduler.evaluations and CPU spikes.
3. Isolate Client Latency: Check the allocation events. If the gap exists between Placement and Running, the bottleneck is the client (e.g., Docker image pull) and not the scheduler tick.

Recommended Range and Overhead

There is no single "golden" range, as it depends on the cluster size and job churn rate. However, the following general guidelines apply:

• Default (100ms): Balanced for most general-purpose workloads.
• Aggressive (<100ms): May reduce latency for high-frequency short-lived tasks but increases CPU overhead on the server leader and may lead to scheduler instability under heavy load.
• Conservative (>100ms): Reduces CPU overhead; suitable for long-running services where a few hundred milliseconds of dispatch latency is negligible.

Missing Diagnostic Detail: To provide a more precise recommendation, please provide the average number of concurrent evaluations per second during your benchmark. High evaluation volume can cause queuing delays that mimic tick latency.