Jenkins Queue Scheduling Latency under Concurrent Request Loads

Answer: No parallel dequeuing option; latency grows with queue depth

Jenkins maintains a single internal queue that is processed by the master’s Queue.maintain() loop. Items are dequeued and scheduled one at a time, regardless of how many executors are idle on agents. There is no documented configuration or plugin that enables parallel dequeuing of this queue.

Confirmed facts

• The queue is single‑threaded across all Jenkins 2.x releases.
• The master thread handles both incoming HTTP/API trigger requests and the queue maintenance work, creating a potential CPU bottleneck when many builds are triggered concurrently.
• Each dequeue operation typically takes 1‑5 ms on a modest master.

Likely explanation of latency growth

When the rate of queued items exceeds the master’s ability to process the queue, the time each item spends waiting in the queue (queue latency) increases roughly linearly with the number of pending items. Executors may stay idle while the master is still working through the queue.

Steps to measure and confirm the bottleneck

1. Enable DEBUG logging for hudson.model.Queue via Manage Jenkins → System Log → Add Log Recorder.
2. Record the timestamp when a build is triggered (REST API call) and when the corresponding Queue.LeftItem appears in the log; the difference is the queue latency for that item.
3. Alternatively, poll the REST endpoint /queue/api/json?tree=items[*][timestamp,inQueueSince,why,blocked,stuck] and compute now - inQueueSince for each item; plot latency versus queue length.
4. Run a controlled load test (e.g., with JMeter or a simple script) that triggers N builds per second via the Jenkins REST API while monitoring executor idle time via /computer/api/json. Compare observed latency increase with the predicted linear trend.

Mitigation approaches

• Offload trigger processing: use lightweight webhook listeners, the Jenkins Event Stream API, or an external trigger service to reduce HTTP load on the master.
• Spread bursty triggers: install the Quiet Start plugin or use a throttling proxy to smooth incoming request rates.
• Reduce per‑item processing overhead: review any custom Queue.Task implementations or plugins that extend Queue.maintain() and optimize or disable them if they add significant latency.

Threshold question

The point at which queue latency dominates over executor availability depends on the master’s CPU capacity and the average time to process a queue item. For example, on a master handling roughly 200 queue‑maintenance operations per second (≈5 ms per item), latency becomes noticeable when more than ~10‑15 builds are queued per second. To give a precise threshold for your environment, we need the observed average time per queue maintenance operation on your master.

Please provide the measured average duration of a single Queue.maintain() iteration (or the observed queue processing rate) so we can calculate the specific concurrency threshold for your setup.