Query concurrency limit queue behavior change in Prometheus 3.0

Concurrency Slot Management and Timeouts

In both Prometheus 2.x and 3.0, a query that times out while waiting in the queue does not occupy a concurrency slot. A slot is only acquired when the query is dequeued and begins execution. Therefore, a timeout occurring during the queuing phase frees the request without ever having consumed a slot from the --query.max-concurrency limit.

If a query times out after it has acquired a slot (during execution), the slot is released immediately upon the termination of the query context. There are no documented guarantees of "slot leakage" in stable releases; the semaphore is managed via Go contexts that ensure release regardless of whether the query completed successfully or was canceled by a timeout.

Queue Ordering and Priority

Queue ordering is strictly FIFO (First-In, First-Out) across all query types. Prometheus does not differentiate between the following when placing them in the concurrency queue:

• Instant queries (API)
• Range queries (API)
• Recording rule evaluations
• Alerting rule evaluations

Because there is no priority inversion protection, a burst of heavy recording rules can block critical alerting evaluations or user-facing dashboard queries, increasing tail latency for all mixed workloads.

Verification Steps

To verify concurrency behavior in your specific environment, use a restricted configuration:

1. Set --query.max-concurrency=1.
2. Execute a long-running query (e.g., a large range query) to occupy the single slot.
3. Execute a second query with a very short client-side timeout.
4. Observe if the second query fails immediately without blocking the first, and if a third query (sent after the second fails) can still enter the queue and execute once the first finishes.

Technical Assumptions

This analysis assumes the use of the standard Prometheus server binary. If you are using a distributed wrapper or a proxy (like Thanos or Cortex), the concurrency limits may be handled at the gateway level, which may introduce different queuing behaviors.

Missing Diagnostic Detail: Are you observing specific latency spikes in prometheus_engine_queries_duration_seconds that correlate with rule evaluation cycles? This would confirm if internal workloads are causing the priority inversion you suspect.