Question
HTTP 504 Gateway Timeout during high-cardinality PromQL execution
Ari Nova
0 reputation · 23 Jan 2020, 01:11 UTC
67.9K views
Loading…
0 reputation · 23 Jan 2020, 01:11 UTC
Prometheus utilizes a global --query.timeout flag to prevent resource exhaustion from long-running queries. While the HTTP API allows for per-request timeout overrides, the server must manage the lifecycle of the underlying PromQL engine process when these limits are reached or when a client abruptly closes the connection.
In environments with high cardinality, there is uncertainty regarding the immediate reclamation of memory and CPU resources when a complex query is cancelled mid-execution, particularly when the engine is processing large datasets across shards.
34775 reputation · 23 Jan 2020, 05:45 UTC
When a client cancels a request or a gateway enforces a timeout during a high-cardinality PromQL execution, Prometheus does not always terminate the underlying PromQL engine thread instantly. The HTTP 504 response typically originates from the upstream gateway nginx ALB etc when its configured request timeout expires before Prometheus returns results.
Latency between the 504 response and actual resource release depends on the PromQL engine's cancellation point. In high-cardinality scenarios, result set expansion across label combinations can delay cleanup by several seconds to minutes.
Recommended steps for this case:
To refine the recommendation, please provide:
the observed query execution time in the Prometheus UI for the same high-cardinality query, and the gateway's current timeout values.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 23 Jan 2020, 12:11 UTC
The key distinction is that Prometheus uses cooperative cancellation via Go's context.Context, not preemptive thread termination. When a gateway returns 504, it's typically because:
prometheusproxy_read_timeout or ALB idle timeout) fired before Prometheus could complete and respondIn high-cardinality queries, the PromQL engine may have already allocated substantial intermediate memory before reaching the next cancellation check point. Go's garbage collector then controls actual memory reclamation—there's no guarantee of immediate release even after the query goroutine exits.
To confirm what's happening in your environment:
--query.timeoutprometheus_engine_queries and go_goroutines during query executioncurl with extended timeout to see if the same query completes when given more timeThis explains why simply increasing gateway timeouts often resolves 504s without changing Prometheus configuration—the proxy was cutting off a query that would have completed.