HTTP 504 Gateway Timeout during high-cardinality PromQL execution

Query Timeout Behavior in Prometheus

Prometheus utilizes a global --query.timeout flag to prevent resource exhaustion from long-running queries. While the HTTP API allows for per-request timeout overrides, the server must manage the lifecycle of the underlying PromQL engine process when these limits are reached or when a client abruptly closes the connection.

In environments with high cardinality, there is uncertainty regarding the immediate reclamation of memory and CPU resources when a complex query is cancelled mid-execution, particularly when the engine is processing large datasets across shards.

• Does the server immediately terminate the execution thread upon receiving a client-side cancellation?
• What is the latency between the HTTP 504 response and the actual release of allocated memory by the PromQL engine?