Query concurrency limit change in InfluxDB 2.7 leads to higher latency for complex Flux queries

To determine if latency in InfluxDB 2.7 is caused by scheduler contention or storage I/O, you must decouple wait-to-start from execution-time.

Identifying Scheduler Bottlenecks

>When the query-concurrency-limit is reached, new queries are queued before execution begins. To identify if the scheduler is the bottleneck, monitor the following internal metrics (typically via the _internal database or the /metrics endpoint):

• influxdb_fluxquery_queued_duration: A high value here indicates queries are waiting for an available slot in the concurrency pool.
• influxdb_query_active_count: If this consistently sits exactly at your configured concurrency limit, the scheduler is actively throttling requests.
• flux_task_execution_duration: While specific to tasks, high spikes here alongside queued queries suggest the engine is saturated.

Distinguishing Storage I/O Pressure

If queries are executing immediately but taking longer than usual, the issue is likely I/O or CPU-bound. Distinguish this by looking at:

• influxdb_storage_engine_read_latency: Increased latency in this metric indicates that the disk subsystem cannot keep up with the read requests from the Flux engine.
• influxdb_storage_read_operations_total: If this peaks while latency rises, the storage engine is struggling with the volume of data retrieval.
• CPU IOWait (System Level):: High %iowait on the host OS confirms the processor is idling waiting for storage responses rather than processing Flux logic.

Isolating the Impact via Testing

To isolate the effect of the query-concurrency-limit, use the following reproducible workload characteristics:

1. Query Shape: Compare a high-cardinality "point lookup" (e.g., filter() on a unique tag) against a "heavy aggregation" (e.g., aggregateWindow() over a large time range). If only the aggregation slows down under load, it is CPU/Memory contention.
2. Time Range Scaling: Narrow the time range. If latency drops significantly with smaller ranges despite high concurrency, the bottleneck is I/O throughput during de-scans.
3. Concurrency Step-up: Run a burst of simple, fast queries. If these also experience latency when the concurrency limit is hit, the scheduler is causing queuing-induced starvation.

Note: Use the influx.profile() function within your Flux scripts to see which specific operators (e.g., joins, pivots) are consuming the most clock time during execution.