Search latency appears only under concurrent request load

Direct Answers

Question 1: Yes — when concurrent search requests exceed the configured thread-pool size (default: number of available processors), the search thread-pool queue size grows proportionally with observed latency. Each queued request waits approximately (queue_position / throughput) before a thread becomes available.

Question 2: Yes — latency can persist even when the thread-pool queue size remains at zero. This indicates the bottleneck is not queuing but increased request processing time caused by CPU contention, GC pauses, I/O saturation, or background operations (refresh, translog flush, segment merges) competing for the same resources.

Likely Explanation vs. Confirmed Facts

Confirmed (Elasticsearch/OpenSearch 7.x–8.x, default fixed thread-pool, bounded queue)

• Search thread-pool queue size is a point-in-time metric; it only grows when active == threads and new requests arrive.
• Rejected count increments only when the queue is full (default 1000). Latency can rise significantly before any rejections appear.
• Refresh operations (default 1s interval) cause brief (~ms) latency spikes when new segments are published; they do not queue requests but can slow in-flight queries.
• Translog flushes are asynchronous by default (flush_threshold_size: 512mb) and rarely block search unless durability settings force synchronous flushes.
• Segment merges run on a separate merge thread pool; they compete for I/O and CPU but do not directly queue search requests.

Likely but Requires Verification in Your Cluster

• If queue_size > 0 correlates with p99 latency spikes → queuing is the primary contributor.
• If queue_size == 0 while p99 spikes → search threads are spending more time in CPU, GC, I/O wait, or lock contention (e.g., global refresh lock).
• Heavy merge activity often manifests as elevated search latency without search queue growth.

Diagnostic Steps (Run During Load Test)

1. Collect thread-pool stats at 1–2s intervals:

   GET _nodes/stats/thread_pool?level=node

   Plot thread_pool.search.queue_size and thread_pool.search.rejected against client-side p50/p99 latency.
2. Capture hot threads during a latency spike:

   GET _nodes/hot_threads?interval=500ms&type=cpu,wait,block

   Look for search threads in WAIT (I/O, lock) or BLOCKED (synchronization) states.
3. Check background operation counters:

   GET _stats/refresh,flush,merge

   Correlate deltas in refresh_total_time_in_millis, flush_total_time_in_millis, and merge_total_time_in_millis with latency spikes.
4. Enable search slowlog temporarily:

   PUT _settings
   {
     "index.search.slowlog.threshold.fetch.warn": "100ms",
     "index.search.slowlog.threshold.query.warn": "100ms"
   }

   Determine whether individual queries slow down or queue wait dominates.

One Missing Diagnostic Detail

What is the sampling interval of your current thread-pool metrics collection? If it is longer than 2 seconds, transient queue spikes may be missed, leading to a false "zero queue" observation. Sub-second sampling (1–2s) is required to confidently rule out queuing.