Job Inspector shows high command.search.index cost: which metric actually marks the bottleneck?

Question

I am profiling a slow Splunk search using the documented Job Inspector before attempting any optimization. The inspector reports per-command execution costs such as command.search.index and command.search.typer, plus a total count of events scanned, but I am unsure which of these numbers should be treated as the definitive bottleneck signal.

My uncertainty is interpretive rather than operational. A large event count early in the pipeline suggests the search retrieves more raw data than needed, while concentrated cost in a single non-streaming command (for example sort or transaction) suggests serialized work on the search head. In a distributed deployment the per-indexer view adds a third possibility: skewed data or one slow peer rather than a query problem. Search mode (Fast vs. Verbose) also changes the numbers substantially, so a single measurement may mislead.

Assume a recent Splunk Enterprise version; I understand metric names vary by release and will confirm against my installed documentation.

Which Job Inspector metrics should be checked first to distinguish an index-time retrieval problem from a search-head serialization problem?
Is there a documented way to decide when per-indexer runtime differences indicate data skew rather than normal variance?
Should Fast vs. Verbose mode comparisons be made before or after interpreting per-command costs?

Tasadduq Burney · Answer

Answer the question first
The definitive bottleneck signal is the Job Inspector metric that occupies the largest percentage of total search time. Compare the percentages for command.search.index, command.search.filter, and command.search.fetch (or any other per‑command cost shown). The metric with the highest share indicates where the search spends most wall‑clock time and therefore where optimization will have the greatest impact.
How to distinguish index‑time retrieval from search‑head serialization

Open the Job Inspector for the slow search.
  Record the % time for command.search.index (raw event read from disk).
  Record the % time for command.search.filter (streaming filtering on indexers/search head).
  Record the % time for command.search.fetch (field extraction and result fetching).
  If command.search.index dominates (>~40‑50% of total), the bottleneck is likely index‑time retrieval – consider narrower time ranges, better indexes, or storage latency.
  If command.search.filter dominates, the bottleneck is search‑head (or indexer) CPU‑bound filtering – look at inefficient regexes, WHERE clauses, or lack of indexed fields.
  If command.search.fetch dominates, the bottleneck is field extraction/result fetching – consider disabling unnecessary fields, using fields early, or enabling data model acceleration.

Per‑indexer runtime differences and data skew

Calculate the average runtime across all indexers for the same metric (e.g., command.search.index).
  Identify any indexer whose runtime exceeds roughly 2× the average (or shows a clear outlier).
  If such an outlier exists, investigate that peer’s data distribution (bucket size, hot/warm/cold balance) – it likely points to data skew rather than normal variance.
  If all indexers are within a tight band (e.g., ±20% of the mean), the variation is normal and the bottleneck is elsewhere.

Fast vs. Verbose mode considerations

Run the search in the mode you intend to use for production (usually Fast mode for dashboards).
  Collect the percentages in that mode.
  Only compare Fast vs. Verbose if you need to understand the overhead of Verbose; do not let Verbose numbers drive optimization decisions for Fast‑mode workloads.

One missing diagnostic detail that could change the recommendation
If the search uses a transforming command that forces a reduce phase (e.g., stats, chart, transaction), the bottleneck may shift to the search‑head reduce step, which is not captured by the three streaming metrics above. Knowing whether the search includes a reducing command would tell us to also examine command.search.reduce or command.search.aggregator percentages.

Job Inspector shows high command.search.index cost: which metric actually marks the bottleneck?

1 answer

Answer the question first

How to distinguish index‑time retrieval from search‑head serialization

Per‑indexer runtime differences and data skew

Fast vs. Verbose mode considerations

One missing diagnostic detail that could change the recommendation

Your perspective belongs here.

0 question comments