Job Inspector shows high command.search.index cost: which metric actually marks the bottleneck?
21K reputation · 13 Mar 2026, 19:31 UTC
I am profiling a slow Splunk search using the documented Job Inspector before attempting any optimization. The inspector reports per-command execution costs such as command.search.index and command.search.typer, plus a total count of events scanned, but I am unsure which of these numbers should be treated as the definitive bottleneck signal.
My uncertainty is interpretive rather than operational. A large event count early in the pipeline suggests the search retrieves more raw data than needed, while concentrated cost in a single non-streaming command (for example sort or transaction) suggests serialized work on the search head. In a distributed deployment the per-indexer view adds a third possibility: skewed data or one slow peer rather than a query problem. Search mode (Fast vs. Verbose) also changes the numbers substantially, so a single measurement may mislead.
Assume a recent Splunk Enterprise version; I understand metric names vary by release and will confirm against my installed documentation.
- Which Job Inspector metrics should be checked first to distinguish an index-time retrieval problem from a search-head serialization problem?
- Is there a documented way to decide when per-indexer runtime differences indicate data skew rather than normal variance?
- Should Fast vs. Verbose mode comparisons be made before or after interpreting per-command costs?