SmartStore Cache Eviction on Low‑Traffic Indexes: Uncertain Search Latency

Short answer

SmartStore does not apply a per‑index fairness policy. When cached data exceeds max_cache_size, the cache manager evicts buckets based on recency of use — roughly least‑recently‑used behavior — across the indexer's cache as a whole. On a low‑traffic index, that is exactly the problem: your buckets are always the coldest thing in the cache, so they age out between infrequent searches and get refetched from remote storage almost every time. The eviction logic is working as designed; the workload is just pathological for it.

There is no published "recommended" max_cache_size, because the right value is derived from your searched working set, not your ingest rate. And a low cache hit ratio on a sparse index is not, by itself, evidence of misconfiguration.

Separating fact from likely explanation

Confirmed behavior: SmartStore keeps recently needed buckets in local indexer cache and downloads missing buckets from remote storage during search. When cache pressure exceeds configured limits, cached buckets are evicted to make room. Misses translate directly into remote fetch latency on the first search that touches those buckets.

Likely explanation for your latency spikes: with only a few MB/day ingested, the issue is not volume — it is the gap between searches. If users search once a day over a 30‑day window, every search arrives after the previous one's buckets have been evicted by other indexes' activity, so every search is effectively a cold search. Exact eviction internals and setting names are version‑sensitive in Splunk Enterprise 9.x, so confirm against your installed release's documentation before changing production values.

How to verify this is actually your problem

1. Run the same bounded search twice, back to back, over a fixed historical time range. A slow first run followed by a fast second run confirms cold‑bucket fetches, not query inefficiency.
2. In the Monitoring Console, look at SmartStore metrics — cache hits, misses, remote downloads, evictions, and bytes cached — filtered to the suspect index and indexer. Correlate eviction timestamps with your latency spikes.
3. Estimate the real working set: sum the on‑disk bucket sizes for the retention window users actually search, then add headroom for concurrent searches and replication. Compare that number with your current max_cache_size and observed eviction counts.

Sizing guidance

For a few MB/day, the math usually favors keeping the index cache‑resident rather than tuning a global threshold:

• If the commonly searched window is, say, 90 days at 5 MB/day raw, the compressed on‑disk footprint is small — likely well under a few GB. Setting max_cache_size to comfortably hold that working set plus headroom eliminates the eviction cycle at trivial local‑disk cost.
• If stable latency matters more than minimal disk, consider keeping this low‑traffic index on local storage entirely and reserving SmartStore for high‑volume indexes where remote storage actually saves money. SmartStore's cost benefit scales with data volume; on a tiny index, the savings are negligible while the latency penalty is real.
• Do not lower a global cache threshold to "force" efficiency — that increases evictions for every index sharing the cache.

Cautions

Do not judge health by hit ratio alone; sparse access patterns produce low hit ratios even when everything works correctly. Raising cache size shifts cost back to local disk and can mask underlying remote‑storage or network problems that also appear as latency spikes. Change one variable at a time, ideally on a test indexer or in a maintenance window, and record search runtime, fetch count, and eviction count before and after each change.

One detail that would sharpen this recommendation: the typical time range and frequency of searches against this index. If searches are rare but span a year of data, full cache residency may not be practical, and accepting first‑search latency (or scheduling a warm‑up search) becomes the better trade‑off.