Accelerating Splunk time-series queries with tscollect

The short answer

tscollect pre-computes time-series summaries from indexed events, and tstats reads those summaries instead of scanning raw events. If a report or dashboard spends most of its runtime counting events over time, aggregating the underlying index and rewriting the query to use tstats is often the biggest speedup available without changing retention policy. The cost is freshness: the summaries only change when the aggregation search runs again.

How the mechanism works

A normal stats search reads raw events from index buckets and computes counts at query time. tscollect moves that work into a scheduled job: it scans the events you point it at, groups them by the fields and time spans you choose, and writes compressed tsidx aggregate files into a dedicated tscollect bucket. tstats then answers the same question by reading those files.

Two consequences follow from that design. Query latency drops because the engine reads a small summary rather than every event. Results are only as current as the last aggregation run, because there is no real-time path through this mechanism.

What gets written and where

Aggregate files land in the tscollect area of the index storage, alongside the normal index buckets. They are additional data, not a replacement for raw events, so plan for extra disk usage proportional to the number of fields and time spans you aggregate.

A worked configuration

The usual way to drive this is a scheduled saved search. In Splunk Web, go to Settings > Searches, reports, and alerts > New Search. The search is the filter for what you want summarized, followed by the tscollect command:

index=web sourcetype=access_combined earliest=-1d latest=now
| tscollect

Replace index=web and sourcetype=access_combined with your own index and sourcetype. Schedule it to run at whatever interval matches your freshness requirement, and set the search time range to cover the window you want re-aggregated. Creating or editing a scheduled search requires a role with permission to create saved searches in the target app. Running the search once manually after saving is the quickest way to confirm the command is accepted.

Some Splunk versions and documentation also describe declaring tscollect collections through a configuration stanza rather than a search. The exact stanza name and keys vary by release, so check the reference for the version you run before relying on that route; the scheduled-search approach above is the one that is easiest to verify end to end.

Rewriting a query to use tstats

Once the aggregation has run at least once, change the report to read the summaries. A raw-event version might look like this:

index=web sourcetype=access_combined
| timechart span=1h count

The accelerated equivalent:

| tstats count where index=web sourcetype=access_combined by _time span=1h

Note the shape difference: tstats takes its filter in a where clause and its grouping in a by clause. Fields used in the by clause must be ones the aggregation actually captured. If they are not, the query can quietly fall back to reading raw events and you lose the speedup without seeing an error.

Limits and common mistakes

• Stale results. Newly indexed events do not appear until the next aggregation cycle. Do not point latency-sensitive alerts at tstats-backed searches unless you accept that delay.
• Storage growth. Aggregates are extra data. Reported per-bucket ceilings exist in some deployments, but the practical limit is usually disk space and rebuild time, so monitor the tscollect area and the job duration.
• Field mismatch. If the fields in your tstats query do not match what was aggregated, Splunk may fall back to a raw search. The query still returns results, which is why the problem goes unnoticed.
• Overly broad windows. Aggregating a year of high-volume data in one pass can be expensive in CPU and I/O. Start with a narrow window and widen once you know the cost.
• Cluster replication. On indexer clusters the aggregate buckets follow the same replication rules as other buckets. If replication is under-provisioned, some peers may not hold the summaries and results can be incomplete or slower than expected.

Verifying the result

Do not trust the speedup alone; check that the numbers agree. Run both forms over the same window and compare totals:

| tstats count where index=web by _time span=1h

index=web | stats count by _time span=1h

Matching counts over a period the aggregation has already covered indicate the summaries are being read correctly. Differences usually mean the aggregation window does not cover the whole period, or that the fields do not line up. As a second check, open the Search Job Inspector for the tstats job and confirm which commands actually executed, then look in the tscollect storage area on the indexer for aggregate files with recent timestamps.

stats versus tstats on aggregated data