InfluxDB Retention Policies: Why TSM Shards Make Data Deletion Efficient

The problem: time-series data outgrows disk faster than you can clean it

Time-series workloads are write-heavy by nature. A fleet of sensors emitting metrics every few seconds can generate millions of points per day, and before long your InfluxDB data directory is consuming terabytes while the queries you actually run only touch the last week of data. The instinct is to schedule a nightly DELETE or a cron job that trims old rows, but that approach fights the storage engine. InfluxDB is not designed to scan and remove individual points efficiently. The real solution is to let the storage structure handle expiration for you.

How TSM shard groups turn deletion into a file drop

InfluxDB stores data using the Time-Structured Merge Tree (TSM) engine. Rather than keeping rows in a general-purpose B-tree, TSM writes immutable files that are highly compressed and organized by time range. Those files are grouped into shard groups, and each shard group covers a contiguous time interval determined by the retention policy's duration and optional shard duration.

This structure matters because expiration stops being a row-level operation. When a retention policy's duration elapses, InfluxDB does not walk through indexes looking for timestamps older than the cutoff. Instead, it identifies the shard groups whose time ranges fall entirely outside the policy window and drops the underlying files. Deleting a directory of TSM files is fast and predictable, which is why retention policies scale to high-ingest environments where manual cleanup would not.

A worked example: defining and verifying a retention policy

The commands below assume InfluxDB 1.x with InfluxQL, where retention policies are first-class objects. In InfluxDB 2.x, the equivalent concepts are buckets and the compatibility API, so adjust accordingly.

Run the following in the InfluxDB CLI or via an authenticated HTTP POST to /query. You need database-owner or cluster-admin privileges to create a policy:

CREATE RETENTION POLICY "raw_30d" ON "sensors" DURATION 30d REPLICATION 1 SHARD DURATION 1d

This creates a policy named raw_30d on the sensors database. Data written without an explicit policy lands in the default aut policy, so you must either set raw_30d as the default or specify it at write time.

To verify that shard groups are being created, inspect the data directory on the host filesystem. The default path is /var/lib/influxdb/data/, though it may differ if you customized storage-data-dir. Under data/sensors/raw_30d/ you should see numbered shard directories, each representing a time-bounded container of TSM files:

ls /var/lib/influxdb/data/sensors/raw_30d/

To query against the policy, qualify the measurement with the retention policy name. In InfluxQL the separator is a dot:

SELECT mean("value") FROM "sensors"."raw_30d"."temperature" WHERE time > now() - 1h

If the policy is working, points older than 30 days will not appear in results, and the corresponding shard directories will eventually disappear from disk after the expiration check runs. If you need to remove the policy entirely, DROP RETENTION POLICY "raw_30d" ON "sensors" deletes the policy and all data it contains, so treat that as a destructive rollback, not a pause.

The limitations that surprise operations teams

Retention policies are powerful but not magical. Two behaviors regularly catch engineers off guard.

First, changing a policy's duration does not retroactively delete data that already exceeds the new limit. If you shorten raw_30d to 7d, InfluxDB will not immediately purge the existing 30 days of shard groups. The old shards remain until their original expiration boundaries are reached. If you need to reclaim space urgently, you must manually drop shards or the policy itself.

Second, downsampling with continuous queries (CQs) is the standard pattern for keeping long-term trends without paying for raw resolution, but it introduces a data-fidelity risk. A CQ that aggregates one-minute means into hourly averages and writes them to a downsampled_1y policy is useful. However, if the raw policy is set to expire after only seven days, the high-resolution signal is gone forever after that window. Once the raw TSM files are dropped, no amount of querying can reconstruct the original points. Choose raw retention based on how far back you need to debug anomalies, not on how much disk you hope to save.

Actionable checklist

• Audit your actual query patterns before setting durations. Debugging usually needs more history than dashboards.
• Create explicit retention policies for raw and downsampled data instead of relying solely on aut.
• Verify shard group creation by inspecting the data directory, not just by querying.
• Test downsampling CQs against a copy of production data before shortening the raw policy.
• Monitor shard group counts and disk usage; a sudden stall in shard expiration often signals a configuration or clock issue.

Retention policies work because they align data lifecycle with the TSM storage layout. Let the shard groups expire on their own schedule, and reserve manual intervention for the rare cases where you need to accelerate cleanup.