Optimizing InfluxDB Data Lifecycle with Retention Policies and Shard Groups

The Problem

When a time‑series database grows, the real cost is not just storage but the cost of deleting old data. InfluxDB deletes data in whole blocks called shards rather than individual points, so the way those shards are sized determines how quickly and efficiently data ages out.

How Retention Policies Work

A Retention Policy (RP) tells InfluxDB how long a shard should live. Once a shard’s time range exceeds the RP duration, the entire shard is dropped. The RP does not scan individual points; it simply removes the file that holds them.

Shard Group Tuning

Data is split into shard groups, each covering a contiguous time span. The shardGroupDuration setting controls that span. Two extremes are common pitfalls:

• Too small – e.g., 1 h shards for a year of data – leads to thousands of tiny files, increasing metadata overhead and slowing queries.
• Too large – e.g., 30 d shards with a 7 d RP – forces InfluxDB to scan a huge file to drop a small portion of data, causing CPU and I/O spikes.

Tiered Storage Pattern

A practical engineering pattern is to use multiple RPs on the same database. A high‑resolution RP keeps raw data for a short period, while a low‑resolution RP stores aggregated data for long‑term analysis. Continuous Queries (or Tasks in v2+) automatically copy downsampled metrics from the raw RP to the long‑term RP before the raw data expires.

Practical Example

Below is a concrete example for a database called metrics. The goal is to keep raw data for 7 days and downsampled data for 90 days.

# 1. Create the database with a default 7‑day retention and 1‑day shard group
CREATE DATABASE metrics
CREATE RETENTION POLICY "raw_data" ON metrics DURATION 7d SHARD-GROUP DURATION 1d REPLICATION 1

# 2. Create a long‑term policy with 90‑day retention and 1‑month shard group
CREATE RETENTION POLICY "long_term" ON metrics DURATION 90d SHARD-GROUP DURATION 30d REPLICATION 1

# 3. Verify the policies
SHOW RETENTION POLICIES ON metrics

# 4. Check shard creation
SHOW SHARDS ON metrics

After running SHOW SHARDS, verify that the group column shows 1‑day and 30‑day windows matching the policies above.

Trade‑offs & Limitations

• Irreversible deletion – once a shard is dropped, all points in that time range are lost permanently.
• Changing an RP’s duration does not retroactively delete data; it only affects future shards.
• High tag cardinality can still cause memory pressure during shard deletion because the tag index must be updated.

Actionable Checklist

1. Run SHOW RETENTION POLICIES to confirm current durations.
2. Run SHOW SHARDS to verify shard group sizes match your retention strategy.
3. Test a short RP (e.g., 1 minute) and confirm data drops after the duration passes.
4. Monitor CPU and I/O during deletion periods; if spikes occur, consider adjusting shard group duration.
5. Use continuous queries or tasks to downsample high‑resolution data before it expires.

By aligning shard group duration with retention policy and using tiered storage, you keep I/O low, disk usage predictable, and query performance stable.