Aligning InfluxDB Shard Group Duration with Retention Policies for Efficient Data Expiry

When an InfluxDB instance starts to accumulate months of metrics, operators often notice that query latency climbs and disk usage stays high even after setting a short retention policy. The root cause is frequently a mismatch between the retention policy’s expiration time and the underlying shard group duration.

Understanding Retention Policies and Shard Groups

A Retention Policy (RP) defines how long data should be kept before it is eligible for deletion. InfluxDB does not delete individual points when they expire; instead, it stores data in physical files called shards. The time range covered by each shard is determined by the Shard Group Duration setting.

When data ages past the RP limit, InfluxDB can drop the entire shard file that holds that time range, which is far more efficient than scanning and deleting rows individually.

The Cost of Misalignment

• Shard duration longer than the RP: With a 24‑hour RP and a 30‑day shard duration, a day’s worth of data lives inside a 30‑day shard. InfluxDB cannot drop the shard after 24 hours, so it must resort to row‑level deletions, which are I/O‑heavy.
• Shard duration much shorter than the RP: A 1‑year RP paired with a 1‑hour shard duration creates thousands of tiny files. This can exhaust file descriptors and increase memory pressure as the engine tracks each shard’s metadata.

Practical Configuration: Aligning Lifecycle

In this example we want to keep high‑resolution data for seven days and ensure the shards are sized so they can be dropped automatically.

First, inspect the existing settings. Run these commands in the InfluxDB CLI (requires admin privileges on the target database):

# Show retention policies
SHOW RETENTION POLICIES ON my_database

# Show shard groups
SHOW SHARD GROUPS ON my_database

If the shard duration differs from the RP duration, create a new RP with matching values. Note that you cannot alter the shard duration of an existing RP; you must create a new one and direct new writes to it (or migrate old data).

# Create a 7‑day retention policy with a 7‑day shard duration
CREATE RETENTION POLICY short_term ON my_database DURATION 7d REPLICATION 1 SHARD DURATION 7d

After creation, verify the layout:

SHOW SHARD GROUPS ON my_database

You should see shard groups that each span approximately seven days.

Monitoring and Limitations

Once the policies are in place, monitor the shard count. A healthy setup shows the count rising as new shards are created and then dropping as old shards expire. You can expose the metric influxdb_shard (or influxdb_shard_count depending on your monitoring stack) and alert if the count grows without bound.

An important limitation: changing the duration of a retention policy does not trigger immediate deletion of existing data. It only affects the expiration logic for shards created after the change. To reclaim space right away after shortening an RP, you must manually drop the relevant shards or wait for the next cycle when those shards age out.

Takeaway: set the shard group duration to be equal to (or slightly larger than) your retention policy duration. This lets InfluxDB expire data by dropping whole files, keeping overhead low and query performance stable.