InfluxDB Retention Policy and Shard Group Duration: Choosing the Right Settings for Automatic Data Deletion

Decision Overview

You need to configure InfluxDB so that data older than a certain period is automatically removed. The key choices are the retention policy definition and the shard group duration that determines how data is stored on disk.

Constraints

The retention policy must be defined per measurement or tag set. Shard group duration should align with the retention period to avoid creating many tiny shards. You must have administrative privileges to create or drop policies, and the drop operation requires the drop permission.

The CREATE RETENTION POLICY command requires the admin or write privilege; the DROP RETENTION POLICY command requires the drop privilege, which is granted via the admin role or a custom role with the drop permission.

Option Comparison

Retention Policy Scope	Shard Group Duration	Typical Use Case	Pros	Cons
Per measurement (e.g., sensor_data)	1 day	High‑frequency telemetry kept short	Precise deletion, minimal storage waste	More shards, higher metadata overhead
Per tag (e.g., environment=prod)	7 days	Long‑term monitoring of production data	Simplifies policy management, reduces shard count	May keep unnecessary data if tag scope is too broad
Global (no tag filter)	30 days	General purpose data retained longest	Single policy, easy to maintain	All data shares the same retention window, less flexibility

Trade‑off Analysis

Choosing a shorter shard group duration than the retention period creates many small shards, which increases write overhead and can degrade query performance because the engine must scan more files. Conversely, a longer shard group duration can lead to very large shards that take longer to compact and may cause higher memory usage during queries. The goal is to set the shard group duration close to, but not exceeding, the longest retention period you need.

Concrete Implementation

Example: create a retention policy that keeps data for 7 days on the measurement sensor_data and set shard groups to 1 day.

# Connect with admin credentials
influx -host localhost -port 8086 -execute "CREATE RETENTION POLICY rp_7d ON mydb DURATION 7d REPLICATION 1 DEFAULT"
# Set shard group duration to 1 day
# First, identify the shard group that contains the data
SHOW SHARD GROUPS
# Then alter that group (replace <shard_group_name> with the actual name)
ALTER SHARD GROUP <shard_group_name> DURATION 1d

Replace <shard_group_name> with the name of the shard group that contains the data you want to manage. You can discover the group name with SHOW SHARD GROUPS.

Verification Steps

After creation, verify the policy exists:

SHOW RETENTION POLICIES ON mydb

Check that the policy shows a 7‑day duration and that the shard group duration matches 1 day.

To confirm that data will be deleted, you can manually drop the policy (requires drop permission) and observe the effect:

DROP RETENTION POLICY rp_7d ON mydb

Run SHOW SHARD GROUPS again; any shards older than 7 days should disappear. If they remain, the shard group duration may not align with the retention period.

You can also inspect the size of shards with the du command on the InfluxDB data directory, ensuring that old shards are being removed:

du -sh /var/lib/influxdb/engine/tsm/* | sort -h

Compaction Configuration

Auto‑compaction of TSM files improves read performance. Set the maximum number of concurrent compactions in the InfluxDB configuration file (influxdb.conf):

[influxd]
max-concurrent-compactions = 4

After changing the setting, restart the service and verify the value with:

SHOW DIAGNOSTIC INFORMATION

Look for the max_concurrent_compactions entry.

You can also monitor active compactions with:

SHOW STATISTICS

The output includes the number of ongoing compactions and the size of TSM files, helping you tune max-concurrent-compactions for your workload.

Risk and Rollback Considerations

Creating or altering a retention policy changes data visibility but does not delete data until the policy’s expiration or a manual DROP. If you need to revert, you can recreate the original policy with the same parameters before the change, or restore from a backup. Because the DROP command permanently removes data, ensure you have a recent backup or that the policy truly covers data you intend to lose.

Before executing a DROP, confirm that the target retention policy is the only one affecting the data you intend to lose by listing all policies:

SHOW RETENTION POLICIES ON mydb

This prevents accidental deletion of unrelated data.

Conclusion

Aligning shard group duration with the intended retention period minimizes unnecessary shards and compaction overhead while ensuring automatic deletion works as expected. Use the CREATE RETENTION POLICY and ALTER SHARD GROUP commands to configure, verify with SHOW commands, and, when needed, DROP the policy with the required permissions. Monitoring diagnostic information and compaction settings completes a robust data‑lifecycle strategy in InfluxDB.

Best practice is to test retention changes on a non‑production measurement first, then apply the same steps to critical measurements.

Regularly review the retention policy and shard group settings after any schema change or new data type is introduced, as these events can affect the optimal duration.