Using InfluxDB Retention Policies and Continuous Queries for Tiered Time‑Series Storage

Problem: Raw data fills storage fast

When you ingest high‑resolution metrics every second, the raw series can quickly consume disk space. Keeping all points forever makes long‑term queries slow and raises storage costs, yet you still need recent data at full resolution for alerting or debugging.

Solution: Tiered storage with retention policies and continuous queries

InfluxDB lets you define retention policies (RPs) that automatically drop data older than a chosen duration. By pairing a short‑lived RP for raw points with a longer‑lived RP that receives pre‑aggregated data from a continuous query (CQ), you create two tiers: high‑resolution recent data and lower‑resolution historical archives.

Worked Example: Setting up a 7‑day raw RP and a 90‑day downsampled RP

1. Create the database (if it does not exist)

   influx -execute "CREATE DATABASE mydb"

   Run on the InfluxDB server CLI; requires admin privileges.
2. Create a raw retention policy (7 days) and make it the default

   influx -execute "CREATE RETENTION POLICY \"raw\" ON \"mydb\" DURATION 7d REPLICATION 1 SHARD DURATION 1h DEFAULT"

3. Create a downsampled retention policy (90 days)

   influx -execute "CREATE RETENTION POLICY \"downsampled\" ON \"mydb\" DURATION 90d REPLICATION 1"

4. Define a continuous query that computes a 5‑minute mean and writes it to the downsampled RP

   influx -execute "CREATE CONTINUOUS QUERY \"cq_cpu_mean\" ON \"mydb\" BEGIN SELECT mean(value) AS value INTO \"downsampled\" .\"autogen\" .:MEASUREMENT FROM /.*/ GROUP BY time(5m),* END"

5. Verify the setup
   • List policies: influx -execute "SHOW RETENTION POLICIES ON mydb"
   • List CQs: influx -execute "SHOW CONTINUOUS QUERIES ON mydb"
   • Insert a test point two weeks old and a recent point, then query each RP:

   # Insert points (replace  with a number)
   influx -import -path= - < 1209600000000000000
   cpu,value=1  1000000000
   EOF
   # Query raw RP (should return only the recent point)
   influx -execute "SELECT * FROM cpu WHERE time > now() - 8d" -database mydb -retention-policy raw
   # Query downsampled RP (should show a 5‑minute aggregate)
   influx -execute "SELECT * FROM cpu WHERE time > now() - 8d" -database mydb -retention-policy downsampled
   

   Expected check: the raw RP query returns only the point inserted within the last 7 days; the downsampled RP query returns one or more rows with timestamps aligned to 5‑minute boundaries.

Trade‑offs and Limitations

• Continuous‑query overhead: Each CQ runs on the server and consumes CPU proportional to its frequency and the complexity of the aggregation. Running a CQ every minute on high‑cardinality data can noticeably reduce ingest throughput.
• Retention‑policy changes affect only future writes: Altering the DURATION of an existing RP does not retroactively drop older data. To enforce a new limit on existing points you must manually delete them or rewrite them into a new RP, which can be operationally heavy.
• Version note: In InfluxDB 1.x the feature is called continuous queries; in InfluxDB 2.x the equivalent is a Flux task. The concepts (RP + downsampling) remain the same, but the syntax differs.

Next Steps

Monitor server metrics (CPU, memory, write latency) after enabling a CQ. Adjust the aggregation interval or simplify the query if you observe sustained pressure on ingest. Periodically run SHOW RETENTION POLICIES and SHOW CONTINUOUS QUERIES to confirm that your policies match the intended tiers, and verify that old raw data is no longer returned by queries targeting the short‑lived RP.