Cutting Time-Series Storage Costs with Elasticsearch ILM Rollover and Shrink

The problem: time-series indices that never stop growing

If you push logs, metrics, or events into Elasticsearch, the write index grows until something forces a split. Oversized indices mean uneven shard distribution, heavier heap pressure, and slower queries. The manual alternative—hand-creating new indices, flipping aliases, and shrinking old ones—works until someone forgets a step at 2 a.m.

Thesis: let ILM run the lifecycle for you

Index Lifecycle Management (ILM) moves an index through phases (hot, warm, cold, delete) based on age, size, or document count. Attach the policy to an index template and every new index inherits the same rules: rollover keeps the write index at a sane size, and shrink consolidates old read-only indices to fewer shards, cutting per-shard overhead without losing query access.

What each phase does

• Hot: the index accepts writes. A rollover action fires when the index crosses a size (e.g., 50 GB) or age (e.g., 7 days) threshold, creating a fresh write index.
• Warm: the index is read-only. A shrink action reduces primary shards (e.g., 3 to 1), lowering storage and cluster-state overhead.
• Cold: optional further downscaling or forcemerge for rarely queried data.
• Delete: the index is removed after your retention period (e.g., 30 days).

Worked example: policy, template, and bootstrap index

Run these against your cluster (Kibana Dev Tools or curl) with a user that has cluster management privileges. Assumes Elasticsearch 7.8 or later for the composable _index_template API; on older 7.x versions use the legacy _template endpoint with the same settings.

1. Create the ILM policy. Rolls over at 50 GB or 7 days, shrinks to one primary shard a day after rollover, deletes after 30 days:

PUT _ilm/policy/timeseries_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_size": "50GB", "max_age": "7d" }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "shrink": { "number_of_shards": 1 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

2. Create the index template so matching indices pick up the policy automatically:

PUT _index_template/timeseries_template
{
  "index_patterns": ["timeseries-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.lifecycle.name": "timeseries_policy",
      "index.lifecycle.rollover_alias": "timeseries_write"
    }
  }
}

3. Bootstrap the first write index with the rollover alias marked as the write index:

PUT timeseries-000001
{
  "aliases": {
    "timeseries_write": { "is_write_index": true }
  }
}

4. Verify the policy attached:

GET timeseries-000001/_ilm/explain

Expect the index to report the hot phase and timeseries_policy as its lifecycle.

5. Test rollover manually (optional). Because the alias already points at a write index and the policy carries the conditions, you can trigger rollover directly:

POST timeseries_write/_rollover?wait_for_completion=true

A new index (e.g., timeseries-000002) becomes the write index and the old one begins transitioning toward warm. For a fast end-to-end test, temporarily lower max_size to something like 1mb in a staging policy and bulk-index a few documents.

6. Watch the shrink step:

GET timeseries-000001/_ilm/explain
GET timeseries-000001/_settings

After shrink completes, the settings should show number_of_shards: 1. Note the shrunk index may carry a name like shrink-<original> depending on version defaults, so check _cat/indices/timeseries-*?v to find it.

Trade-offs and limitations

• Shrink prerequisites: the source index must be read-only, the cluster must be green, and all primary shards must be allocatable to a single node. Replicas are not a blocker—they are recreated on the shrunk index from its number_of_replicas setting.
• Disk space: shrink uses hard links for segment files, so it needs metadata and temporary overhead rather than a full second copy of the data.
• Temporary disruption: during shrink the index is blocked for writes, and search latency can rise while shards relocate. Schedule shrink for off-peak if the index still serves queries.
• Version compatibility: the shrink action has been part of ILM since the 6.x era, but exact action availability and defaults vary by version. Confirm against the documentation for the version you run before copying this policy verbatim—especially on 6.x or early 7.x clusters.

How to roll this out safely

1. Deploy the policy and template in a staging cluster with the same index patterns as production.
2. Lower max_size temporarily and bulk-load data to force a rollover and shrink within minutes.
3. Confirm each transition with GET <index>/_ilm/explain and verify the final shard count via _settings.
4. Promote to production with real thresholds, then keep an eye on _cat/indices?v and cluster health to confirm indices are rolling and shrinking on schedule.

With rollover and shrink automated, your time-series indices stay predictably sized, your shard count stops creeping up, and the 2 a.m. alias-flipping ritual goes away.