Automating Time‑Series Index Management with Elasticsearch ILM Rollover and Shrink

Problem: Growing time‑series indices become costly and slow

When you ingest logs, metrics, or sensor data into Elasticsearch, each day (or hour) creates a new index. Over time the number of shards grows, heap pressure rises, and query latency suffers. Manually rolling over indices and shrinking them is error‑prone and consumes operational overhead.

Thesis: A simple ILM policy can automate rollover and shrink, keeping shard counts low while preserving queryability.

How ILM fits the time‑series pattern

Index Lifecycle Management (ILM) lets you define phases—hot, warm, cold, and delete—and attach actions to each phase. For time‑series data the most useful actions are:

• rollover: creates a new write index when the current index exceeds a size or age threshold.
• shrink: after rollover, reduces the primary shard count of the old index (e.g., from 3 to 1) while keeping it searchable.
• forcemerge: optionally merges segments to improve search performance on older, read‑only indices.

By attaching the policy to an index template, every new index that matches the template inherits the lifecycle automatically.

Configuring the policy and template

First, define an ILM policy that rolls over at 50 GB or one day, then shrinks to a single primary shard and moves to warm after three days.

PUT _ilm/policy/time-series-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Next, create an index template that applies the policy to indices matching logs-*. The template starts with an alias (logs-write) that ILM will use for rollover.

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.lifecycle.name": "time-series-policy",
      "index.lifecycle.rollover_alias": "logs-write"
    }
  },
  "aliases": {
    "logs-write": {}
  }
}

Initialize the first write index manually (required before ILM can take over):

PUT logs-000001
{
  "aliases": {
    "logs-write": { "is_write_index": true }
  }
}

Worked example: watching the lifecycle in action

Assume you have a test cluster where you can run the following commands with cluster‑admin privileges.

1. Index a few documents to generate traffic:

POST logs-write/_doc
{
  "timestamp": "2026-10-13T04:00:00Z",
  "message": "test event"
}

1. Check the current phase of the write index:

GET _ilm/explain?index=logs-000001

You should see phase": "hot" and action": "rollover" pending.

1. Force a rollover based on size (for testing, lower the threshold temporarily):

POST logs-write/_rollover
{
  "conditions": {
    "max_size": "1mb"
  }
}

If the condition is met, Elasticsearch responds with a new index name, e.g., logs-000002, and the alias logs-write now points to it as the write index.

1. Verify that the old index entered the warm phase and began shrinking:

GET _ilm/explain?index=logs-000001

Look for phase": "warm" and an action shrink that has completed. After shrink finishes, you can confirm the shard count:

GET logs-000001/_stats

The primaries count should now be 1 (down from the original 3).

Trade‑off: shrink requires a read‑only, single‑shard‑per‑node index

The shrink action can only run on an index that is:

• Marked index.blocks.write: true (read‑only).
• Has exactly one shard per node in the cluster (so if you have 3 nodes, the index must have 3 shards before shrinking to 1).

If you attempt to shrink a busy index that still receives writes, the cluster may turn red because the shrink operation tries to relocate shards while indexing continues. ILM avoids this by only triggering shrink after the rollover has made the old index read‑only and after the warm phase’s min_age condition is satisfied.

Practical way to check the result: after the shrink action completes, run GET /_stats and verify primaries equals the target shard count. Also monitor cluster health with GET _cluster/health?level=shards to ensure no red or yellow states appear during the transition.

Next steps

Start with a modest policy (e.g., rollover at 10 GB, shrink after one day) and observe the _ilm/explain output. Adjust thresholds based on your indexing rate and storage costs. Keep an eye on the ILM explain API and cluster health to catch any delays caused by heavy ingest loads.

By letting ILM handle rollover and shrink, you reduce manual ops, keep shard counts predictable, and maintain query performance for your time‑series data.