Automating Elasticsearch Index Lifecycle with ILM Rollover Shrink and Delete

The problem with manual index management

Elasticsearch indices naturally grow as new data arrives. Without a systematic approach, teams often find themselves manually merging, moving, or removing indices--tasks that become increasingly error-prone as cluster size and data volume scale. Maintaining cost efficiency while keeping recent data query-fast requires a lifecycle strategy that adapts to age size and access patterns.

ILM phases at a glance

• Hot: keeps shards online on fast nodes; ideal for recent frequently queried data.
• Warm: moves shards to less expensive nodes with lower I/O; reduces storage cost for aging data.
• Cold: can relocate shards to archive storage or freeze them entirely, making the segment read-only.
• Delete: permanently removes the index after a defined retention window.

Concrete example: ILM policy configuration

An ILM policy can be defined declaratively and applied to an index template. The policy below demonstrates a typical sequence: a hot phase that triggers rollover at 50gb, a warm phase that shrinks the index and forces a merge, and a delete phase that removes the index after 90 days. Adjust the size threshold and age values to match your write volume and retention requirements.

Apply the policy via the REST API: PUT _ilm/policy/my-ilm-policy with phases defining rollover shrink and delete.

Verifying the lifecycle in practice

Elasticsearch includes a diagnostic endpoint that reports the current phase action and any failures for each index:

The response includes the index's current phase (e.g. warm) the action being executed and timestamps for phase entry and action start-end.

Pair this with the index catalog to confirm shard placement:

Query the alias to ensure search traffic continues hitting the latest hot index without interruption:

If the shrink action ran verify the shard count decreased as expected:

Trade-offs and limitations to keep in mind

• Shrink prerequisites: an index can only be shrunk if it has a single primary shard and no replicas. Before the warm phase triggers shrink remove replicas and set number-of-shards to 1 in the index template or perform a manual clone to a single-shard index first.
• Delete is permanent: the delete phase removes all index data. If future recovery is required create a snapshot before the policy runs or configure the warm phase to clone the index to a preservation index instead of deleting it.
• Aggressive rollover thresholds: setting max-size too low triggers frequent index creation increasing cluster overhead and storage fragmentation. A practical approach is to size the threshold around your typical daily write volume such as 20-50gb per day depending on your workload.
• Custom metric phases: ILM can trigger actions based on CPU or memory only when the Monitoring plugin or X-Pack is enabled. In distributions without those features rely on age- and size-based phases.

Actionable next steps

1. Create the index template with your desired mappings and alias attaching an ILM policy name.
2. Define the ILM policy via the REST API starting with a hot rollover threshold that matches your write profile.
3. Index a sample dataset until the rollover condition is met, then use the _ilm/explain endpoint on the alias to confirm the phase transition.
4. If shard reduction is desired ensure the index meets shrink prerequisites before the warm phase triggers.
5. Schedule a snapshot before the delete phase runs or configure the policy to clone data to a preservation index first.