Automating Elasticsearch Index Management with ILM: Rollover, Shrink, and Delete

Problem & Takeaway

Manually managing index rollover, shrinking, and deletion in a growing Elasticsearch cluster is error‑prone and resource‑heavy. Index Lifecycle Management (ILM) automates these tasks based on size or age, freeing operators and ensuring predictable data retention. This guide shows how to create a robust ILM policy, attach it via a template, verify the lifecycle, and recover if something goes wrong.

Prerequisites

• Elasticsearch 8.x cluster with at least one data node.
• Cluster‑wide privileges: cluster:monitor*, indices:admin*, indices:data/read* for the user running the commands.
• At least one snapshot repository configured for backup before deletion.
• Existing index naming convention that supports rollover (e.g., my-index-000001).

Desired Outcome

New indices automatically rollover after reaching 50 GB or 30 days, shrink to half the shard count in the warm phase, and delete after 90 days, all while keeping snapshots before deletion.

Step‑by‑Step Procedure

1. Define the ILM Policy

Create a JSON policy that outlines the hot, warm, and delete phases. Use the PUT /_ilm/policy API.

PUT /_ilm/policy/my-index-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_size": "50gb", "max_age": "30d" }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": { "number_of_shards": "1" },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Check the policy with GET /_ilm/policy/my-index-policy to confirm it appears exactly as defined.

2. Create an Index Template that Assigns the Policy

The template ensures every new index matching the pattern inherits the ILM policy.

PUT /_index_template/my-index-template
{
  "index_patterns": ["my-index-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "my-index-policy",
      "index.lifecycle.rollover_alias": "my-index"
    }
  }
}

Verify with GET /_index_template/my-index-template.

3. Create the Initial Data Index with an Alias

When creating the first index, reference the rollover alias so ILM can manage it.

PUT /my-index-000001
{
  "aliases": {
    "my-index": {
      "is_write_index": true
    }
  }
}

After creation, GET /my-index-000001/_ilm/explain should show the index in the hot phase.

4. Verify Rollover Availability

• Check the .ilm.rollover setting: GET /my-index-000001/_settings?include_defaults=true should include .ilm.rollover with the alias name.
• Use POST /my-index-/_rollover to manually trigger a rollover for testing.

5. Monitor ILM Progress

Query the explain endpoint regularly:

GET /my-index-*/_ilm/explain

The response contains phase, phase_time, and next_phase_time. Confirm the index moves from hot → warm → delete according to schedule.

6. Ensure Snapshots Before Delete

ILM does not snapshot automatically. Configure a snapshot lifecycle policy that runs before the delete phase, e.g., nightly snapshots of indices older than 60 days. Verify snapshots with GET /_snapshot/_all.

7. Recovery & Rollback Options

• Pause ILM on an index: POST /my-index-000001/_ilm/pause stops transitions. Resume with POST /my-index-000001/_ilm/unpause.
• Reapply ILM to an existing index: Use PUT /my-index-000001/_ilm/policy?policy=my-index-policy to assign the policy without reindexing.
• Undo a rollover: If a rollover happened incorrectly, delete the new rollover index and re‑apply the policy to the original. No automatic undo exists; manual intervention is required.

Expected Checks

• GET /_ilm/policy/my-index-policy – policy definition matches intent.
• GET /my-index-*/_ilm/explain – index phase and next transition times are correct.
• Snapshot API – ensure a snapshot exists for the index before its delete time.
• Cluster health – GET /_cluster/health should be green; ILM actions can fail if the cluster is not healthy.

Limitations & Caveats

• ILM actions require cluster‑wide permissions; missing rights cause failures.
• Delete phase is irreversible; double‑check the min_age before activation.
• Rollover size must consider shard allocation; large indices may cause unbalanced load.
• Snapshots are not handled by ILM; you must configure a separate snapshot lifecycle policy.

Conclusion

By combining a well‑structured ILM policy, an index template, and snapshot safeguards, you can automate rollover, shrink, and deletion of indices in Elasticsearch 8.x. Regularly monitor the explain endpoint and cluster health to catch misconfigurations early. When recovery is needed, pause ILM or reapply the policy to affected indices. This approach reduces manual effort, ensures predictable data retention, and keeps your cluster healthy.