Configure Elasticsearch Index Lifecycle Management for Automatic Rollover

Desired outcome

Automate the lifecycle of time‑series indices so that Elasticsearch creates a new write index when the current index exceeds a defined size (e.g., 50 GB) or age (e.g., 30 days). The old index moves to a warm phase for read‑only use and is later deleted after a retention period, eliminating manual rollover tasks.

Prerequisites

• Elasticsearch cluster version 6.6 or later (ILM introduced in 6.6).
• A user with the manage_ilm and monitor cluster privileges.
• An index template that matches the write alias pattern (e.g., logs-*).
• Sufficient disk space to hold at least one full shard during the rollover window.

Procedure

1. Define the ILM policy

   Use the _ilm/policy API to create a policy named logs_policy. The policy includes a hot phase that rolls over when the index reaches 50 GB or 30 days, a warm phase that forces a merge, and a delete phase that removes the index after 90 days.

   PUT _ilm/policy/logs_policy
   {
     "policy": {
       "phases": {
         "hot": {
           "actions": {
             "rollover": {
               "max_size": "50gb",
               "max_age": "30d"
             }
           }
         },
         "warm": {
           "actions": {
             "forcemerge": {
               "max_num_segments": 1
             }
           }
         },
         "delete": {
           "actions": {
             "delete": {
               "min_age": "90d"
             }
           }
         }
       }
     }
   }
   

   Run this command on any node in the cluster (e.g., via curl or Kibana Dev Tools). The user must have manage_ilm privilege.

2. Create an index template

   Attach the policy to a template that matches the write alias. The template also defines the number of shards and replicas.

   PUT _template/logs_template
   {
     "index_patterns": ["logs-*"],
     "settings": {
       "number_of_shards": 1,
       "number_of_replicas": 1,
       "index.lifecycle.name": "logs_policy",
       "index.lifecycle.rollover_alias": "logs_write"
     },
     "aliases": {
       "logs_write": {}
     }
   }
   

   The placeholder logs_write is the alias that applications will use for indexing. The user needs manage_index_templates privilege.

3. Bootstrap the initial index

   Create the first index that the alias will point to. The index name must end with a numeric suffix (e.g., 000001) so that ILM can increment it on rollover.

   PUT logs-000001
   {
     "aliases": {
       "logs_write": {
         "is_write_index": true
       }
     }
   }
   

   After this step, the alias logs_write points to logs-000001 and the index is under ILM management.

4. Verify policy application

   Check that the template is applied and the policy is attached.

   GET logs-000001/_ilm/explain
   

   The output should show the current phase as hot, the action as await_rollover, and the conditions that trigger rollover (size ≥ 50 GB or age ≥ 30 days).

Expected checks

• Policy definition: GET _ilm/policy/logs_policy confirms the phases, thresholds, and actions match the intended values.
• Current state: GET logs_write/_ilm/explain (using the alias) returns the phase, action, and time until the next step for the index behind the alias.
• Validate rollover: Optionally force a rollover to test the flow: POST logs_write/_rollover. Then GET logs_write/_alias should show the alias pointing to a newly created index (e.g., logs-000002) while the previous index (logs-000001) appears in the warm phase via its own _ilm/explain.

Recovery options

If a rollover fails (e.g., due to insufficient disk space), the ILM explain output will include an error message. You can:

• Inspect cluster health: GET _cluster/health?level=shards.
• Retry the policy for the affected index: POST logs-000001/_ilm/retry.
• Manually trigger a rollover: POST logs_write/_rollover after resolving the underlying issue (e.g., freeing disk space).
• As a last resort, restore the index from a recent snapshot: POST _snapshot/my_repo/snap-001/_restore with the appropriate index pattern.

Limitations and practical verification

ILM does not shrink the number of shards automatically; if you need fewer shards in the warm phase, add a shrink action in the warm phase and ensure the source index meets the shrink requirements (one shard per node, same settings). Overly aggressive rollover thresholds (very small max_size or short max_age) can create many small shards, increasing cluster overhead and potentially degrading query performance. To verify that the policy is not causing excessive shards, periodically run GET _cat/indices/logs-*?v&s=store.size:desc and observe the shard count and size distribution.

All commands shown assume you have access to the Elasticsearch REST API (e.g., via curl -u username:password or Kibana Dev Tools) and that the user possesses the required privileges. Replace placeholder names (logs_policy, logs_template, logs_write) with values that match your naming conventions.