Automate Elasticsearch log index rollover and retention with ILM

Logs are filling the cluster, and someone has to roll indices by hand

Your team indexes application logs into Elasticsearch. At first, one index per day works fine. Six months later, the cluster holds hundreds of indices. Some shards are tiny and waste heap on cluster-state metadata; others have grown so large that rebalancing takes hours. Disk usage keeps climbing because old indices are never deleted. You need a repeatable way to create new indices, keep primary shards near a target size, and retire old data on schedule.

The useful takeaway: Elasticsearch Index Lifecycle Management (ILM) can automate this. A rollover alias lets applications keep writing to one name while Elasticsearch creates a new backing index when the current one meets an age, document-count, or size condition. A delete phase can remove indices older than your retention window.

How rollover keeps shard size under control

ILM is Elasticsearch’s built-in index lifecycle manager. A rollover action creates a new write index when the current index satisfies a condition such as max_primary_shard_size, max_docs, or max_age. Applications write to an alias, not directly to an index. Elasticsearch routes writes to the current backing index and can search across every index that matches the alias.

Primary shard size matters because each shard consumes memory and CPU in the cluster state. A common target for time-series logs is roughly 20 to 50 gigabytes per primary shard, though the right number depends on your node count, query patterns, and retention policy. Rollover conditions use index-level statistics, so actual shard size can vary after segment merges and with replica shards.

Worked example: configure a rollover policy

Run these requests in Kibana Dev Tools or with curl against your cluster. You need the manage_ilm cluster privilege and index privileges to create aliases and indices. This assumes your cluster has ILM enabled.

1. Create the ILM policy

PUT _ilm/policy/nginx-logs-retention
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "30gb",
            "max_age": "7d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

This policy rolls the index when it reaches 30 gigabytes or seven days, whichever comes first, and deletes indices after 90 days. Adjust max_primary_shard_size to your cluster’s target shard size.

2. Create the alias and first backing index

PUT nginx-logs-write
{
  "aliases": {
    "nginx-logs-write": {
      "is_write_index": true
    }
  }
}

PUT nginx-logs-000001
{
  "aliases": {
    "nginx-logs-write": {}
  }
}

Applications should write to nginx-logs-write. The first backing index is nginx-logs-000001. ILM attaches to the index through the alias.

3. Enable ILM on the index

PUT nginx-logs-000001/_settings
{
  "index": {
    "lifecycle": {
      "name": "nginx-logs-retention",
      "rollover_alias": "nginx-logs-write"
    }
  }
}

4. Preview rollover without creating an index

POST nginx-logs-write/_rollover/nginx-logs-000001?dry_run=true

Use dry_run=true on a staging cluster first. The response tells you whether each condition would trigger without creating a new index. On production, run the same request without dry_run only when you are ready for Elasticsearch to create nginx-logs-000002 and move the write pointer.

5. Verify the result

GET _ilm/explain/nginx-logs-000001
GET _cat/indices?v

GET _ilm/explain shows the current phase, action, and step. GET _cat/indices shows store size, document count, and whether a new index appeared after rollover runs. Confirm that writes still succeed against the alias after rollover.

Limitations and what to check before production

• Rollover conditions are index-level, not exact per-shard disk usage. A single index with multiple primary shards can roll over before any individual shard reaches the target size.
• The delete phase removes data immediately. Validate retention rules on staging before applying them to production.
• ILM manages indices, not durable recovery. You still need snapshots for cluster restoration.
• Changing the write alias or naming pattern after launch requires coordinated alias swaps or reindexing. Otherwise, writes can fail against an index that no longer accepts them.

If you need to undo a rollover, delete the newly created backing index and point the alias back to the previous index. Any documents written to the new index after rollover must be reindexed manually, so treat rollover as a state-changing operation.

Closing: start small, verify, then automate

Pick one log stream, set a conservative shard-size target, and run the rollover in dry-run mode on staging. Watch _cat/indices and _ilm/explain for at least one rollover cycle before enabling delete on production. ILM removes the manual toil of index rotation, but only if you verify the retention and shard-size behavior against your actual traffic.