Elasticsearch Index Rollover: An ILM Architecture Note for Time-Series Data

The Problem: One Index That Never Retires

A time-series index that receives continuous writes—application logs, metrics, or events—does not age gracefully. As the index grows, its primary shards (the main copies of index data that handle writes and searches) become larger, segment merges cost more I/O, and retention by index deletion becomes coarse and disruptive. The useful takeaway is to roll the index before shards oversize, not after: attach an Index Lifecycle Management (ILM) rollover policy to a write alias, target primary shards in the roughly 20 GB to 40 GB range, and verify rollover state with the ILM explain API instead of inferring it from cluster health.

Requirements

• A write workload that appends time-series documents continuously.
• A retention requirement: delete data after a set age, or move it to a colder tier.
• Elasticsearch 7.x or 8.x. Data streams automate rollover in these versions; manual indices with ILM remain supported.
• Cluster permissions to create and manage ILM policies and aliases.

Smallest Suitable Design

Start with one alias, one initial index, and one ILM policy.

Alias and initial index

Create an alias that clients use for writes. The alias must designate a write index, and the index name must end with a numeric suffix that ILM can increment.

PUT /events-000001
{
  "aliases": {
    "events-write": {
      "is_write_index": true
    }
  }
}

Run this in Kibana Dev Tools or via curl with a user holding index write and alias management permissions. The placeholder events-000001 is the first index; ILM will later create events-000002, events-000003, and so on.

ILM policy

Define a policy that rolls over when the primary shard size or age reaches a threshold, then deletes the index after retention expires.

PUT /_ilm/policy/events-rollover
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "30gb",
            "max_age": "7d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Attach the policy to the index template or directly to the first index so ILM manages it from creation.

Trust and Data Boundaries

• Clients write to the alias, not the index. Applications send requests to events-write. ILM owns the alias and updates which physical index receives writes after rollover.
• Do not manually repoint the alias. Removing the write index or pointing the alias at a different index outside ILM breaks rollover and retention. Change behavior through the ILM API.
• ILM runs on master-eligible nodes; actions execute on data nodes. The cluster state records the policy assignment, but node roles and license level determine whether actions such as freeze or cold-tier migration are available.

Operational Checks

Use these calls to confirm the design is behaving as intended.

Confirm alias and write index

GET /events-write/_alias

Expected: the response lists one index with "is_write_index": true. Required permission: read alias metadata.

Check policy attachment

GET /events-000001/_settings?pretty

Expected: index.lifecycle.name equals events-rollover.

Test rollover without creating an index

POST /events-write/_rollover?dry_run=true
{
  "conditions": {
    "max_primary_shard_size": "30gb",
    "max_age": "7d"
  }
}

Run with rollover execution permissions. The response reports whether conditions are met but does not create a new index. A dry_run result of rollover: false means the threshold has not been reached or ILM has not evaluated it yet.

Inspect lifecycle state

GET /events-000001/_ilm/explain

Expected: phase, action, and step fields show where the index sits in the policy. An error field indicates a stuck lifecycle, often caused by an unmet condition or a missing node role for a required action.

Verify shard sizes after rollover

GET /_cat/indices?v

Expected: primary shard store sizes remain near the target range, and index count does not grow unbounded. If indices are far smaller than 20 GB, rollover is too aggressive; if shards exceed 40–50 GB, raise rollover frequency or lower the size threshold.

Failure Modes

Symptom	Likely cause	Check
Rollover threshold crossed but no new index	ILM evaluates on a schedule; threshold crossing is not instant	GET /<index>/_ilm/explain and wait for the next ILM cycle
Lifecycle step stuck with an error	Missing node role, license restriction, or unmet condition	_ilm/explain error detail; review node roles and license
Alias no longer has a write index	Manual alias modification outside ILM	GET /<alias>/_alias; restore alias through ILM, not direct API edits
Too many small indices	Rollover conditions too aggressive (for example, max_age too short)	_cat/indices; increase max_age or max_primary_shard_size
Oversized shards after rollover	Size threshold too high or indexing burst outpaces evaluation	_cat/indices; lower max_primary_shard_size toward 30 GB

Conditions That Change the Design

• Use data streams instead if you are on Elasticsearch 7.x or 8.x and want logs, metrics, or traces managed with less manual index handling. Data streams hide backing indices and automate rollover.
• Add warm, cold, or frozen tiers only if your license and node roles support those ILM actions. The basic free distribution does not include every ILM action.
• Shorten rollover age if writes burst and a 7-day index produces shards larger than 40 GB before ILM evaluates.
• Do not rely on ILM for exact deletion time. Delete actions run on the ILM schedule, so data may remain briefly past the retention date.

Limitations and Verification

Rollover is eventually consistent with threshold crossing. A manual POST /<alias>/_rollover forces evaluation, but automatic rollover waits for the ILM interval. Force merge or shrink after rollover temporarily doubles disk usage and can trigger cluster rebalancing; schedule those operations during low traffic. After any rollover, run GET /_cat/indices?v and confirm shard counts and store sizes stay within the 20 GB to 40 GB target range before treating the design as stable.