Designing Elasticsearch ILM with Rollover and Hot/Warm/Cold Tiers

The problem: time-series indices grow forever

Log and metrics indices accumulate until disks fill or query latency degrades. Manual index rotation is error-prone. Index Lifecycle Management (ILM) automates rollover and moves indices through hot, warm, and cold tiers based on age or size. This note covers the smallest design that works, where data and trust boundaries sit, and what breaks it.

Requirements

ILM is built into Elasticsearch 7.x and 8.x. You need at least one data tier (hot) and a rollover alias. The design assumes a 3-node cluster with dedicated node roles, but the same pattern scales to larger clusters.

Minimal design

A single ILM policy, one write alias, and node roles are sufficient. The policy defines rollover at 50 GB or 30 days, then moves indices to warm after 7 days, cold after 30 days, and deletes after 90 days. The alias points to the current write index. Node roles (hot, warm, cold) enforce tier separation via shard allocation awareness.

ILM policy and alias

Create the policy with a rollover alias and phase transitions. The alias is the write alias; all writes go through it. The policy must be attached to the alias, not the index directly.

PUT _ilm/policy/timeseries_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "allocate": {
            "require": {
              "data": "warm"
            }
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "require": {
              "data": "cold"
            }
          }
        }
      },
      "delete": {
        "min_age": "90d"
      }
    }
  }
}

Attach the policy to a write alias. The alias must be the write alias for the index. The policy applies to all indices using the alias. Test the policy in a non-production cluster first.

Trust and data boundaries

Hot tier holds recent data for fast queries. Warm tier retains searchable but less frequently accessed data. Cold tier stores archival data with reduced I/O. Snapshots to external storage (S3, NFS) protect against data loss. Cold tier nodes must have sufficient IOPS for search latency; otherwise, query performance will degrade, especially for time-series data.

Operational checks

Monitor ILM phase transitions via the _ilm/explain API. Validate shard allocation via cluster health. Verify snapshot success rates. Run the checks after each phase transition. Use the Cluster API to confirm shard allocation to the correct tier and verify that indices in warm and cold tiers are read-only after transition. Run snapshot jobs to external S3 or NFS and confirm that cold tier indices are removed after snapshot completion.

Failure modes

Network partitions can delay phase transitions. Insufficient disk on cold tier may trigger index deletions. Misconfigured retention policy could cause data loss. Ensure watchdog alerts and fallback to manual rollover if ILM stalls. Rollover requires a write alias; accidental writes to the primary index bypassing the alias can break rollover logic and create orphaned indices. Cold tier nodes must have sufficient IOPS for search latency; otherwise, query performance will degrade, especially for time-series data.

When to change the design

Change the design if you need to enforce strict data residency (add a dedicated cold tier with allocation awareness), if you have compliance requirements for data retention (add a frozen tier with searchable snapshots), or if you need to support high-cardinality time-series data (add a dedicated machine learning node). The minimal design assumes a 3-node cluster with dedicated node roles. The same pattern scales to larger clusters.