Managing Time‑Series Data with Elasticsearch ILM Rollover: A Practical Guide

Why Rollover Matters for Time‑Series Workloads

When you ingest logs, metrics, or any continuously growing data stream into Elasticsearch, you quickly hit a problem: a single index can become huge, shards grow beyond the size that the cluster can manage efficiently, and queries slow down. Traditionally, operators would schedule cron jobs to move data out or delete old indices, but that approach is brittle and hard to maintain.

Elasticsearch’s Index Lifecycle Management (ILM) solves this by automating the lifecycle of an index. The rollover action is the core of this automation: it lets an alias point to the current write index and automatically swaps that index for a new one when you hit a threshold such as age, document count, or primary shard size.

TL;DR – The Rollover Workflow

1. Create an index template that defines settings, mappings, and the is_write_index flag.
2. Define a composable ILM policy that includes a hot phase with a rollover action.
3. Bootstrap the first index with the alias marked as the write index.
4. Index data; once the rollover condition is met, Elasticsearch automatically creates a new index and switches the alias.
5. Use the warm and cold phases to shrink shards and optimize search performance.

Step‑by‑Step Example (Elasticsearch 8.10)

Below is a minimal, reproducible example that you can run on a single‑node cluster. Replace <your_cluster> with your own cluster name if you’re not using the default.

1. Define an Index Template

PUT /_template/logs_template
{
  "index_patterns": ["logs-*"] ,
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "index.lifecycle.name": "logs_policy"
  },
  "aliases": {
    "logs_write": {"is_write_index": true}
  }
}

Explanation:

• index_patterns matches any index that starts with logs-.
• The index.lifecycle.name setting attaches the ILM policy we’ll create next.
• The alias logs_write is marked as the write index; Elasticsearch will use it for indexing.

2. Create an ILM Policy with Rollover

PUT /_ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_docs": 5,
            "max_primary_shard_size": "50mb"
          },
          "set_priority": {"priority": 100}
        }
      },
      "warm": {
        "actions": {
          "shrink": {"number_of_shards": 1},
          "forcemerge": {"max_num_segments": 1},
          "set_priority": {"priority": 50}
        }
      },
      "cold": {
        "actions": {
          "freeze": {},
          "set_priority": {"priority": 0}
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {"delete": {}}
      }
    }
  }
}

Key points:

• The rollover action triggers when any of the thresholds are reached.
• In production you’d likely set max_age to a day or a few hours and max_docs to a high number; the example uses small values for demonstration.
• After rollover, the index enters the warm phase where we shrink and force‑merge to reduce segment overhead.

3. Bootstrap the First Index

PUT /logs-000001
{
  "aliases": {
    "logs_write": {"is_write_index": true}
  }
}

Now the alias logs_write points to logs-000001. All subsequent indexing will go to this index.

4. Index Some Documents

POST /logs_write/_doc
{ "message": "First log" }
POST /logs_write/_doc
{ "message": "Second log" }
POST /logs_write/_doc
{ "message": "Third log" }
POST /logs_write/_doc
{ "message": "Fourth log" }
POST /logs_write/_doc
{ "message": "Fifth log" }
POST /logs_write/_doc
{ "message": "Sixth log" }

After the sixth document, the max_docs threshold of 5 is exceeded. The cluster will create a new index logs-000002 and move the alias to point to it. The original index will now be read‑only and ready for the warm phase.

5. Verify the Rollover

GET /_ilm/status
GET /logs-000001/_ilm/explain
GET /logs-000002/_ilm/explain

These calls show which phase each index is in. You should see logs-000001 in the warm phase and logs-000002 in hot. The alias logs_write should point to logs-000002.

Trade‑Offs and Limitations

• Retention Granularity: ILM deletes entire indices, not individual documents. If you need per‑document TTL, you must design your rollover cadence accordingly.
• Overshoot: Rollover conditions are evaluated periodically (every 30 seconds by default). A document that pushes the index just over the threshold may still be indexed into the old index before rollover occurs.
• Version Sensitivity: The exact field names and available actions differ between 7.x and 8.x. Always check the docs for your version before copying examples.
• Existing Indices: ILM does not retroactively apply to indices created before the policy is attached. If you need to bring legacy data under ILM, you’ll have to reindex.

When to Use ILM Rollover

• High‑volume logs: Keeps shards at a manageable size and automates aging.
• Metrics and time‑series data: Ideal for data that naturally partitions by time.
• Cost‑conscious environments: The warm and cold phases can move data to cheaper storage while still keeping it searchable.

Actionable Checklist

1. Decide on your retention policy: how long do you need the data in hot, warm, and cold phases?
2. Choose rollover thresholds that match your shard size and query patterns.
3. Create a reusable index template that includes the ILM policy name.
4. Bootstrap the first index with the is_write_index alias.
5. Monitor ILM status with GET _ilm/status and GET <index>/_ilm/explain to ensure indices move through phases as expected.
6. Adjust shrink and forcemerge parameters in the warm phase to balance search speed against storage usage.

By following these steps, you replace ad‑hoc cron jobs with a declarative, versioned lifecycle that scales automatically and reduces operational overhead.