Using Elasticsearch Data Streams with ILM for Time‑Series Ingestion and Retention

Requirements

When ingesting high‑volume time‑series data (logs, metrics, traces) you need:

• A single write endpoint that hides the complexity of index naming and rollover.
• Automatic movement of data through storage tiers based on size or age.
• Retention policies that delete old data without manual index deletion.
• Visibility into phase transitions and the ability to alert on failures.

Smallest Suitable Design

The minimal building blocks that satisfy the above are:

1. Index template with data_stream.enabled:true that defines mappings and settings for the stream’s backing indices.
2. ILM policy that specifies phases (hot → warm → cold → frozen → delete) and actions (rollover, shrink, force‑merge, searchable‑snapshot mount, delete).
3. Data stream created with a name that matches the template; the stream automatically points to the current write index.

All other components (ingest pipelines, node roles, snapshot repository) are optional for the core flow but recommended for production.

Example Index Template

PUT _index_template/ts-metrics-template
{
  "index_patterns": ["ts-metrics-*"],
  "data_stream": {
    "enabled": true
  },
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "value":   { "type": "double" },
        "host":    { "type": "keyword" }
      }
    }
  },
  "priority": 500
}

Example ILM Policy

PUT _ilm/policy/ts-metrics-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "force_merge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "7d",
        "actions": {
          "allocate": {
            "include": {
              "_tier_preference": "data_cold"
            }
          }
        }
      },
      "frozen": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": {
            "repository": "ts-backup-repo",
            "snapshot_name": "snap-{now/d}"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Creating the Data Stream

PUT _data_stream/ts-metrics

After the stream exists, any index request to ts-metrics will be routed to the write index that matches the template and is currently in the hot phase.

Trust / Data Boundaries

• Ingest pipelines (if used) run on ingest nodes before documents reach the write index. They can mutate fields but cannot alter the stream name.
• Index templates and ILM policies are cluster‑level metadata. Only users with the manage_index_templates and manage_ilm privileges (typically admin or delegated roles) can create or modify them.
• Tenants should not share a data stream unless document‑level security (DLS) or field‑level security (FLS) is enforced via Elasticsearch Security. Without DLS/FLS, any user with write access to the stream can ingest into all backing indices.
• The snapshot repository used for the frozen tier must be accessible from all master‑eligible and data nodes; credentials are stored in the cluster’s secure settings.

Operational Checks

Routine verification helps catch problems before they affect ingestion or query latency.

• ILM status: GET _ilm/status shows which indices are managed by each policy and any error messages.
• Data stream stats: GET _data_stream/_stats provides rolling‑up counts of documents, storage size, and lag between the write index and the most recent rollover.
• Phase transitions: GET _cat/ilm?v lists each backing index, its current phase, and the time it entered that phase.
• Searchable snapshot health: GET _snapshot/ts-backup-repo/_status reports snapshot success/failure; frozen‑tier indices appear with STORE set to searchable_snapshot in GET _cat/indices?v.
• Disk watermarks: Monitor node.stats.fs or use Watcher alerts for the low (85%) and high (90%) watermarks to pre‑empt rollover stalls.

Failure Modes and Mitigations

• Rollover stall: Occurs when the write index cannot be created (e.g., mapping conflict, insufficient shards, or disk pressure). ILM logs show failed to create new write index. Mitigation: fix the mapping or template, ensure enough free space, then retry via POST _ilm/retry?policy=ts-metrics-policy.
• Snapshot repository unreachable: Causes frozen‑tier mount failures, making older data unsearchable. Verify repository credentials and network connectivity; test with POST _snapshot/ts-backup-repo/_verify.
• Frozen‑tier query latency: Searchable snapshots read from remote storage, so latency is higher than hot tier. Mitigate by enabling the frozen tier cache (indices.frozen.cache.enabled:true) and warming frequently accessed indices with a forced search.
• ILM policy changes not affecting existing indices: Updates apply only to newly created indices. To move existing indices immediately, use the ILM retry API or manually invoke POST _ilm/execute?policy=ts-metrics-policy on the target index.

When the Design Should Change

Re‑evaluate the architecture if any of the following conditions arise:

• Ingestion rate exceeds the capacity of a single write index shard (consistently > 1000 docs/sec per shard). Consider enabling routing or splitting the stream into multiple streams.
• Retention requirements demand different policies for subsets of the data (e.g., high‑resolution metrics vs. downsampled aggregates). In that case, create separate data streams with distinct ILM policies.
• Cost analysis shows that the frozen tier’s searchable‑snapshot overhead outweighs its storage savings; you may opt for a pure delete phase after cold instead of mounting snapshots.
• Security mandates require isolating write access per tenant; then you would move to index‑level security with separate streams or use alias‑based write routing with DLS.

Verification Steps (Non‑Exhaustive)

1. Deploy the template and policy as shown above on a test cluster.
2. Create the data stream (PUT _data_stream/ts-metrics).
3. Index a few thousand documents with a timestamp field; observe the write index name via GET _data_stream/ts-metrics.
4. Call POST _ilm/start (if not already auto‑started) and then poll GET _cat/ilm?v to see the index move from hot to warm after the rollover condition is met.
5. Simulate a disk‑pressure scenario by filling a data node to > 90 % usage; verify that ILM logs report a rollover failure and that alerts fire.
6. Free space and retry the policy (POST _ilm/retry?policy=ts-metrics-policy) to confirm recovery.
7. Mount a searchable snapshot manually (POST _snapshot/ts-backup-repo/snap-2026-10-20/_restore) and query the stream; compare latency against a hot‑tier query using GET _nodes/stats?metric=indices.search.

These steps give confidence that the design behaves as expected; they are not a substitute for production‑grade monitoring and testing.