Automating Time-Series Schema with Elasticsearch Index Templates

The Problem: Schema Drift in Rotated Indices

When managing time-series data—such as logs, metrics, or traces—it is common practice to rotate indices daily or weekly (e.g., logs-2026-09-21). Without a centralized configuration, each new index is created with default settings. This leads to "schema drift," where a field might be indexed as a text type in one index and a keyword in another, breaking aggregations and search queries across the dataset.

The solution is the Index Template. This feature allows you to define a blueprint of settings and mappings that Elasticsearch automatically applies to any new index matching a specific name pattern.

Prerequisites

• An Elasticsearch cluster (version 7.8+ recommended for Composable Index Templates).
• API access via Kibana Dev Tools or curl.
• Administrative permissions to manage cluster state (manage_index_templates privilege).

Implementing a Composable Index Template

Modern Elasticsearch uses "composable" templates, which separate the index pattern from the actual settings. This allows you to reuse common configurations across different types of data.

Step 1: Define a Component Template

Component templates are modular building blocks. For example, if all your time-series indices require a specific shard count and a @timestamp field, define that here first.

PUT _component_template/time-series-base
{
  "template": {
    "settings": {
      "index.number_of_shards": 3,
      "index.number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" }
      }
    }
  }
}

Step 2: Create the Index Template

The index template links the component template to a specific naming pattern. Use the priority field to ensure this template overrides more generic ones; higher numbers take precedence.

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "priority": 100,
  "template": {
    "composed_of": ["time-series-base"],
    "mappings": {
      "properties": {
        "log_level": { "type": "keyword" },
        "message": { "type": "text" }
      }
    }
  }
}

Verification and Testing

Templates are not retroactive. To verify the configuration, you must create a new index that matches the pattern.

1. Create a test index

Run this command to trigger the template application:

PUT logs-test-001

2. Check the applied mappings

Verify that both the component template (@timestamp) and the index template (log_level) mappings are present:

GET logs-test-001/_mapping

3. Check the index settings

Confirm the shard and replica counts match the component template:

GET logs-test-001/_settings

Comparison: Index Templates vs. Dynamic Mapping

Critical Limitations and Risks

• Mapping Explosions: If your template allows too many dynamic fields, you may hit the index.mapping.total_fields.limit (default 1000), causing the cluster to reject new fields.
• Non-Retroactivity: Changing a template does not update indices that already exist. To apply changes to old data, you must use the _reindex API.
• Priority Conflicts: If two templates match logs-* with the same priority, the resulting configuration is a merge of both, which can lead to unpredictable field types.

Rollback Procedure

If a template causes incorrect mapping or performance degradation, remove it to prevent new indices from inheriting the flawed configuration:

DELETE _index_template/logs-template

Note: This will not fix indices already created with the template. You must delete those indices or reindex the data into a new index with the corrected schema.