Ship Beats to Elasticsearch with setup.ilm Instead of Daily Indices

If you are still creating daily indices by hand—setting output.elasticsearch.index to something like logs-app-2026.10.19 and relying on a cron job or curator to delete old data—you are doing work the Beat can do itself. The useful takeaway: enable setup.ilm.enabled, pin your rollover alias and policy names, and let the Beat setup process install the lifecycle policy, index template, and rollover alias. Then write to the alias, not a date-stamped index.

How setup.ilm Automates the Index Lifecycle

Index Lifecycle Management (ILM) is an Elasticsearch feature that automates index rollover, retention, and tiering based on policies you define. When Filebeat or Metricbeat starts with setup enabled, or when you run filebeat setup manually, the Beat reads the policy file you specify and performs three actions in Elasticsearch:

1. It creates or updates the ILM policy named by setup.ilm.policy_name.
2. It installs an index template whose index.lifecycle.name setting references that policy and whose alias entry points to your setup.ilm.rollover_alias. The initial backing index is marked is_write_index: true.
3. It ensures the rollover alias exists and is writable.

The Elasticsearch output then sends bulk write requests to that alias. When the policy conditions—max_size, max_age, or max_docs—are met, Elasticsearch creates a new backing index, flips the write index, and the alias continues to point at the current target. Existing indices are not renamed or reindexed; only the write pointer moves.

Worked Filebeat Configuration

The following filebeat.yml shows the minimum settings to ship logs through ILM. Run the setup step with a user that has manage_ilm, manage_index_templates, and manage privileges on the alias and matching indices, or setup will fail.

# /etc/filebeat/filebeat.yml
setup.ilm.enabled: true
setup.ilm.rollover_alias: "logs-app"
setup.ilm.pattern: "logs-app-*"
setup.ilm.policy_name: "logs-app-default"
setup.ilm.policy_file: /etc/filebeat/ilm.json

output.elasticsearch:
  hosts: ["https://es01.example.com:9200"]
  # Do NOT set output.elasticsearch.index when ILM is enabled

The referenced /etc/filebeat/ilm.json contains a standard Elasticsearch ILM policy. A minimal hot-and-delete policy looks like this:

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Keep output.elasticsearch.index unset. If you set it explicitly, Beats writes to that concrete index name and bypasses the rollover alias entirely.

Verifying the Policy, Template, and Write Alias

Run these checks in Kibana Dev Tools or with curl against your cluster. Replace logs-app, logs-app-default, and the template name with your actual values.

• Policy exists: GET _ilm/policy/logs-app-default. Confirm the phases and rollover conditions match your policy_file.
• Template binds the alias: GET _index_template/<template_name> (or legacy GET _template/<name>). Look for index.lifecycle.name set to your policy and an aliases entry for logs-app.
• Alias points to a write index: GET _alias/logs-app. The response should list backing indices such as logs-app-000001, with one marked is_write_index: true.
• Rollover works: POST logs-app-000001/_rollover forces a rollover if you need to test immediately. Then GET _ilm/explain/logs-app-000001 shows the current phase and step.
• Events land correctly: Send a test event and search via the alias: GET logs-app/_search. The hit should appear in the newest write index, not a date-stamped name.

Also check the Beat logs and publisher metrics for setup or ILM errors, and confirm the events published count is not dropping after you enable ILM.

Limits, Licensing, and Version Differences

ILM requires an Elasticsearch cluster with index-lifecycle support. The hot and delete phases are broadly available, but warm, cold, frozen, and searchable snapshot actions depend on your subscription and node tiers. Do not assume those phases are available unless you have verified cluster and license support.

Changing setup.ilm.policy_name or rollover conditions does not retroactively move indices that were already written. Attach the new policy to existing indices separately with the Elasticsearch update settings API.

Names, defaults, and whether ILM is on by default vary across Beats 7.x, 8.x, and 9.x and between different Beats. Confirm the exact setting names and template names against the reference documentation for your versions before deploying.

Setup behavior is also controlled by setup.ilm.check_exists and setup.ilm.overwrite. Leaving check_exists enabled can skip updates when a newer Beat version ships a changed policy or template, so plan a deliberate overwrite during upgrades if you need those changes.

Common Mistakes That Break ILM

• Setting an explicit index: Configuring output.elasticsearch.index while ILM is enabled sends events to a concrete index, not the alias. Bulk writes then fail or bypass lifecycle management.
• Leaving setup.ilm.enabled false: The other setup.ilm.* settings are ignored unless this is true.
• Alias and pattern drift: setup.ilm.rollover_alias must match the alias in the installed template, and setup.ilm.pattern must match the backing indices. If they drift, writes fail or the alias points at the wrong indices.
• Multiple Beat versions on one alias: Running different Beat versions against the same alias without versioned patterns causes index template conflicts.
• Assuming rollover reindexes: Rollover only moves the write index forward. It does not rename, merge, or reindex already-written data.