Kibana Runtime Fields: Fix Mapping Mistakes Without Reindexing

You open Kibana to build a chart, but Lens will not let you aggregate a field because Elasticsearch mapped it as text. The usual fix is a reindex: create a new index with the correct mapping, copy the data, swap an alias, and delete the old index. That is manageable for gigabytes. For terabytes of historical logs it is expensive, slow, and often requires a change window.

Kibana runtime fields offer a narrower alternative. A runtime field is a field defined inside a Kibana data view rather than in the Elasticsearch index mapping. Elasticsearch evaluates the field’s Painless script at query time for each matching document. You avoid touching the index, but you move the cost from ingestion to every query that uses the field.

Runtime fields became generally available in Elasticsearch 7.11. Kibana menu paths and scripting options differ across 7.x, 8.x, and 9.x releases, so check your release notes before documenting a team standard.

Worked example: rescuing a numeric string

Suppose an ingest pipeline wrote user.age as a string, so Elasticsearch dynamically mapped it as text with a keyword subfield. You cannot sort or aggregate it numerically, and reindexing is not an option this quarter.

In a non-production Kibana space, open the data view editor for the affected index pattern. You need the Kibana privilege to manage data views in that space. Add a runtime field with these values:

• Name: user_age_long
• Type: number (long)
• Script: a Painless expression that parses the string

def value = doc['user.age.keyword'].value;
if (value != null) {
  try {
    emit(Long.parseLong(value));
  } catch (Exception e) {
    emit(null);
  }
}

Replace user.age.keyword with the actual field path in your index. Save the data view. In Lens, user_age_long now behaves like a numeric field for charts and filters.

Two caveats apply. First, the script assumes user.age.keyword has doc values, which is the default for keyword fields. If the original mapping lacks a keyword subfield, the script must read from _source instead, which is slower. Second, every unhandled exception fails the query, so the try/catch guard above prevents one bad record from breaking the whole dashboard. Guards add CPU overhead.

The trade-off: query latency, CPU, and portability

An indexed field is computed once at ingest. A runtime field is computed on every query, for every document that matches the query filter. Five analysts opening a dashboard once a day is a different load than a wall display refreshing every thirty seconds or an application used by thousands.

The deeper problem is aggregation. Indexed fields use an inverted index or doc values that let Elasticsearch skip large portions of data. Runtime fields do not. A terms aggregation on a high-cardinality runtime field—such as a user ID or session token—forces Elasticsearch to evaluate the script for every candidate document and build the result from scratch. That can be dramatically slower than the indexed equivalent.

Painless scripts also consume CPU. If many users open the same dashboard simultaneously, cumulative script evaluation can pressure the cluster and increase search thread pool rejections. This is the most common reason teams abandon runtime fields after a proof of concept.

Operationally, a runtime field lives in the Kibana data view saved object, not in cluster state. Exporting the data view from one environment and importing it into another carries the definition with it, but only if you include the data view itself in the export. Other Kibana spaces and direct Elasticsearch clients do not see the field unless they use the same data view.

Verify before you adopt

Do not adopt a runtime field on faith. Profile it in a non-production data view that mirrors production volume.

1. Create the runtime field in the non-production data view.
2. Run a representative query, such as a terms aggregation on the runtime field, in Discover or Lens.
3. Open the same query in the Elasticsearch Search Profiler and compare the collect phase timing against an equivalent indexed field.
4. Watch cluster CPU and search thread pool rejections during the test.

If the profiled query is noticeably slower, or CPU climbs under concurrent load, treat the runtime field as a temporary patch, not a permanent mapping strategy.

Saving a data view changes state, so rollback is manual but safe. Edit the data view and remove the runtime field, or restore a previous version of the saved object from Saved Objects management. No Elasticsearch index is modified, so there is no reindex to undo.

The actionable rule

Use runtime fields when the alternative is an urgent reindex, the data volume is moderate, and the field will power exploratory dashboards for a small audience. Profile the query, guard your scripts against malformed data, and set a review date. If the field graduates to a hot path—used in frequent alerts, anomaly detection jobs, or high-traffic aggregations—the correct long-term fix is still a reindex into a properly mapped index. Runtime fields buy time; they do not eliminate mapping debt.