Runtime Fields in Kibana Data Views: Quick Prototyping vs. Full Reindexing

Why Runtime Fields Matter

When you need a derived value—say, turning an HTTP status code into a bucket like "2xx" or "5xx"—you usually think of adding a new field to your mapping. That means re‑indexing all existing documents, which is costly. Kibana’s runtime fields let you compute the value on the fly with a short Painless script, so you can iterate instantly without touching the index.

The Core Decision: Reindex vs. Runtime

Choose reindexing when:

• The derived field is queried heavily (e.g., in alerts, retention, or external clients).
• Performance is critical; indexed fields are far faster than script evaluation.
• You need the field to be visible to all consumers, not just Kibana.

Choose runtime fields when:

• You want to prototype a transformation quickly.
• The field is used only in Kibana visualizations or dashboards.
• Data volume is moderate or you can tolerate the per‑document script cost.

Defining a Runtime Field in a Data View

In Kibana 8.x/9.x, data views replace the old index patterns. The UI lets you add a runtime field that lives only in that view.

1. Open Stack Management > Data Views and select your view.
2. Click Add runtime field.
3. Give the field a name (e.g., status_class).
4. Select the type that matches the output of your script. For a bucket string, choose keyword.
5. Enter a Painless script in the editor. For example:

if (doc.containsKey('response.status')) {
  int code = doc['response.status'].value;
  if (code >= 200 && code < 300) {
    emit('2xx');
  } else if (code >= 400 && code < 500) {
    emit('4xx');
  } else if (code >= 500) {
    emit('5xx');
  } else {
    emit('other');
  }
} else {
  emit('unknown');
}

Click Save. The field now appears in Discover, Lens, and any Kibana visualization that can use keyword fields.

Testing the Runtime Field

Verify that the field works and measure its cost.

1. Open Discover and add a filter: status_class: "5xx".
2. Open the Inspect panel and look at the took value. Compare it to a query that filters on the raw response.status field.
3. In Dev Tools, run:

GET //_search
{
  "size": 0,
  "query": {
    "term": { "status_class": "5xx" }
  }
}

Check the profile section (add "profile": true to the body) to see how much time the script spends per document.

When to Promote to a Mapped Field

If the runtime field becomes a core part of your data pipeline—used in alerts, external dashboards, or compliance logs—you should consider moving it to a real indexed field.

1. Create a new field in the index mapping (e.g., status_class as keyword).
2. Write a reindex job that copies the existing documents and sets the new field using the same script logic.
3. Once reindexing completes, remove the runtime field from the data view.

Afterward, queries on status_class will be as fast as any other keyword field and visible to all clients.

Governance & Visibility Caveats

• A runtime field defined only in a data view is invisible to external Elasticsearch clients. If you need the field in Logstash, Beats, or alerting rules outside Kibana, define it at the mapping level.
• Runtime fields do not support all aggregation types in every visualization. Test the specific feature (e.g., a Lens histogram) before relying on it.
• Older Kibana versions (pre‑7.15) use scripted fields, which have a different syntax. Ensure you’re using the runtime field editor.

Practical Checklist

Step	What to Check
Define Field	Type matches output; script emits correctly.
Test Query	Inspect panel latency; profile API shows CPU cost.
Validate Visibility	Run GET //_mapping to confirm runtime field is in the data view only.
Promote if Needed	Reindex with script; remove runtime field.

Takeaway

Runtime fields give you a zero‑migration, rapid‑iteration path to add derived data in Kibana. Use them for prototyping, dashboards, and ad‑hoc analysis. When the field becomes a core, heavy‑use component, migrate it into the index mapping via reindexing to unlock full performance and cross‑tool visibility.