Stop Re-indexing: Using Kibana Runtime Fields for On-the-Fly Data Transformation

The Re-indexing Dilemma

You have a production Elasticsearch cluster with terabytes of data. Suddenly, a stakeholder asks for a new metric—perhaps the duration between a 'request_start' and 'request_end' timestamp—that wasn't calculated at ingest. Traditionally, this requires a three-step nightmare: updating your ingest pipeline, creating a new index, and running a massive _reindex operation that consumes cluster resources and takes hours or days to complete.

The alternative is to use Runtime Fields. These allow you to define a field using a Painless script directly within a Kibana Data View (formerly known as Index Patterns). Instead of storing the value on disk, Elasticsearch calculates the value at query time. This shifts the burden from storage and write-time processing to search-time CPU, enabling immediate data transformation without touching the underlying indices.

Defining Logic at the Data View Level

In Kibana, a Data View tells the interface which indices to query using a wildcard (e.g., logs-app-*). By adding a Runtime Field to this Data View, you create a virtual column that appears in the Discover tab and is available for all visualizations tied to that view.

Runtime fields use Painless, a secure, performant scripting language designed for Elasticsearch. Because these fields are defined in the Data View, they are "portable" across your Kibana dashboards without requiring changes to the Elasticsearch mapping. This is particularly useful for:

• Derived Metrics: Calculating ratios or differences between two existing numeric fields.
• String Manipulation: Concatenating first and last names or extracting a specific substring from a URL.
• Conditional Labeling: Using if/else logic to categorize numeric ranges into human-readable buckets (e.g., 0-100ms as "Fast").

Worked Example: Calculating Request Latency

Assume your indices contain request_start and request_end as date fields, but no pre-calculated latency field. Here is how to implement a runtime field to calculate the duration in milliseconds.

1. Navigate to Stack Management > Data Views.
2. Select your target Data View.
3. Click Add field.
4. Configure the field as follows:
   • Name: request_duration_ms
   • Type: Long
   • Script:

     emit(doc['request_end'].value.toInstant().toEpochMilli() - doc['request_start'].value.toInstant().toEpochMilli());

Execution Context: This script runs on the Elasticsearch data nodes. The emit() function is required to return the calculated value to the search results.

Verification: Go to the Discover tab. Search for your data and add the request_duration_ms field to the columns. If the field appears with calculated integers, the script is functioning. If the field is empty, verify that both source fields exist in the documents being queried.

The Performance Trade-off

Runtime fields are not a "free lunch." Because they are computed during the search phase, they are significantly slower than indexed fields.

Feature	Indexed Field	Runtime Field
Write Cost	High (CPU/Disk at ingest)	Zero
Read Cost	Low (Direct lookup)	High (Script execution)
Storage	Increases index size	No additional storage
Flexibility	Requires re-index to change	Instant updates to logic

Limitations: Complex scripts can spike CPU usage on your cluster, especially when performing aggregations over millions of documents. Furthermore, some specialized aggregations that rely on optimized BKD trees (used for numeric ranges) may perform poorly or be unsupported when using runtime fields.

When to Commit to an Indexed Field

Use Runtime Fields as a prototyping tool or for low-volume queries. If you find that a specific runtime field is being used in a high-traffic dashboard or is causing slow query responses (which you can verify by checking the Elasticsearch slow logs), it is time to move that logic into an ingest pipeline.

To transition, add the logic to your pipeline, use the _reindex API to populate the field for historical data, and then replace the Kibana Runtime Field with the newly indexed physical field. This gives you the agility of rapid iteration with the performance of indexed data.