Kibana Scripted Fields for On‑the‑Fly Calculations

Problem: Needing a Derived Metric Without Re‑indexing

Imagine you have an index of sales documents that contains 'price' and 'cost' fields. You want to visualize profit margin in a Lens chart, but re-indexing to add a pre-computed 'profit_margin' field would require downtime and extra storage. You need a way to calculate the value at query time.

How Scripted Fields Work in Kibana

Kibana lets you define a 'scripted field' on an index pattern. The field is written in Elasticsearch’s Painless language and is evaluated for every document that matches a query. The result is not stored in the index; it exists only while the search runs, so you can use it in visualizations, lenses, or saved objects just like any other field.

Worked Example: Adding a Profit Margin Scripted Field

1. Open Kibana → Management → Index Patterns and select the pattern that matches your sales index (e.g., 'sales-*').
2. Click the 'Scripted fields' tab and then 'Add scripted field'.
3. Give it a name like 'profit_margin' and set the type to 'Number'.
4. In the 'Painless script' box enter:

   if (doc['cost'].value != 0) {
       (doc['price'].value - doc['cost'].value) / doc['cost'].value;
   } else {
       0
   }

5. Click 'Save field'. The field now appears in the field list.
6. Open Lens, add a new chart, and drag 'profit_margin' onto the Y‑axis. You should see the calculated margin for each document.

Performance Trade‑offs and Limitations

• Query‑time cost: The script runs for every document that matches the search, adding CPU overhead. In a large index, this can noticeably increase latency, especially when the field is used in aggregations.
• No doc values: Because the value is not stored, it cannot benefit from the doc‑values cache that powers fast term aggregations. Some aggregations (e.g., significant_terms) may fall back to slower execution.
• Painless sandbox: Scripts run in a protected sandbox, but a poorly written script (e.g., a loose loop) can still consume excessive memory or CPU, affecting cluster stability.
• Not persisted across re‑indexing: If you re‑index the data into a new index, you must recreate the scripted field in the new index pattern.

Actionable Guidance

1. Test the script on a small time‑range first. Use the Dev Tools console to run a sample query with the 'Profile' API (GET sales-*/_search?profile=true) and compare the 'query.time_in_nanos' before and after adding the field.
2. Keep the Painless script simple: avoid loops, use built‑in math functions, and guard against null or zero values as shown.
3. If you find the performance impact unacceptable, consider alternatives:
   • Ingest pipeline processor ('script') to compute and store the field at ingest time.
   • Elasticsearch runtime fields (introduced in 7.11) which are also evaluated at query time but can be defined per‑search and may have better integration with certain aggregations.
   • Pre‑compute the field with Logstash or a downstream ETL job if the calculation is stable.
4. Document the scripted field in your team’s runbook, including the exact Painless source, the index pattern it belongs to, and any known performance notes.

By following these steps you can add useful derived metrics quickly, while staying aware of the query‑time cost and taking steps to mitigate impact on your Elasticsearch cluster.