Scripted Fields vs Ingest Pipelines: When to Compute Data in Kibana or at Index Time

Problem Statement

In many dashboards you need a field that isn’t stored in the original documents – for example, concatenating a first and last name, normalising dates, or calculating a derived metric. Kibana offers two ways to create such fields without modifying the source data:

• Scripted fields – evaluated on‑the‑fly when a search runs.
• Ingest pipelines – run once during indexing and store the calculated value in the document.

Choosing the right approach depends on performance, maintenance, and data‑freshness requirements. This guide helps you decide by laying out constraints, trade‑offs, and a concrete implementation example.

Decision Context

When you’re building a Kibana dashboard, you often face these questions:

• Do I need the derived value for many queries or just a few?
• Is the calculation heavy or trivial?
• How often does the underlying data change?
• Can I afford to reindex the index or run a background job?

Answering these will steer you toward the appropriate feature.

Comparison Table

Feature	When to Use	Pros	Cons	Example Use Case
Scripted field (Kibana)	Quick prototypes, infrequent calculations, low query volume.	Zero reindexing, no pipeline maintenance, instant changes.	Display a calculated “full_name” for a small user list.
Ingest pipeline (Elasticsearch)	High‑volume dashboards, frequent queries, heavy calculations.	Fast queries, lower client load, reusable across indices.	Pre‑compute a “transaction_score” for real‑time sales analytics.

Trade‑Off Analysis

Performance

Scripted fields add latency to every search because the script runs in the Kibana server for each document returned. Ingest pipelines move that work to the ingestion path, so queries read a pre‑computed value and are typically 3–5× faster on large result sets.

Maintenance

Scripted fields live in Kibana’s UI. Changing the logic is a UI click and the change propagates immediately. Ingest pipelines are part of the Elasticsearch cluster; updating them requires API calls and, if you need to re‑apply to existing documents, a reindex operation.

Reusability

Scripted fields are tied to a single index pattern. If you have multiple indices that need the same calculation, you must duplicate the logic in each pattern. Ingest pipelines can be shared across indices by referencing the same pipeline name.

Data Freshness

Scripted fields always see the latest document data because they compute on demand. Ingest pipelines only reflect the state at indexing time; if a document changes after indexing, the derived value must be updated by re‑indexing or a background job.

Concrete Implementation

1. Create an Ingest Pipeline

Run the following on the Elasticsearch node that hosts the index. You need cluster‑admin privileges.

PUT /_ingest/pipeline/full_name_pipeline
{
  "description": "Build full_name from first_name and last_name",
  "processors": [
    {
      "script": {
        "lang": "painless",
        "source": "ctx.full_name = ctx.first_name + ' ' + ctx.last_name"
      }
    }
  ]
}

Verify the pipeline with a simulation (no data is written):

POST /_ingest/pipeline/full_name_pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "first_name": "Alice",
        "last_name": "Smith"
      }
    }
  ]
}

Check the _ingest._pipeline field in the response to confirm full_name is added.

2. Attach the Pipeline to an Index

When indexing new documents, specify the pipeline in the index request:

POST /users/_doc?pipeline=full_name_pipeline
{
  "first_name": "Bob",
  "last_name": "Jones",
  "age": 34
}

Existing documents can be re‑indexed:

POST /users/_reindex
{
  "source": {
    "index": "users"
  },
  "dest": {
    "index": "users_reindexed",
    "pipeline": "full_name_pipeline"
  }
}

After reindexing, full_name will be stored in every document.

3. Create a Kibana Scripted Field (Optional Alternative)

Navigate to Stack Management > Index Patterns, select the users pattern, and add a scripted field named full_name_script with the following painless script:

doc['first_name'].value + ' ' + doc['last_name'].value

This field will appear in search results and visualizations but is computed at query time.

Validation Checklist

1. Pipeline Simulation – Ensure the _simulate response contains the expected field.
2. Indexing Test – Index a single document with the pipeline and run GET /users/_search?pretty to verify full_name appears in _source.
3. Query Performance – Measure the took value for a dashboard query that uses the derived field with and without the pipeline. A noticeable reduction indicates success.
4. Data Freshness – Update an existing document’s last_name and confirm whether full_name changes (it should not unless reindexed).

Limitations & Caveats

• Scripted fields can’t be used in aggregations that require field data; you must enable field data or use a scripted metric, which further adds latency.
• Ingest pipelines lock the index during reindexing; plan for maintenance windows.
• Large ingest pipelines can increase the ingestion pipeline queue size; monitor _cat/ingest/pipelines for performance.
• Both approaches use painless, which has a limited set of functions compared to full JavaScript.

Practical Takeaway

If you need a derived value for a handful of dashboards and can tolerate a small query‑time overhead, a scripted field is the fastest path. If you’re serving thousands of queries per minute, the calculation is non‑trivial, or you want the value to be query‑ready, create an ingest pipeline and reindex or re‑apply to existing data. Always validate performance and data freshness before rolling out to production.