Creating and Managing Kibana Index Patterns for Time‑Series Visualization

Desired Outcome

Define a Kibana Index Pattern (now called a Data View) that links to a set of Elasticsearch indices, configures a time field for the time‑picker, and allows scripted fields for on‑the‑fly calculations. The pattern should be active, searchable in Discover, and usable in dashboards.

Prerequisites

• Elasticsearch 8.x cluster with indices that contain a date field (e.g., @timestamp).
• Running Kibana 8.x with access to Stack Management and Discover tabs.
• Admin or Kibana user with kibana_admin role.
• Optional: API access via curl or Kibana Dev Tools console.

Step‑by‑Step Procedure

1. Inspect the Index Mapping

Before creating a pattern, confirm the date field. Replace my-index-* with your index name pattern.

curl -X GET "http://localhost:9200/my-index-*/_mapping?pretty"

Look for a field named @timestamp (or your chosen time field) with type date. If absent, either add it via index template or choose a different field.

2. Create the Index Pattern via UI

1. Navigate to Stack Management > Data Views (previously Index Patterns).
2. Click Create data view.
3. Enter the index pattern name, e.g., my-index-*, and click Next step.
4. In the Time field drop‑down, select @timestamp (or the date field discovered earlier).
5. Review the preview of the first 10 documents to confirm the time range appears correct.
6. Click Save and close to activate the pattern.

3. Create the Index Pattern via API (Optional)

For automation, use the POST /api/data_view endpoint. Replace placeholders accordingly.

curl -X POST "http://localhost:9200/api/data_view" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -d '{
  "name": "my-index-*",
  "timeFieldName": "@timestamp"
}'

Check the response for created: true and a valid id.

4. Add a Scripted Field (Optional)

Scripted fields allow runtime calculations. For example, compute a 24‑hour rolling average of a metric called cpu_load.

1. In the Data Views list, click the created pattern.
2. Go to Scripted fields and click Add scripted field.
3. Enter:
   • Name: cpu_load_avg_24h
   • Language: painless
   • Type: number
   • Script: doc["cpu_load"].value (replace with actual aggregation logic if needed).
4. Save and test in Discover.

5. Verify in Discover

1. Open the Discover tab.
2. From the data view selector, pick my-index-*.
3. Use the time picker to set a recent range (e.g., last 7 days).
4. Check that documents populate and the cpu_load_avg_24h column appears.
5. Test field filtering: click the cpu_load field and apply a range filter.

6. Refresh Field List After Mapping Changes

If you add new fields to the index (e.g., via index template), you must refresh the pattern:

curl -X POST "http://localhost:9200/api/data_view/my-index-*/_refresh" -H "kbn-xsrf: true"

Verify the new field appears in the Data View’s field list.

Expected Checks

• Index Pattern status shows Active in Stack Management.
• Discover displays documents within the selected time range.
• Fields are correctly typed: numeric fields allow range filtering; date fields enable time‑picker.
• Scripted fields evaluate without errors in Discover.

Recovery Options

• If the time field is wrong, delete the pattern and recreate it with the correct field.
• To revert scripted field changes, delete the scripted field from the Data View UI.
• When a mapping change breaks dashboards, refresh the Index Pattern and update any saved searches that reference the old field name.

Limitations and Practical Checks

• Wildcards like * can match many indices; limit the pattern (e.g., logs-2024.*) to avoid performance hits.
• Changing the time field after dashboards are built may invalidate time‑based visualizations. Test dashboards after modifications.
• Scripted fields are evaluated per document at query time; heavy scripts can degrade query performance. Use them sparingly.
• Always validate the pattern in Discover before embedding it into dashboards.

Conclusion

By following this guide, you set up a robust Kibana Data View that links to your Elasticsearch indices, configures an appropriate time filter, and optionally extends data with scripted fields. Verification in Discover and periodic refreshes keep the view in sync with the underlying index mapping.