Configuring Kibana Data Views for Efficient Elasticsearch Discovery

The Challenge: Connecting Kibana to Dynamic Indices

Kibana cannot query Elasticsearch indices directly by name in the Discover or Dashboard interfaces. Instead, it requires a Data View (formerly known as an Index Pattern). Without a properly configured Data View, you cannot perform time-series analysis, filter logs using KQL, or build visualizations.

The primary goal is to create a bridge that allows Kibana to target a group of indices—often those that rotate daily—without needing to update the configuration every time a new index is created.

Prerequisites

• An active Elasticsearch cluster with at least one index containing data.
• Kibana access with all or manage privileges for Stack Management.
• Knowledge of the index naming convention (e.g., logs-app-prod-2026.10.15).

Step-by-Step Configuration

1. Define the Index Pattern

Navigate to Stack Management > Data Views and select Create data view. Use a wildcard (*) to ensure the view captures all current and future indices matching the naming scheme.

# Example Pattern: logs-app-*
# This matches: 
# logs-app-prod-01
# logs-app-dev-01
# logs-app-test-01

2. Configure the Time Filter

If your data contains timestamps, you must select the correct time field. This enables the global time picker in Kibana and allows the system to generate histograms for your data.

• Timestamp field: Select the field (e.g., @timestamp) that represents when the event occurred.
• Risk: Selecting the wrong field or changing it later will break any existing dashboards or saved searches that rely on that specific time axis.

3. Implementing Runtime Fields

When the underlying Elasticsearch mapping is missing a calculated value, use a Runtime Field. This allows you to create a new field at query time using Painless scripting without re-indexing your data.

Example Scenario: You have bytes_sent and bytes_received, but you need a total_traffic field for a visualization.

// Runtime Field Script (Painless)
emit(doc['bytes_sent'].value + doc['bytes_received'].value);

Comparison: Static vs. Wildcard Patterns

Feature	Static Pattern (e.g., logs-app-01)	Wildcard Pattern (e.g., logs-app-*)
Scope	Single specific index	All matching indices
Maintenance	High; must create new view for every index	Low; automatically includes new indices
Performance	Fastest; targeted query	Variable; depends on the number of indices matched

Verification and Diagnostics

To ensure the Data View is functioning correctly, perform the following checks in the Discover tab:

1. Pattern Selection: Select your newly created Data View from the dropdown. If no documents appear, verify that the index exists in Elasticsearch using the Dev Tools console: GET _cat/indices/logs-app-*?v.
2. KQL Validation: Enter a query in the search bar (e.g., status: 500). If the results are empty but you know the data exists, check if the field mapping was correctly detected from the index template.
3. Time Alignment: Adjust the time picker to a range where you know data exists. If the histogram shows no bars, the timestamp field selected in the Data View may not match the data format in the index.

Limitations and Risks

• Field Explosion: If your indices have thousands of unique fields, the Discover field list will load slowly. Consider using an index template to limit the number of fields.
• Over-broad Wildcards: Using a pattern like * will query every index in the cluster, which can lead to severe performance degradation and timeout errors in large environments.

Rollback Procedure

Because creating a Data View is a metadata operation within Kibana and does not modify the underlying Elasticsearch data, the rollback is simple:

1. Navigate to Stack Management > Data Views.
2. Select the problematic Data View.
3. Click the Delete button. This removes the bridge but leaves the indices intact.