Kibana Threshold Alerts: Configure, Verify, and Avoid Common Pitfalls

Start Here: What You Need to Know

Threshold alerts in Kibana detect when a metric exceeds a defined limit, such as CPU over 90% or error rate above 5%. Alerts fail silently if configured incorrectly. The key is understanding that Kibana stores alerts as Saved Objects and runs queries on its own schedule, not Elasticsearch's.

How Threshold Alerts Work Under the Hood

When you create a threshold alert, Kibana:

1. Saves the rule definition in the .kibana index as a saved object of type alert
2. Schedules execution based on the check interval, for example every minute
3. Runs the embedded Elasticsearch query against your data index
4. If results exceed your threshold, triggers the configured action connector

The alerting engine runs on the Kibana server process. This matters for performance and troubleshooting.

Concrete Example: CPU Usage Alert

Create an alert that triggers when average CPU exceeds 90% for 5 minutes.

Step 1: Define the Rule via Kibana UI

Navigate to Stack Management, then Rules and Connectors, then Create rule. Select Elasticsearch query threshold.

Step 2: Configure the Query

In the query builder:

• Index pattern: metrics-* (must match your data index)
• Time field: @timestamp
• Aggregation: Average on field system.cpu.total.pct
• Time range: Last 5 minutes (critical: matches your threshold window)

Step 3: Set Threshold Conditions

Condition: is above
Threshold: 90

Trigger when:
- All conditions are met (single condition in this case)
- For at least 1 time(s)

Step 4: Configure Action Connector

Select or create a connector. For email, enter SMTP settings in kibana.yml or via the UI:

xpack.actions.email:
  host: smtp.example.com
  port: 587
  user: "[contact removed]"
  password: "your-password"
  tls: true

Verify Your Alert is Saved Correctly

After saving, verify the alert exists as a saved object. Run this in the Kibana Dev Tools console:

GET .kibana/_search
{
  "query": {
    "term": {
      "type": "alert"
    }
  }
}

This should return your alert's JSON definition. If it is missing, the rule failed to persist.

Common Mistakes and How to Fix Them

Mistake 1: Wrong Index Pattern

Symptom: Alert never triggers, no errors in logs.

Fix: Verify the index pattern exists and contains data. Check the index name in Stack Management and run a simple count query against the target index in Dev Tools.

Mistake 2: Time Range Mismatch

Symptom: False negatives. Data exists but the alert does not fire.

Fix: Ensure the query time range, such as Last 5 minutes, aligns with your threshold window. Kibana's alert engine uses the time filter literally.

Mistake 3: Missing Action Configuration

Symptom: Rule shows as active but no notifications are sent.

Fix: Check Kibana server logs for connector errors. On a systemd host, you can inspect the service logs:

sudo journalctl -u kibana | grep -i connector

Common issues include SMTP authentication failures, webhook URL typos, and missing headers.

Mistake 4: High-Frequency Polling

Symptom: Kibana becomes unresponsive, Elasticsearch cluster shows high query load.

Fix: Avoid intervals below 1 minute for complex queries. Each alert creates a separate query execution. For example, 100 alerts checking every 30 seconds would generate 200 queries per minute against your cluster.

License Limitations

Certain advanced alerting features and action connectors require a paid license, such as Platinum or Enterprise, beyond the Basic tier. Review your current license before relying on specific connectors or advanced workflows.

Practical Verification Checklist

1. Test rule execution: In the rule UI, run the rule manually and verify the action fires.
2. Check saved object: Query the .kibana index for your alert's type and ID.
3. Monitor server logs: Look for alert or connector keywords after execution.
4. Validate time fields: Ensure your data index has @timestamp populated correctly.

Key Takeaway

Kibana threshold alerts are powerful but require careful attention to index patterns, time ranges, and connector configuration. The alerting engine runs on Kibana's server process, not Elasticsearch, so performance and debugging must account for this architecture. Always verify saved objects exist and test rule execution manually before relying on automated alerts.