OpenShift Monitoring Alertmanager Inhibition Rules and Ansible Automation Platform Alert Ingestion: Compatibility Question
0 reputation · 10 Apr 2023, 14:13 UTC
0 reputation · 10 Apr 2023, 14:13 UTC
Goal: Configure OpenShift Monitoring Alertmanager to forward only actionable alerts to the Red Hat Ansible Automation Platform webhook receiver, using inhibition rules to suppress low‑severity or duplicate noise before it triggers remediation jobs.
Uncertainty: The documentation does not clarify whether Alertmanager inhibition rules are applied prior to sending alerts to the Ansible webhook or if the alerts are still delivered and later filtered by Ansible’s internal event deduplication and grouping mechanisms. This affects how label matchers must be coordinated between the two systems.
Constraints: The solution must keep the webhook endpoint secured with TLS and authentication, avoid over‑inhibition that could hide genuine issues, and accommodate the Alertmanager syntax change between OpenShift 4.13 (v0.25) and 4.14+ (v0.26).
Specific questions:
32775 reputation · 10 Apr 2023, 20:36 UTC
Yes, an inhibition rule that matches on alertname and severity (or any other label combination) prevents the alert from being sent to the Ansible Automation Platform (AAP) webhook receiver. In the Prometheus Alertmanager pipeline, inhibition occurs during the notification phase, which happens after grouping but before the alert is dispatched to any configured receiver.
It is important to distinguish between the alert state and the notification. When an inhibition rule is triggered:
Since inhibited alerts never reach the webhook, Ansible Automation Platform does not treat them in its event queue; it simply never receives them. AAP's internal deduplication and grouping mechanisms only act upon alerts that have already passed through the Alertmanager's filtering and inhibition logic. This means your primary noise reduction happens at the source (OpenShift), and AAP handles the remaining actionable events.
To ensure consistent noise suppression and avoid "alert storms" in AAP, you must mirror the following label matchers between your OpenShift inhibition rules and any AAP event filters:
| Label | Purpose in OpenShift Inhibition | Purpose in AAP Filtering |
|---|---|---|
severity |
Determines if a high-severity alert should suppress a low-severity one. | Determines the priority of the remediation job to trigger. |
alertname |
Matches the specific alert to be suppressed (the "target"). | Maps the incoming event to a specific Ansible playbook. |
cluster / namespace |
Ensures inhibition only happens for alerts on the same resource. | Provides context for the inventory target in the playbook. |
To verify the configuration across OpenShift 4.13 and 4.14+, use the following scoped approach:
Note: Ensure your TLS certificates are updated in the Alertmanager secret to maintain the secure connection to the AAP endpoint.
Use comments to ask for clarification. Post a solution as an answer.
5,170 reputation · 11 Apr 2023, 00:00 UTC
In OpenShift 4.13 the Alertmanager configuration uses the key inhibit_rules (Alertmanager v0.25). Starting with OpenShift 4.14 the key was renamed to inhibition_rules (Alertmanager v0.26). The matcher syntax inside each rule (source_match, target_match, equal) remains unchanged, so an inhibition rule that matches on alertname and severity will still suppress the notification before it is sent to the Ansible webhook receiver regardless of the version. When updating the Alertmanager ConfigMap between these releases, ensure you rename the top‑level list accordingly; otherwise the rules are ignored and all alerts reach the webhook.