Question
Pulumi Service Alert Configuration: Mitigating Duplicate Notifications Across Channels
Nia Nova
0 reputation · 01 Feb 2024, 15:44 UTC
114.1K views
Loading…
0 reputation · 01 Feb 2024, 15:44 UTC
When configuring Pulumi Service alerts, the objective is to receive actionable notifications for stack failures, drift, and provider mismatches without overwhelming teams with duplicate messages. Each alert can be routed to multiple channels—email, Slack, Teams, or webhooks—so a single event may generate several identical notifications.
Alert rules can be scoped to individual stacks, projects, or the entire organization. Triggers include stack update failures, successes, drift detection, provider version mismatches, and custom runtime errors. The UI allows filtering by severity, resource type, and channel.
The Pulumi Service does not provide built‑in throttling or deduplication across channels. Consequently, a single stack update failure can result in simultaneous alerts in Slack, email, and a webhook, creating noise and potential alert fatigue.
How can we configure the alert system to suppress or de‑duplicate notifications when multiple channels are active? Is there an option to set per‑channel throttling or a global deduplication window? What best‑practice patterns exist for balancing timely alerts with minimal noise across the Pulumi alert ecosystem?
35275 reputation · 01 Feb 2024, 18:22 UTC
The Pulumi Service does not currently feature a native global deduplication window or per-channel throttling mechanism. When an alert rule is configured to send notifications to multiple channels (e.g., Slack, Email, and Webhooks), the service triggers an independent notification for each enabled channel simultaneously upon the event trigger.
Since there is no built-in toggle to suppress duplicates across channels, you must manage noise through rule scoping and external routing logic.
Avoid assigning every channel to every alert rule. Instead, map specific channels to the severity of the event to ensure critical alerts are redundant while routine updates are singular.
For teams requiring strict deduplication or throttling, the most effective pattern is to route all Pulumi alerts through a single Webhook integration rather than using the native direct integrations for every channel.
Ensure you are not overlapping rules at different levels of the Pulumi hierarchy. If an alert is set at the Organization level and a similar alert is set at the Stack level, a single event will trigger both rules, doubling the notifications across all assigned channels.
To verify your current noise level and the impact of these changes, use a test stack:
pulumi
slack# Trigger a controlled failure to observe notification behavior
pulumi up --yes # (with a known configuration error)Check the Notifications section in the Pulumi Service console to ensure no overlapping rules exist between the Project and Organization scopes.
To provide a more specific routing recommendation: Are you currently using the native Pulumi UI integrations for all channels, or are you utilizing a custom webhook for any part of your pipeline?
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 01 Feb 2024, 19:28 UTC
To mitigate duplicate notifications, it is important to distinguish between multi-channel broadcasting (one rule sending to many destinations) and overlapping scopes (multiple rules matching one event). In the Pulumi Service, duplicates often occur when an organization-wide alert rule and a stack-specific rule both trigger for the same failure.
Since the service lacks built-in cross-channel throttling, consider these structural adjustments:
To verify the source of duplicates, trigger a controlled failure and compare the payload IDs across channels; identical IDs suggest a single rule broadcasting, while different IDs indicate overlapping rule scopes.