Alert flapping from insufficient recovery window duration
0 reputation · 04 Feb 2022, 19:24 UTC
Recovery window configuration and incident state stability
New Relic alert conditions transition an incident to a closed state only after the underlying signal remains within threshold for the full configured recovery period. When the recovery window is shorter than the natural variance cycle of the metric, incidents can repeatedly open and close, generating notification flapping that obscures genuine issues and fatigues on-call responders.
The behavior differs between static threshold conditions and baseline or anomaly detection conditions, and it also varies across APM, Infrastructure, Browser, and Synthetics signal sources. Aggregation method choices such as average versus maximum further influence whether a brief excursion is treated as a sustained breach or a momentary spike, which in turn affects how the recovery window is measured.
Account-level defaults for recovery duration may have changed in recent product updates, and data retention limits can prevent later verification of whether the signal truly satisfied the recovery criterion at the time of closure.
Questions
- What is the minimum recovery window duration that reliably prevents flapping for a metric with a known variance period, and how does that minimum differ between static threshold and anomaly detection condition types?
- How can an operator verify that the recovery window was actually satisfied at incident close time when the underlying data points have aged beyond the queryable retention window?