Alert flapping from insufficient recovery window duration

Answer

For a metric whose natural variance repeats every P minutes, the recovery window must be at least that period to stop flapping. The exact minimum differs by condition type:

• Static threshold: set recovery window ≥ P. If the window equals or exceeds the full oscillation cycle, the metric cannot cross the threshold again before the window elapses, so the incident stays closed.
• Baseline or anomaly detection: set recovery window ≥ P × (1 + buffer), where a typical buffer is 20‑50 % (i.e., 1.2 P to 1.5 P). The extra time accommodates baseline drift and ensures the anomaly score stays within bounds for a stable period.

Note: The aggregation method influences the needed window. Average aggregation smooths short spikes, often allowing a window close to P even for anomaly detection. Maximum aggregation preserves spikes, so you may need to add the buffer even for static thresholds.

Verification steps

1. Identify the incident’s open and close timestamps from the Alerts UI or via GET /v2/alerts/incidents/{id}.
2. Run an NRQL query that covers that interval, using a TIMESERIES bucket size equal to the configured recovery window:

SELECT average(value) FROM Metric
WHERE timestamp >= :open AND timestamp <= :close
TIMESERIES :recoveryWindow

3. For static thresholds, verify that every returned point stays ≤ threshold (or ≥ threshold for lower‑bound alerts). For anomaly detection, verify that each point’s deviation from the baseline stays within the anomaly‑score bound used in the condition.
4. If the interval falls outside the account’s data retention window (default ~8 days for high‑resolution custom metrics, ~30 days for APM), the NRQL query will return no data. In that case:
• Enable extended data retention or export the metric to an external store (e.g., New Relic Data Plus, S3) before the data ages out.
• Without exported data, you cannot directly verify the recovery condition; rely on the incident’s evaluationWindow and closedReason fields from the Alerts API, which indicate whether New Relic’s internal logic deemed the recovery satisfied at close time.

Missing diagnostic detail

To give a precise window recommendation, confirm the aggregation method used in the condition (average, maximum, min, or percentiles). If you are using maximum aggregation, add the anomaly‑detection buffer even for static thresholds; otherwise the basic ≥ P rule suffices.