High false‑positive rate in nginx stub_status alerting

Operators want to configure alerting for nginx's stub_status metric to detect abnormal request rates without generating excessive noise from normal traffic fluctuations.

The goal is to set thresholds that distinguish genuine anomalies (e.g., sudden drops or spikes indicative of backend issues) from routine variance caused by bursty client behavior or scheduled jobs. Constraints include the need to avoid alert fatigue, maintain visibility of real incidents, and work with the default 1‑second scrape interval of common monitoring systems.

Uncertainty remains about the appropriate statistical method (simple threshold vs. moving average vs. percentile‑based) and the safe window size that balances responsiveness with noise suppression.

• What thresholding strategy provides the best signal‑to‑noise ratio for stub_status request‑per‑second metrics?
• How large a look‑back window should be used to smooth normal bursts while still catching rapid degradations?
• Should alerts be based on absolute values, relative deviation from a baseline, or a combination of both?