mysqld_exporter ↔ Prometheus Alertmanager: static versus dynamic thresholds for variable workloads

Goal

Define alert thresholds for MySQL metrics exposed by mysqld_exporter that remain actionable across fluctuating workloads, including batch jobs and backup windows, while keeping the alerting pipeline simple enough for a small operations team.

Constraints

Static thresholds are easy to maintain but generate noise during predictable load spikes; dynamic or seasonal baselines reduce false positives but require additional tooling such as forecasting models or external storage, increasing operational overhead.

Open questions

• Which threshold strategy offers the best trade‑off between noise reduction and operational simplicity for a team without dedicated ML expertise?
• Can Prometheus recording rules combined with for clauses and inhibition windows sufficiently approximate seasonal behavior without external systems?
• What minimal set of duration and inhibition rules covers typical maintenance windows without masking genuine incidents?