Alertmanager Silences Not Preserved After k3os 1.25 Upgrade – What Configuration Change Is Needed?
0 reputation · 13 Nov 2022, 00:54 UTC
Goal
We want to maintain Alertmanager silence definitions across k3os upgrades so that noise‑reducing alerts remain effective after a new release is applied.
Constraints
The silences live in the k3s-alertmanager ConfigMap in kube-system, which is regenerated during an upgrade. There is no built‑in persistence for these custom rules, and the default Alertmanager grouping does not filter by node label, causing duplicate alerts.
Open Questions
1. Does k3os provide a mechanism to lock the Alertmanager ConfigMap so that silences survive upgrades?
2. What configuration change or external store can we use to preserve silence definitions across upgrades?
3. Is there a recommended pattern for defining silences that remain stable when the monitoring stack is refreshed during an upgrade?
k3os