node_exporter and Alertmanager interoperability on CentOS Stream: repeat_interval default uncertainty

Answer

The minimum repeat_interval that prevents Alertmanager from suppressing re‑notifications while avoiding duplicate pages is any positive value; in practice the lowest usable interval is about 30 seconds. Values below ~30 s can generate a notification storm when many alerts fire repeatedly, and values below ~1 minute increase fatigue without improving reliability. Therefore a safe starting point is repeat_interval: 1m, which can be lowered to 30 s if your on‑call process tolerates the extra load.

Why this works

• Alertmanager enforces at most one resend per repeat_interval regardless of group_interval, so lowering repeat_interval does not create duplicate pages.
• Suppression only occurs when repeat_interval is set to zero or omitted; any positive value triggers a resend after the configured period.
• Operational experience shows that intervals under 30 s cause noticeable overhead, while 1 min balances responsiveness and fatigue.

Configuration steps

1. Edit the Alertmanager configuration file (usually /etc/alertmanager/config.yml on CentOS Stream).
2. Set the global repeat_interval:

   global:
     repeat_interval: 1m   # or 30s if you accept the higher load

3. Define group_by to keep unrelated incidents separate while preserving repeat behavior:

   route:
     group_by: ['alertname', 'instance']   # instance = host identifier from node_exporter
     # optional refinements: add 'job' or 'severity' if needed
   

4. Add an inhibition rule to silence dependent alerts during a host outage:

   inhibit_rules:
     - source_match:
         alertname: HostDown   # or match up == 0 via a recording rule
         instance: '$1'
       target_match:
         instance: '$1'        # same instance label
       equal: ['instance']
   

5. Reload Alertmanager: systemctl reload alertmanager.

Verification

• Trigger a node_exporter alert (e.g., CPU usage > 90 % via stress) and confirm that notifications are resent every repeat_interval until the alert resolves.
• Simulate a host shutdown (systemctl stop node_exporter) and verify that the HostDown alert repeats per repeat_interval while all other alerts sharing the same instance are suppressed.
• Check the Alertmanager UI or logs to ensure that alerts from different hosts appear in separate groups (based on alertname + instance).

If you observe that notifications are not being resent as expected, check whether repeat_interval is inadvertently set to zero or commented out in the configuration.