Replica Set Alert Evaluation: Primary‑Only vs Per‑Node Assessment in MongoDB Cloud Manager/Atlas
0 reputation · 16 Jun 2025, 06:14 UTC
0 reputation · 16 Jun 2025, 06:14 UTC
Determine whether alert configurations for replication‑lag metrics in MongoDB Cloud Manager or Atlas are evaluated on the primary node only or independently on each secondary when setting alerts on replica set members.
Documentation does not specify if the alert engine aggregates lag from secondaries or checks each node separately, causing ambiguity when configuring thresholds and interpreting notifications.
Does an alert fire when any secondary exceeds the lag threshold, or only when the primary’s view of lag exceeds it? Are thresholds applied per‑node and, if so, how are notifications aggregated? What behavior should be expected when a secondary steps down or becomes unavailable?
In MongoDB Cloud Manager and Atlas, replication‑lag alerts are not hard‑coded to the primary or to each secondary; they are evaluated according to the Scope option you set when creating or editing an alert.
Primary only and All members (per‑node).Primary only, the alert condition is evaluated once per replica set, using the lag value reported by the primary.All members, the condition is evaluated independently on each host that is currently reachable; a separate alert incident is generated for any member that exceeds the threshold.If you are unsure which scope is currently applied to your replication‑lag alert, please let me know the present Scope setting so I can confirm whether the alert will fire on any secondary or only when the primary’s view exceeds the threshold.
Use comments to ask for clarification. Post a solution as an answer.
31,775 reputation · 16 Jun 2025, 09:38 UTC
When Scope is set to All members, the alert engine creates a separate incident per secondary whose lag series breaches the threshold. Each incident carries the specific host:port in the notification payload, so you can route or auto‑remediate by member. The “for at least N minutes” duration is also evaluated per series—a brief spike on one secondary won’t trigger if it recovers before the window elapses, while another secondary that stays lagged will fire independently.
If you need a single cluster‑level alert (e.g., “any secondary > 30 s for 5 min”), the built‑in REPLICATION_LAG metric does not aggregate; you would create a custom alert using an MQL expression against admin.replSetGetStatus or a dashboard‑derived max‑lag series. This behavior reflects the current Atlas/Cloud Manager monitoring agent (2024+); legacy Ops Manager agents may differ.