Diagnosing Frequent Primary Elections in MongoDB Replica Sets

Recognizable Condition

Applications begin to see write failures such as NotPrimary or NotMaster errors, and the MongoDB logs show the replica set stepping down a primary and electing a new one every few minutes. Running rs.status() reveals a rapidly changing primary field and short intervals between electionTime values.

Cause and Diagnostic Table

Possible Cause	What to Look For
Network latency or packet loss between members	Missed heartbeats; members appear as RECOVERING or STARTUP2 before a new election.
Oplog window too small	Secondaries fall behind the primary faster than the oplog can retain entries, causing them to become ineligible to vote.
Inappropriate priority or votes settings	Members with priority: 0 still have votes: 1, or no member has both priority >0 and votes=1.
Heartbeat timeout too low for network jitter	Transient spikes trigger false failures even when the network is otherwise healthy.
Heavy write load saturating the oplog	Primary’s oplog fills quickly; secondaries cannot keep up, leading to lag and forced step‑downs.

Ordered Checks

1. Run rs.status() on any member (requires clusterMonitor role). Look for:

   • myState values changing frequently.
   • electionTime timestamps spaced less than 5 minutes apart.
   • members array showing a member with stateStr: "PRIMARY" that changes.

   rs.status()

2. Measure network latency between each pair of hosts. You can use ping from a shell or mongostat --host <host>:27017 to see network‑related metrics.

   # Example: from a management host
   ping mongodb2.example.net
   ping mongodb3.example.net
   # or using mongostat
   mongostat --host mongodb1.example.net:27017 --rowcount 5
   

   Required permission: none for ping; mongostat needs clusterMonitor on the target.

3. Check the oplog size and current lag.

   rs.printReplicationInfo()   # shows oplog size and time range
   rs.printSlaveReplicationInfo()  # shows lag of each secondary

   Look for:

   • Oplog time range (e.g., 2026-10-10T12:00:00Z to 2026-10-15T12:00:00Z) that is only a few hours.
   • Secondary lag reported as a large percentage of that range (e.g., >30%).

   Required permission: clusterMonitor.

4. Inspect the replica set configuration.

   rs.conf()

   Check each member for:

   • priority values (0 means cannot become primary).
   • votes (should be 1 for voting members, 0 for non‑voters).
   • heartbeatTimeoutMillis (default 10000 ms).

   Required permission: clusterMonitor (to read) or clusterManager (to change).

5. Review the MongoDB server logs for election‑related messages.

   Typical lines include:

   • [repl] election called because timeout
   • [repl] could not find member to sync from

   Logs are located under /var/log/mongodb/ (or the path set by systemLog.path). No special permission is needed to read them if you have file‑system access.

Fixes Tied to Findings

• High network latency / packet loss: Move members to the same availability zone or use a dedicated low‑latency link. If latency cannot be reduced, consider increasing heartbeatTimeoutMillis (see step 4) only after confirming the underlying link is stable.

• Oplog window too small: Re‑size the oplog using the replSetResizeOplog admin command. Example (run on the primary):

  use admin
  db.runCommand({
    replSetResizeOplog: 1,
    size: 10240   # size in MB; adjust to your workload
  })
  

  Required permission: clusterManager. Ensure sufficient free disk space; the operation may take a few minutes and will temporarily increase I/O.

• Incorrect priority/votes: Adjust the configuration so at least one member has priority > 0 and votes: 1. If you need an extra vote without adding a data‑bearing node, add an arbiter:

  rs.addArb("mongodb-arbiter.example.net:27017")
  

  Required permission: clusterManager. After adding, run rs.conf() to verify the new member appears with arbiterOnly: true.

• Heartbeat timeout too low: Increase heartbeatTimeoutMillis via a reconfiguration. Example:

  conf = rs.conf()
  conf.settings.heartbeatTimeoutMillis = 20000   # 20 seconds
  rs.reconf(conf)
  

  Required permission: clusterManager. Raising the timeout masks brief network glitches; verify that latency is not chronically high before applying.

• Write load saturating the oplog: Distribute writes by sharding the collection or adjust application write patterns (e.g., batch writes, use w:1 with appropriate jitter). If sharding is not feasible, consider adding a second secondary to share the read load, which reduces the pressure on the primary’s oplog.

Escalation Criteria

• Elections occur more than once every 5 minutes or the election process takes >30 seconds repeatedly.
• Application logs continue to show NotPrimary errors despite retry logic.
• Secondary lag reported by rs.printSlaveReplicationInfo() grows beyond the oplog window size even after increasing the oplog.
• After applying the above fixes, the condition persists for more than 30 minutes of observation.

When any of these criteria are met, involve the DBA team for a deeper architecture review (e.g., adding more nodes, revising sharding keys, or upgrading to a newer MongoDB release that may have improved election algorithms).

Verification

• After a change, run rs.status() and confirm that the same primary remains in place for at least 15 minutes with no step‑downs.
• Check replication lag: rs.printSlaveReplicationInfo() should show lag well under 10% of the oplog window (e.g., if the window is 4 hours, lag should be under 24 minutes).
• Monitor application logs for the absence of NotPrimary errors and verify that write throughput returns to baseline levels.

Limitations and Practical Checks

Increasing heartbeatTimeoutMillis can hide real network problems; always pair this change with latency measurements (ping or mongostat) to ensure the link is stable. Resizing the oplog requires free disk space equal to the new size plus a safety margin; verify with df -h on the data directory before running replSetResizeOplog. Adding an arbiter does not increase storage overhead but does add a vote; ensure an odd number of voting members to avoid tied elections.