Diagnosing MongoDB WiredTiger Index Build Write Blockage

Recognizable Condition

Write operations on a MongoDB collection suddenly experience high latency or time‑outs. In the server log you see messages like "waiting for lock" or "operation was killed because it exceeded the maxTimeMS limit". The cluster appears healthy, but the write throughput drops sharply while an index build is in progress.

Cause/Diagnostic Table

Possible Cause	What to Look For
Long‑running foreground index build on a large collection	db.currentOp() shows an operation with "op" : "createIndexes" and "waitingForLock" : true; the operation has been running for many seconds/minutes.
Disk I/O saturation slowing the temporary index file creation	WiredTiger cache percent used is high (>90%) and LRU percent pinned is rising; iostat or mongotop shows elevated write wait times.
Collection‑level lock during index build blocking concurrent writes	write operations appear in the "waitingForLock" queue with the same lock mode (W) as the index build.
Secondary indexes on the same collection compounding lock contention	Multiple index build operations (foreground or background) appear simultaneously for the same namespace.
System resource limits (e.g., ulimit for open files) throttling the build	Server status shows "recordStalls" or "wiredTiger.concurrentTransactions" write tickets exhausted; OS logs show "too many open files".

Ordered Checks

1. Confirm an index build is in progress

   # Run on the mongod instance (requires clusterMonitor role)
   db.currentOp({ "$or": [ { "waitingForLock": true }, { "inprog": true } ] })
   

   Look for entries where "op" : "createIndexes" and "secs_running" is large.
2. Check for blocked write operations

   db.currentOp({ "waitingForLock": true, "op": { "$ne": "createIndexes" } })
   

   If write ops appear here, they are being queued by the index build lock.
3. Assess WiredTiger I/O pressure

   db.serverStatus().wiredTiger.cache
   # fields: percentUsed, bytesReadIntoCache, bytesWrittenFromCache
   

   High percentUsed (>85%) combined with rising "bytesWrittenFromCache" suggests the disk cannot keep up.
4. Monitor write latency during the build

   db.collection.stats().writeLatency
   # Compare historic baseline vs. current value
   

   A spike concurrent with the index build confirms the impact.
5. Verify replica set primary load (if applicable)

   rs.status()
   # Look at the primary's "optimeDate" lag and "memory" usage
   

   If the primary shows high lag or CPU > 80%, the index build may be contributing to replication pressure.

Fixes Tied to Findings

• If the index build is foreground and long‑running – Re‑create the index with the background:true option.

  db.collection.createIndex( { field: 1 }, { background: true, maxTimeMS: 300000 } )
  

  Note: background builds still take a brief metadata lock at start and end, but they allow concurrent writes during the bulk of the work. Risk: If the build is interrupted, the partial index is discarded; verify with db.collection.getIndexes() after completion.
• If disk I/O is saturated – Throttle the build using maxTimeMS or split the index into smaller batches. Example: create the index on a sharded key range or on a subset using a partial index filter, then merge. Check: After throttling, re‑run the WiredTiger cache metrics; percentUsed should stabilize.
• If multiple indexes are being built concurrently – Serialize the builds. Issue one createIndexes command, wait for it to finish (monitor with currentOp), then proceed to the next.
• If system ulimits are limiting open files – Increase the limit for the mongod process (requires OS admin). Example: ulimit -n 64000 before starting mongod, or set fs.file-max in /etc/sysctl.conf. Verification: After the change, db.serverStatus().wiredTiger.concurrentTransactions should show available write tickets.
• If the primary is overloaded in a replica set – Offload reads to secondaries or add another secondary to distribute read load, leaving more CPU for the index build. Escalation: If write latency does not drop after the above steps, consider pausing the build (db.currentOp().killOp on the index build operation) and rescheduling during a maintenance window.

Escalation Criteria

• Write latency remains > 2× baseline after applying the relevant fix.
• The index build operation shows "waitingForLock" with a growing queue (>10) despite background mode.
• OOM or lock starvation messages appear in the mongod log.
• Replication lag on secondaries exceeds the configured slaveDelay or causes election instability.

When any of these criteria are met, open a support ticket with your MongoDB provider or schedule a dedicated maintenance window to drop and rebuild the index offline (e.g., using a snapshot or rolling restart).

Practical Way to Check the Result

After applying a fix, repeat the checks in section 2:

1. Run db.currentOp() and confirm no "createIndexes" operation has "waitingForLock": true.
2. Verify that write operations no longer appear in the waitingForLock queue.
3. Ensure db.collection.stats().writeLatency returns to within 10 % of the pre‑build baseline.
4. If using a replica set, check rs.status() for normal optime lag (< 5 seconds under typical load).

If all checks pass, the index build is no longer blocking writes.