Minimal Architecture for MongoDB Change Streams on a Replica Set

Problem and Takeaway

Many applications need to react instantly to data changes in MongoDB without resorting to periodic polling. The challenge is to capture those changes reliably while respecting data trust boundaries, maintaining operational health, and handling failures such as primary step‑downs or network interruptions.

The useful takeaway is that a minimal replica set with a change stream, proper resume token handling, and simple health checks can provide a production‑ready real‑time pipeline without additional infrastructure.

Requirements

• A replica set (or sharded cluster with a config server replica set) is mandatory; standalone instances cannot use Change Streams.
• MongoDB version 4.0 or later, because Change Streams were introduced in 4.0.
• The originating write must occur on a primary node; the stream pauses during primary elections.
• Cursor timeout (default 10 minutes) must be managed to avoid unexpected closure.
• Resume tokens must be stored durably; loss leads to duplicate or missing events after restarts.
• Sufficient memory on the primary to hold the cursor during high write throughput.

Smallest Suitable Design

The minimal architecture consists of a single‑node replica set (one primary, one secondary) deployed on two servers or two VMs. This satisfies the replica set prerequisite while keeping the deployment simple.

Replica Set Requirement

Change Streams read from the oplog of a replica set member. A primary is required to write to the oplog and to emit events. If the primary steps down, the cursor is paused until a new primary is elected, after which the stream resumes automatically.

Trust and Data Boundaries

The stream only delivers events that are replicated to the secondary members. Writes that never reach a secondary (e.g., due to network partition) are not visible to the client. Therefore, the client should consider the replica set’s write concern when defining trust boundaries.

Operational Checks

1. Verify replica set status: rs.status(). Ensure at least one secondary is up and that the primary is reachable.
2. Check cursor timeout: db.serverStatus().metrics.cursor shows the default 600 seconds (10 minutes). Adjust with db.adminCommand({ setParameter: 1, cursorTimeoutMillis: 3600000 }) if longer periods are needed.
3. Monitor primary health: rs.status().myState should be 1 (primary). Alert if it changes to 2 (secondary) unexpectedly.
4. Track memory usage of the cursor: db.serverStatus().metrics.globalLock and db.serverStatus().mem give an indication of cursor memory pressure.
5. Persist the resume token after each batch: store it in a reliable datastore (e.g., a separate MongoDB collection or a durable key‑value store).

Failure Modes and Design Adjustments

• Primary step‑down: the stream pauses. Implement a reconnection loop that re‑creates the cursor after election.
• Cursor timeout: after 10 minutes of inactivity the server closes the cursor. Keep the cursor active by sending a ping or by processing events continuously; if idle periods are expected, increase the timeout.
• Network partition: events may be missed if the primary cannot send them to the secondary. Use a write concern of “majority” to ensure durability before the event becomes visible to the stream.
• High write throughput: the cursor can consume significant RAM. Periodically checkpoint the resume token and consider scaling the primary hardware or adding a dedicated secondary for oplog replication.
• Resume token loss: if the token is not persisted, the client may start from the beginning after a restart, causing duplicate processing. Use a transaction or atomic update to store the token.

Concrete Example

The following steps demonstrate a minimal, verifiable setup. All commands assume a user with administrative privileges on the primary.

1. Start a two‑node replica set. On host A and host B, create a data directory and run:

mongod --replSet rs0 --port 27017 --dbpath /data/rs0 --bind_ip_all

2. Connect to host A and initiate the replica set:

mongo --host hostA:27017
rs.initiate(
  { _id: "rs0", members: [
    { _id: 0, host: "hostA:27017" },
    { _id: 1, host: "hostB:27017" }
  ]}
)

3. Verify the set is healthy:

rs.status()

4. Create a test collection and insert a document:

use testdb
db.createCollection("events")
db.events.insertOne({ name: "sample", value: 42 })

5. Open a client (e.g., Node.js, Python, or the mongo shell) on any secondary and subscribe to the change stream:

db.events.watch([], { resumeAfter: "" })

6. The watch cursor will emit an event corresponding to the insert. Store the resumeToken from the first event:

const token = event.resumeToken;

7. Simulate a restart by closing the client, then re‑open and re‑create the watch with the saved token:

db.events.watch([], { resumeAfter: token })

8. Insert another document and confirm that the stream delivers the new event without duplication.

Verification Steps

1. Confirm MongoDB version: mongod --version should report 4.0+.
2. Ensure a replica set exists: rs.status() must show a primary and at least one secondary.
3. Perform the test insert and observe the change event in the client. The event type should be "insert" with the correct document payload.
4. Interrupt the client (e.g., kill the process) and restart it. Verify that the watch resumes from the stored resume token, indicated by the same resumeAfter value in the server logs.

Limitations and Practical Validation

• Version constraint: Change Streams are unavailable on MongoDB 3.6 or earlier. Verify the version before design.
• Idle timeout: if the application pauses longer than the server’s cursor timeout, the stream closes and you must re‑open it, potentially losing events that occurred during the pause. Mitigate by keeping the cursor alive or by using a longer timeout.
• Memory pressure: on workloads with >10 k writes per second, the cursor may approach the server’s memory limit. Monitor db.serverStatus().metrics.cursor and set up alerts for high cursor memory usage.
• Token durability: loss of the resume token results in duplicate processing or gaps. Store the token in a transactional write or a replicated collection to guarantee durability.

Summary

A minimal replica set provides the necessary data trust boundary for MongoDB Change Streams. By creating the stream on the primary, handling primary step‑downs, managing cursor timeout, and persisting the resume token, you can build a reliable real‑time pipeline with only two nodes. Operational checks such as replica set status, cursor timeout, and memory monitoring keep the system healthy, while the concrete example demonstrates end‑to‑end verification without external services.