MongoDB Change Streams: Architecture Note for Low‑Latency, Exactly‑Once Event Capture

Requirements

The goal is to capture every insert, update, delete, and replace operation on a MongoDB collection (or an entire database) with low latency and to deliver each event exactly once to downstream consumers. The solution must survive transient network glitches, tolerate short pauses in the consumer, and provide a clear recovery path when the oplog window is exceeded.

Smallest Suitable Design

A single‑node replica set (or a sharded cluster with a dedicated mongos) is sufficient to enable change streams. The consumer application opens a change‑stream cursor on the target collection, requests a resume token after each batch, and persists that token durably (e.g., in a file, Redis, or a small metadata collection). The consumer uses the resumeAfter option to restart from the last known token.

# Python example (pymongo>=4.0)
import pymongo
client = pymongo.MongoClient('mongodb://host1:27017,host2:27017/replicaSet=rs0')
db = client['mydb']
coll = db['events']
# Load token from persistent store; None if first run
token = load_token()
pipeline = [{'$match': {'operationType': {'$in': ['insert','update','delete','replace']}}}]
options = {'fullDocument': 'updateLookup', 'maxAwaitTimeMS': 5000}
if token:
    options['resumeAfter'] = token
with coll.watch(pipeline, **options) as stream:
    for change in stream:
        process_change(change)          # application‑specific handling
        store_token(stream.resume_token) # persist after successful processing

This pattern satisfies the core requirements with the fewest moving parts: a replica set provides the oplog, the change‑stream cursor reads from it, and the resume token guarantees exactly‑once delivery as long as the token remains within the oplog window.

Trust and Data Boundaries

The change stream runs under the same authentication as the consumer. To limit exposure:

• Grant the consumer role readAnyDatabase or the more specific changeStream role on the target database.
• Enable network isolation (firewall rules, VPC security groups) so only approved services can reach the MongoDB ports.
• If the consumer runs in a different trust zone, enforce TLS and mutual authentication between the client and the MongoDB nodes.

These boundaries ensure that an compromised consumer cannot read collections outside its permitted scope, and that unauthorized hosts cannot inject or intercept change‑stream traffic.

Operational Checks

Monitoring focuses on three areas: cursor health, token persistence, and server load.

• Cursor latency – Periodically run db.serverStatus().metrics.cursor or enable the MongoDB profiler (db.setProfilingLevel(2)) to capture getMore command latency. A rising latency indicates the consumer is falling behind.
• Resume token storage – Verify that the token is written atomically and can be read back after a consumer restart. A simple checksum (e.g., SHA‑256 of the binary token) stored alongside the token helps detect corruption.
• Oplog pressure – Watch db.serverStatus().opcounters.insert and the oplog size via db.printReplicationInfo(). If the oplog window approaches the expected consumer downtime, increase the oplog size or add more RAM to reduce pressure.

Set maxAwaitTimeMS to a value that balances latency and CPU usage (e.g., 5000 ms). Too low a value causes busy‑looping; too high a value increases event‑delivery latency.

Failure Modes

Network partition – When the primary becomes unreachable, the change‑stream cursor raises NotMaster or NodeNotFoundError. The consumer should catch the exception, pause briefly, and attempt to reconnect; the stored resume token allows recovery once the topology heals.

Transient consumer pause – As long as the pause is shorter than the oplog retention window, the cursor can resume using the last token without missing events.

Oplog window loss – If the oplog is truncated (due to size limit or excessive write volume) before the consumer restarts, the change stream errors with Resume token not found. Recovery requires either a full resync of the collection or rebuilding the consumer state from a snapshot.

In a sharded cluster, events from different shards may interleave; the change stream guarantees order only per shard. If global ordering is required, the consumer must implement application‑level sequencing (e.g., using a monotonic timestamp field).

Conditions That Would Change the Design

The minimal design may need revision when:

1. Exactly‑once across shards is required – Introduce a deduplication layer (e.g., a deterministic event ID stored in a durable table) or switch to a change‑stream‑based connector like MongoDB Kafka Connect that provides exactly‑once semantics.
2. Throughput exceeds oplog capacity – Increase the oplog size, enable oplog throttling, or shard the collection to distribute write load.
3. Multiple collections or databases must be watched – Use a database‑ or cluster‑level change stream and add a $match stage to filter by ns.coll or ns.db as needed.
4. Regulatory audit demands immutable logs – Forward change‑stream events to an append‑only store (e.g., Apache Kafka with log compaction disabled or a write‑once object store) immediately after processing.
5. Latency must be sub‑millisecond – Consider using the MongoDB Wire Protocol directly or a dedicated change‑stream‑optimized driver, and colocate the consumer with the MongoDB primary.

Each trigger should be evaluated against the operational cost, complexity increase, and the specific business need before altering the baseline architecture.