MongoDB Change Streams: When to Build Your Event Pipeline on the Database

You need to react when data changes in MongoDB — sync a search index, send a notification, feed a cache invalidation. The usual options are polling the collection on a timer or publishing events from your application code. Both have real costs: polling adds latency and load, and dual writes (database plus message broker) can silently diverge when one succeeds and the other fails. Change Streams offer a third path: the database itself emits an ordered, resumable feed of every write. This post covers what that actually buys you, where it breaks down, and how to validate it before committing.

What Change Streams actually give you

A change stream is a cursor over the replica set oplog — MongoDB's internal write-ahead log — exposed through a normal driver API. You open it against a collection, a database, or an entire deployment, and you receive an event for each insert, update, replace, and delete, in the order they were committed. Because it reads the oplog rather than a separate log, the events reflect what the database actually persisted, not what your application intended to write. That eliminates the dual-write divergence problem by construction.

The feature has existed since MongoDB 3.6 and matured meaningfully since: version 4.0 added deployment-wide streams, 4.2 added startAtOperationTime, and 4.4 added pre- and post-images so an update event can carry the document as it looked before the change. The examples below assume 4.2 or later; check the version-specific manual for exact syntax.

Two properties matter most for engineering decisions:

• Resume tokens. Every event carries a token. Persist it, and after a crash or deploy your consumer restarts exactly where it left off — no missed events, no duplicates beyond your own at-least-once handling.
• Server-side filtering. You can attach an aggregation pipeline to the stream so only matching operations and fields cross the network.

A worked example: a filtered, resumable consumer

Suppose you keep an Elasticsearch index in sync with an orders collection, and you only care about status changes. Run this in a Node.js service using the official MongoDB driver (any recent 4.x/5.x/6.x driver; the API is stable):

const { MongoClient } = require('mongodb');

const pipeline = [
  { $match: {
      operationType: { $in: ['insert', 'update', 'replace'] },
      $or: [
        { 'updateDescription.updatedFields.status': { $exists: true } },
        { operationType: 'insert' }
      ]
  } },
  { $project: { fullDocument: 1, operationType: 1, ns: 1 } }
];

async function run() {
  const client = await MongoClient.connect(process.env.MONGO_URI);
  const coll = client.db('shop').collection('orders');
  const tokens = client.db('ops').collection('resumeTokens');

  const saved = await tokens.findOne({ _id: 'orders-es-sync' });
  const options = { fullDocument: 'updateLookup' };
  if (saved) options.resumeAfter = saved.token;

  const stream = coll.watch(pipeline, options);
  for await (const event of stream) {
    await upsertIntoSearchIndex(event.fullDocument); // your code
    await tokens.updateOne(
      { _id: 'orders-es-sync' },
      { $set: { token: event._id } },
      { upsert: true }
    );
  }
}
run().catch(console.error);

Three things to notice. First, fullDocument: 'updateLookup' asks the server to fetch the current document for update events, so your consumer doesn't need a second query. Second, the resume token is stored after the downstream write succeeds — that ordering is what makes the consumer at-least-once rather than at-most-once. Third, the pipeline filter runs server-side, so a busy collection where only 2% of writes touch status doesn't flood your consumer with noise.

To verify resume behavior: insert a document, note that the consumer processes it, kill the consumer process, insert two more documents, restart, and confirm exactly those two arrive. Do this on a throwaway replica set before trusting it in production.

The oplog window is your real SLA

Resume tokens are only valid while the corresponding oplog entries still exist. The oplog is a capped collection: old entries are overwritten as new writes arrive. If your consumer is down longer than the oplog window, its token expires and the stream errors out — your only recovery is a full collection rescan to rebuild downstream state.

This makes oplog sizing an operational decision tied directly to your tolerance for consumer downtime. Check the current window on the primary with rs.printReplicationInfo() in mongosh (requires replica set access), which reports the time span the oplog currently covers. If your longest plausible outage — a bad deploy, a weekend incident — is 6 hours, the oplog must comfortably cover more than that at peak write volume. On self-managed deployments you adjust it via the replication oplog size setting; on Atlas it's configurable per cluster tier. Recheck the window after traffic growth; a size that was fine at launch can quietly become inadequate.

Where Change Streams are the wrong tool

• Standalone instances. Streams require a replica set or sharded cluster. There is no workaround; a single-node dev box must run as a one-member replica set.
• Multi-consumer fan-out. A change stream is a cursor, not a queue. Five independent services each need their own stream (and their own resume token bookkeeping), and each stream adds read load on the primaries. If you need many independent consumers or competing-worker semantics, pipe one change stream into Kafka or a similar broker and let consumers attach there.
• Very high write throughput. Every watched write becomes an event. Consumers must handle backpressure; a slow consumer doesn't block the database, but it does fall behind and eventually risks token expiry. Batch downstream writes and monitor lag.
• Pre/post images cost real overhead. Enabling changeStreamPreAndPostImages (4.4+) stores extra document copies, growing storage and oplog traffic. Enable it per collection, only where consumers genuinely need the before-state.
• Schema coupling. Downstream consumers parse your document shape. A field rename is a breaking change for every listener. Treat event payloads as a versioned contract, or make consumers tolerant of missing fields.

A practical decision rule

Reach for Change Streams when you have one to three consumers, you want guaranteed fidelity to committed writes, and your downtime tolerance fits inside an oplog window you're willing to provision. Reach for a dedicated broker when you need fan-out to many consumers, long replay history measured in days, or decoupling from database topology. Many production systems use both: one change stream feeding a broker, combining fidelity with fan-out.

Before committing, run the kill-and-restart test above, measure your oplog window under realistic write load, and confirm your MongoDB version supports the options you need. An afternoon of validation here is much cheaper than discovering token expiration during your first real outage.