MongoDB Change Streams: Trading Polling Overhead for Real-Time Events

The Polling Tax

If your application queries a MongoDB collection every few seconds to detect new documents, you are paying for latency and wasted database load. Change Streams flip that model: instead of asking the database repeatedly, you subscribe to a cursor and let the server push change events to your application. The feature has been available since MongoDB 3.6, but it carries infrastructure requirements that are easy to overlook.

How Change Streams Work Under the Hood

Change Streams are built on the oplog (operations log), the same capped collection that replica sets use to replicate writes. Because they rely on this internal replication mechanism, they only work on replica sets or sharded clusters. A standalone mongod instance has no oplog, so the server will reject change stream requests. You also need majority write concern on the writes you want to observe; without it, events may not appear with the durability your application expects.

When you open a change stream, the driver returns a cursor. Each event contains the operation type (insert, update, delete, replace) and the document key. By default, update events include only the _id and a description of what changed. If you need the entire updated document, request fullDocument: 'updateLookup' in the aggregation options. The server performs an additional lookup to fetch the current version, which is convenient but adds latency and load.

Filtering Before the Network

One underused capability is server-side filtering. You can wrap the change stream in an aggregation pipeline using stages like $match and $project. This means you can discard irrelevant events before they cross the network. For example, if you only care about orders that have shipped, a $match stage on the fullDocument field reduces traffic and client-side processing.

A Minimal Node.js Example

The following pattern illustrates a change stream on an orders collection. It assumes a replica set is already running and that your application has network access to all members.

const { MongoClient } = require('mongodb');

async function watchOrders() {
  const uri = "mongodb://user:password@host1:27017,host2:27017,host3:27017/?replicaSet=rs0";
  const client = new MongoClient(uri);
  await client.connect();
  const collection = client.db("app").collection("orders");

  const changeStream = collection.watch(
    [
      { $match: { "fullDocument.status": "shipped" } }
    ],
    { fullDocument: "updateLookup" }
  );

  for await (const change of changeStream) {
    console.log("Change detected:", change.operationType);
    // Process change.fullDocument here
  }
}

watchOrders().catch(console.error);

Run this script from your application server or any machine with network access to the replica set. The connection string must include all replica set members and the replicaSet parameter. The script requires a user with the find and changeStream privileges on the app database. After inserting or updating a document in the orders collection, the script should log the operation type. If nothing appears, verify the write used majority write concern and that the document matches the $match filter.

Trade-offs You Cannot Ignore

Change Streams are not free. Because they are powered by the oplog, heavy write throughput enlarges the oplog and increases network traffic between replica set members. If your application generates thousands of writes per second, monitor oplog lag using db.serverStatus() and consider increasing the oplog size with the oplogSizeMB setting during initial replica set configuration.

A change stream cursor can also become invalid if the replica set undergoes a rollback or if the oplog window is exceeded before the cursor resumes. Your application must handle resume tokens and reconnection logic. Another hard limitation: Change Streams do not work on standalone instances. If you are developing locally with a single mongod, you will need to convert it to a replica set or use a managed service that provides replication.

Where to Start

Before you commit to Change Streams in production, measure your write pattern. If writes are infrequent, polling might still be simpler. If you need real-time reactions to inserts, updates, and deletes, Change Streams eliminate the guesswork of polling intervals. Start with a $match filter, enable fullDocument only when you need the whole document, and build resume logic from day one. The payoff is an application that responds to data the moment it changes, not seconds later.