Heap Space exhaustion during large Meteor.publish result sets

Identifying the Memory Spike

To pinpoint where server memory spikes during a large Meteor.publish, you must instrument the lifecycle of the publication observer. In cursor-returning publications, the heap typically peaks during the initial materialization phase—the window between the observer starting and the ready signal being sent to the client. This is when the server fetches documents from MongoDB, converts them to BSON/JSON, and generates the burst of DDP added messages.

Distinguishing Large Publications from Concurrent Subscriptions

Memory growth from one massive publication differs from growth caused by many small subscriptions in its timing and heap distribution:

• Single Large Publication: Characterized by a steep, rapid increase in old_space usage within a single subscription's initial-add window. Memory remains high if the observer must retain a large diff state to track changes.
• Concurrent Small Subscriptions: Memory growth tracks the number of active subscription IDs. You will see a "sawtooth" pattern of repeated Garbage Collection (GC) cycles across many different subscription contexts rather than one singular spike.

Instrumentation Steps

To capture the precise moment of exhaustion, avoid high-overhead snapshots in production. Instead, use process.memoryUsage() and v8.getHeapSpaceStatistics() at these specific hooks:

1. Observe Start: Log baseline heapUsed and rss.
2. First added Message: Mark the start of the materialization burst.
3. Last added Message / ready: Capture peak old_space usage before the initial burst completes.
4. Subscription Stop: Verify if memory is reclaimed or if a leak persists.

// Example instrumentation logic
const stats = v8.getHeapSpaceStatistics();
const oldSpace = stats.find(s => s.space_name === 'old_space');
console.log(`Heap Used: ${process.memoryUsage().heapUsed}, Old Space: ${oldSpace.used_heap_size}`);

The Role of the Client MergeBox

The client-side MergeBox buffers incoming messages to ensure consistency, but this occurs after the server has already processed and sent the data. Therefore, MergeBox overhead does not contribute to server-side heap exhaustion. To isolate this, compare the server's old_space spike against the client's message backlog; if the server crashes while the client is still receiving the first 10% of documents, the bottleneck is entirely server-side materialization.

Verification Method

To verify the cause, run a controlled test in staging with a restricted heap (e.g., NODE_OPTIONS=--max-old-space-size=512):

• Test A: One subscription for 10,000 documents.
• Test B: 10 subscriptions for 1,000 documents each.

If Test A crashes while Test B survives (despite the same total document count), the issue is the materialization burst of a single large result set.

Missing Diagnostic Detail: Does your publication return a Mongo cursor, or does it manually call this.added() and this.changed()? If manual, the instrumentation must shift from the observer to the server-side arrays used to build the result set.