Memory footprint of stream_code during lazy transformations

Answer

When using stream_code for lazy transformations, memory stays bounded by the stream’s internal highWaterMark. Each transform holds only the current chunk and its callback; once the callback returns, the chunk reference is dropped, making it eligible for garbage collection. As long as backpressure is honored (the writable side stops accepting data when its buffer reaches highWaterMark and resumes on the drain event), the buffer never grows beyond that limit, preventing OOMs.

Likely Explanation

• Each Transform stream keeps a single chunk in memory. The chunk is passed to _transform, processed, and pushed downstream; the reference is dropped immediately after callback() resolves.
• Backpressure is propagated automatically when streams are chained with .pipe() or pipeline(). The writable side’s write() returns false when the internal buffer exceeds highWaterMark, causing the readable side to pause.
• If the consumer ignores backpressure (e.g., buffers all data in an array or never listens to drain), the internal buffer can grow unbounded, leading to OOM.
• Calling stream.destroy() after the end event clears any remaining buffers immediately, helping the GC reclaim memory sooner.

Confirmed Facts

• Default highWaterMark is 16 KB in many Node.js stream implementations; it can be overridden per stream.
• Once a chunk’s callback returns, its reference is removed, allowing the GC to reclaim it in the next cycle.
• Explicit .destroy() frees internal buffers immediately.

Practical Steps for Your Pipeline

1. Set a low highWaterMark (e.g., 8 KB) on each transform if you expect very large chunks.
2. Use pipeline() or chained .pipe() calls to ensure backpressure is automatically propagated.
3. In the consumer, listen to the drain event before writing more data. If you’re reading from a file, use readable.read() in a loop and pause when write() returns false.
4. After the end event, call stream.destroy() on all streams in the chain to clear any lingering buffers.
5. Profile memory with a heap snapshot tool (Chrome DevTools, VisualVM, or Python memory_profiler) to confirm that the heap size stabilizes after each chunk.

Missing Diagnostic Detail

To fine‑tune the recommendation, could you confirm whether the consumer side is listening to the drain event (or its equivalent) when the writable side’s buffer is full?