Processing Large Files in Node.js with Streams and Backpressure

The problem with naive file reads

When a Node.js script needs to process a multi‑gigabyte log or data file, the instinct is often to reach for fs.readFile or fs.readFileSync. Those methods load the entire file into a single Buffer before your code ever sees it. On a system with limited heap, that’s an instant crash regardless of how efficiently you later iterate over the data.

Streams give you chunked I/O — and built‑in backpressure

A readable stream in Node.js doesn’t emit all its data at once. Instead, it emits small Buffer chunks one at a time, and the stream emitter keeps track of how much data its consumer has absorbed. This is backpressure: when the consumer’s internal buffer fills up, the stream automatically pauses production until the consumer calls read() or pipe() again. The fs.createReadStream API is the canonical example. It opens the file, feeds chunks into the event loop, and respects backpressure without you needing to manually call stream.pause(). Note: this stream behavior has been consistent since Node.js 12; later releases have not altered the core backpressure mechanism.

Worked example: piping a read stream with error handling

The safest way to process a large file is createReadStream piped into a writable or transform stream, with an explicit 'error' listener on the source:

const fs = require('fs');
const readline = require('readline');

async function countLines(path) {
  const rl = readline.createInterface({
    input: fs.createReadStream(path, { highWaterMark: 64 * 1024 })
  });
  let count = 0;
  for await (const _ of rl) {
    count += 1;
  }
  return count;
}

// usage: countLines('/data/server-log-2024.txt').then(console.log);

In practice, pipe() shortens this further. The following pattern is idiomatic for line‑by‑line processing without loading the whole file:

const fs = require('fs');
const readline = require('readline');

async function countLines(path) {
  const rl = readline.createInterface({
    input: fs.createReadStream(path, { encoding: 'utf8' })
  });
  let count = 0;
  for await (const _ of rl) {
    count += 1;
  }
  return count;
}

// usage: countLines('/data/server-log-2024.txt').then(console.log);

Notice a few things:

• highWaterMark controls the maximum number of bytes stored in the internal buffer (default 64 KB for read streams). Adjusting it can tune backpressure sensitivity for very fast or very slow consumers.
• The for await loop pulls chunks only as fast as the event loop can handle, so backpressure naturally limits memory use.
• If the stream emits an 'error' event and no listener is attached, Node.js will exit. That’s why attaching an error handler — even a simple one — is non‑negotiable.

A trade‑off: backpressure only works when you consume

Backpressure is not magic. If you attach a listener but never read from the stream — or if you transform it into a fire‑and‑forget pattern without awaiting consumption — the internal buffer grows unchecked and memory limits can still be exceeded. This is especially true in worker threads or background jobs where the process may exit before the stream finishes.

Practical check: run your script against a test file larger than your available heap (e.g., a 200 MB file on a machine with 256 MB heap). Open the Node.js inspector or use process.memoryUsage().heapTotal before and after the stream ends. With proper pipe() or for await consumption, heap growth should stay proportional to highWaterMark, not the file size. If you see growth climbing toward the file size, you’re either not consuming fast enough or you’ve detached the pipeline.

Closing: a concrete habit to adopt

Whenever you need to process a file that might exceed available memory, start with fs.createReadStream and pipe() or readline.createInterface, and attach an 'error' listener before the stream is opened. Verify with a memory check that heap growth stays bounded. That habit keeps your scripts robust across log files, dumps, and datasets of any size.