Streaming Large JSON in Node.js Without Blowing the Heap

Problem: huge JSON files crash your service

When a Node.js process calls JSON.parse on a multi‑megabyte payload, the entire string is turned into a JavaScript object in memory. Heap usage spikes to roughly the size of the source plus the object graph, and for files larger than the available V8 heap the process aborts with an OutOfMemoryError.

Thesis: stream the JSON piece‑by‑piece

By using a readable stream and a streaming JSON parser (e.g., the jsonparse package), each token is emitted as it is read, keeping memory usage proportional to the current chunk rather than the whole file.

Worked example: parsing a NDJSON‑style array

Assume a file data.json that contains a single large JSON array of objects:

[{"id":1,"value":"..."},{"id":2,"value":"..."}, … ]

We want to handle each object without loading the whole array.

Setup

1. Install the parser: npm i jsonparse (run in your project directory, requires read/write access to node_modules).
2. Create a file parseStream.js with the following code:

const fs = require('fs');
const { Transform } = require('stream');
const jsonparse = require('jsonparse');

// Create a readable stream from the file
const source = fs.createReadStream('data.json', { encoding: 'utf8' });

// jsonparse emits a value for each completed JSON token
const parser = new jsonparse(true); // true → collect strings

// Transform stream that receives parsed values and yields objects
const collector = new Transform({
  objectMode: true,
  transform(chunk, enc, cb) {
    // chunk is the parsed value from jsonparse
    if (typeof chunk === 'object' && chunk !== null) {
      // Emit each object as it arrives
      this.push(chunk);
    }
    cb();
  }
});

// Wire the pipeline
source.pipe(parser).pipe(collector);

// Process each object
collector.on('data', obj => {
  // Replace with your business logic
  console.log(`Received id ${obj.id}`);
});

source.on('end', () => {
  console.log('Stream finished');
});

Running the example

Run the script with Node.js ≥12 (the jsonparse module works on all active LTS lines):

node parseStream.js

No special permissions are needed beyond read access to data.json. The script does not write to disk, so there is no risk of corrupting the source file.

What to expect

• Memory usage stays low: you can verify with process.memoryUsage().heapUsed before and after the data event; the delta should be only a few kilobytes per object, not the size of the whole file.
• Throughput is limited by the disk read speed and the parser’s tokenisation overhead; back‑pressure is automatically handled because fs.createReadStream pauses when the downstream transform is not ready.

Trade‑off: validation

The built‑in JSON.parse validates the entire document and throws on malformed syntax. Streaming parsers like jsonparse are tolerant; they will emit partial values until a syntax error is encountered, after which they may emit an error event. Therefore you should add a schema validator (e.g., ajv) inside the data handler if data integrity is critical.

Actionable checklist

1. Use streaming JSON parsing for files > 10 MB or when you notice heap growth near the V8 limit.
2. Prefer a well‑maintained streaming parser (jsonparse, stream-json, or native Readable with JSONStream).
3. Add schema validation or sanity checks for each emitted object.
4. Monitor heap with process.memoryUsage() during load tests; aim for heapUsed staying well below heapTotal.
5. Handle the error event on both the source stream and the parser to avoid silent data loss.

By treating JSON as a stream you keep your Node.js service responsive and avoid the dreaded out‑of‑memory crash, while still enjoying the simplicity of an event‑driven processing loop.