Stream Large JSON Files in Java Without OutOfMemoryError

The Problem: Parsing 1GB JSON Files Kills Your JVM

When processing JSON documents larger than available heap space, ObjectMapper.readValue() loads the entire structure into memory as a tree of objects. This fails with OutOfMemoryError for files exceeding ~50-100MB depending on JVM settings.

The Solution: Jackson Streaming API

Jackson's JsonParser and JsonGenerator process JSON token-by-token, maintaining constant memory usage regardless of document size. Memory stays O(1) instead of O(n).

Working Example: Parse and Extract Array Elements

Run this code with read permissions on your JSON file:

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
import java.io.FileInputStream;

public class JsonStreamExample {
    public static void main(String[] args) throws Exception {
        JsonFactory factory = new JsonFactory();
        JsonParser parser = factory.createParser(new FileInputStream("large.json"));
        
        while (parser.nextToken() != null) {
            String fieldName = parser.getCurrentName();
            if ("items".equals(fieldName)) {
                parser.nextToken(); // Move to first array element
                parseArrayItems(parser);
            }
        }
        parser.close();
    }
    
    private static void parseArrayItems(JsonParser parser) throws Exception {
        while (parser.nextToken() != JsonToken.END_ARRAY) {
            String id = null;
            String name = null;
            
            while (parser.nextToken() != JsonToken.END_OBJECT) {
                String field = parser.getCurrentName();
                parser.nextToken();
                
                if ("id".equals(field)) {
                    id = parser.getText();
                } else if ("name".equals(field)) {
                    name = parser.getText();
                }
            }
            
            // Process extracted data
            System.out.println("ID: " + id + ", Name: " + name);
        }
    }
}

Writing JSON Incrementally

Use JsonGenerator to create large JSON files without holding them in memory:

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import java.io.FileOutputStream;

JsonFactory factory = new JsonFactory();
JsonGenerator generator = factory.createGenerator(new FileOutputStream("output.json"));

generator.writeStartObject();
generator.writeFieldName("data");
generator.writeStartArray();

for (int i = 0; i < 1000000; i++) {
    generator.writeStartObject();
    generator.writeNumberField("id", i);
    generator.writeStringField("name", "Item " + i);
    generator.writeEndObject();
}

generator.writeEndArray();
generator.writeEndObject();
generator.close();

Limits and Common Mistakes

• Manual state management: Streaming requires tracking token state yourself. Missing a nextToken() call or misinterpreting nesting levels causes silent data corruption.
• No forward references: Cannot handle circular structures or forward references that require full document knowledge.
• No automatic schema validation: Validation must be implemented separately; streaming parsers don't validate against JSON Schema.
• Diminishing returns on small files: For documents under ~10MB, ObjectMapper tree model overhead is negligible and development time is lower.
• Error handling complexity: Malformed JSON throws exceptions at the token level; recovery requires careful try-catch placement around parsing loops.

Verification Steps

1. Create a test file >100MB JSON and compare memory usage between ObjectMapper.readValue() and JsonParser streaming approach.
2. Use jvisualvm or similar profiler to monitor heap consumption during both parsing methods.
3. Verify Jackson version 2.12+ includes improved streaming performance and bug fixes.
4. Test with malformed JSON to ensure proper exception handling in streaming code paths.