Batch importing large datasets in Neo4j: when to use explicit transactions

The problem: slow bulk imports with auto‑commit

When you load hundreds of thousands of nodes or relationships with Cypher, each statement is committed immediately in auto‑commit mode. This generates a round‑trip to the server for every CREATE or MERGE, and the underlying lock manager acquires and releases locks repeatedly. The result is higher latency and more contention, especially on indexed properties.

Thesis: batch statements inside an explicit transaction

Using the driver’s session.beginTransaction() (or the equivalent in the Neo4j Browser with :begin) lets you group many Cypher statements into a single transaction. The commit happens only once per batch, reducing network overhead and lock churn. For imports of 10 k–50 k nodes per transaction you can typically see a 30‑70 % speed‑up compared with statement‑level auto‑commit.

Worked example: importing 100 k nodes with a 5 k‑node batch

The following pseudo‑code shows the pattern in a Java‑like driver (the same idea applies to Python, JavaScript, Go, etc.). Replace the placeholders with your actual values.

// Assume driver is already created and you have WRITE permission
try (Session session = driver.session()) {
    int batchSize = 5_000;
    int processed = 0;
    Transaction tx = session.beginTransaction();
    try (BufferedReader br = Files.newBufferedReader(Paths.get("nodes.csv"))) {
        String line;
        while ((line = br.readLine()) != null) {
            // CSV format: id,name
            String[] parts = line.split(",");
            long id = Long.parseLong(parts[0].trim());
            String name = parts[1].trim();
            tx.run(
                "MERGE (n:Node {id: $id}) SET n.name = $name",
                parameters("id", id, "name", name)
            );
            processed++;
            if (processed % batchSize == 0) {
                tx.commit();   // commit the batch
                tx = session.beginTransaction(); // start next batch
            }
        }
        // commit any remainder
        tx.commit();
    } catch (Exception e) {
        if (tx != null && tx.isOpen()) {
            tx.rollback(); // ensure partial batch is undone
        }
        throw e; // re‑throw after rollback
    } finally {
        if (tx != null && tx.isOpen()) {
            tx.rollback();
        }
    }
}

Key points:

• Each tx.run adds work to the transaction but does not hit the network until commit().
• The catch block guarantees a rollback on failure, preventing a half‑committed batch.
• Monitor memory usage while the transaction is open; a batch of 5 k nodes usually stays well below the default dbms.memory.transaction.max_size (typically 1 GB).

Trade‑off and limitation

Larger batches reduce commit overhead but increase the amount of data held in memory and the length of time locks are held. If a transaction exceeds the configured dbms.transaction.timeout or consumes too much heap, you may see OutOfMemoryError or delayed visibility for concurrent readers. A practical safeguard is to start with a modest batch size (e.g., 5 k nodes) and increase it only after checking :sysinfo or the Prometheus metric neo4j_transaction_memory_usage stays comfortably under the limit.

Actionable closing

1. Set up a test import with a known dataset (e.g., 100 k nodes from nodes.csv).
2. Run the import once with auto‑commit (default driver behavior) and record the wall‑clock time.
3. Run the import again using the explicit‑transaction batch pattern above, starting with a 5 k‑node batch.
4. Compare the times and check :sysinfo for transaction size and lock counts.
5. If memory or lock metrics look high, halve the batch size; if they are low and you still have headroom, try increasing the batch size to 10 k or 20 k nodes.

By matching batch size to your hardware and index density you can reap the throughput benefits of explicit transactions while keeping the system stable.