Designing a Minimal Neo4j Causal Cluster for ACID‑Compliant Graph Writes

Problem

You need a graph database that guarantees ACID properties for write operations while allowing read‑scale across multiple replicas. A Neo4j causal cluster can provide this, but you must understand where writes are accepted, how data is replicated, and what operational signals indicate healthy behavior.

Requirements

• Atomicity, consistency, isolation, and durability for every transaction that mutates nodes or relationships.
• Read‑scale capability: followers should serve read‑only queries without affecting write throughput.
• Clear separation of write authority (leader) and read‑only access (followers) to simplify trust boundaries.
• Automatic failover when the leader becomes unavailable.

Smallest Suitable Design

A three‑node causal cluster satisfies the requirements with minimal overhead:

• Leader (single): accepts all write transactions via Bolt, persists them to the native storage engine, and forwards the transaction log to followers.
• Two Followers: receive the replicated transaction log, apply it to their local stores, and serve read‑only Cypher queries.
• Communication uses the default Bolt protocol on port 7687; no additional routing layer is required for this basic setup.

Trust and Data Boundaries

Write traffic is confined to the leader node. Followers never execute write transactions; they only apply the replicated log. This enforces a clear trust boundary:

• Clients that require writes must connect to the leader (or use a driver that routes writes to the leader automatically).
• Read‑only clients can be directed to any follower, knowing they will never see uncommitted or partially applied writes.

Operational Checks

Monitor the cluster to ensure the design behaves as expected:

• Transaction commit latency: measure the time from BEGIN to COMMIT on the leader; sustained increases may indicate storage pressure or network congestion.
• Cluster status: run neo4j admin show status on any node to verify the leader role, follower health, and protocol versions.
• Replication lag: compare the lastCommittedTx identifier on the leader with that on a follower; a small, bounded lag is normal.

Failure Modes and Indicators

• Leader unavailability: if the leader stops responding, followers detect the missing heart‑beat and trigger an election. A new leader is elected automatically; clients should experience a brief write‑outage followed by resumption.
• Split‑brain or election storms: frequent leader flips suggest network instability or misconfigured timeouts. Check neo4j.log for LeaderElection messages and verify symmetric network latency between nodes.
• Write pipeline saturation: when the leader’s write throughput exceeds disk or network capacity, the Bolt protocol begins to back‑pressure, causing client timeouts. Look for rising writeQueueSize metrics in neo4j admin show metrics.

Conditions That Would Change the Design

Revisit the three‑node layout if any of the following arise:

• Write volume consistently exceeds the leader’s capacity, requiring sharding or a multi‑leader configuration (not supported natively in Neo4j 5.x causal clusters).
• Regulatory or latency constraints demand that reads be served from a specific geographic region, prompting the addition of regional follower nodes.
• The need for hot‑standby for rapid leader failover without election delay, which could be addressed by adding a passive replica that never serves reads.

Verification Steps

Confirm ACID behavior and causal consistency with the following procedures (run from a machine with network access to the cluster):

1. Atomicity test:

   # Using cypher-shell, replace <LEADER_BOLT> with bolt://<leader_ip>:7687
   cypher-shell -a <LEADER_BOLT> -u neo4j -p <password> <<'EOF'
   BEGIN
   CREATE (p:Person {name: 'Alice'})
   CREATE (p)-[:KNOWS]->(:Person {name: 'Bob'})
   # Force an abort by triggering a constraint violation
   CREATE (p:Person {name: 'Alice'})
   COMMIT
   EOF
   

   If the transaction aborts, query the database for any Person nodes named Alice; none should exist, confirming that no partial data was persisted.

2. Consistency and isolation test:

   # Write on leader
   cypher-shell -a <LEADER_BOLT> -u neo4j -p <password> -c "CREATE (:Counter {value: 1})"
   # Immediately read from a follower (replace <FOLLOWER_BOLT>)
   cypher-shell -a <FOLLOWER_BOLT> -u neo4j -p <password> -c "MATCH (c:Counter) RETURN c.value AS val"
   

   The follower should return either val = 1 (seeing the write) or no rows (if the write has not yet been replicated). It must never return an older value such as null when a counter existed previously.

3. Durability test:
   1. Commit a transaction that creates a node with a unique property.
   2. Power‑off the leader node (simulate a crash).
   3. After the follower is promoted to leader, verify the node still exists.
4. Failover test:

   # On any node, check current leader
   neo4j admin show status
   # Stop the leader process (e.g., sudo systemctl stop neo4j)
   # Wait a few seconds, then re‑run the status command
   neo4j admin show status
   

   The output should show a different node assuming the leader role, confirming automatic election.

Limitations and Practical Checks

• Read‑only queries on followers are eventually consistent; there is no guarantee of monotonic reads across different followers without explicit bookmarking.
• The leader can become a bottleneck under heavy write bursts. Monitor neo4j.admin.show.metrics for write.time and write.queue; if they approach thresholds, consider scaling write capacity (e.g., faster SSD, dedicated network) or revisiting the data model to reduce write volume.
• Network partitions that isolate the leader from a majority of followers will prevent leader election; ensure a stable inter‑node network and consider using a dedicated VLAN or low‑latency interconnect.

To verify that the cluster remains healthy after any change, run the status command and check that:

• Exactly one node reports role: LEADER.
• All nodes report status: LIVE.
• Replication lag metrics stay below a threshold appropriate for your workload (e.g., < 100 ms under normal load).