Minimal Neo4j + GDS Architecture for Community Detection

Problem Statement

You need to run community detection on a graph that contains millions of nodes and relationships. The goal is to keep the deployment simple while ensuring data isolation, predictable resource usage, and reliable job execution. The Graph Data Science (GDS) library inside Neo4j provides a ready‑made algorithm set, but it must be used carefully to avoid memory exhaustion and accidental data exposure.

Requirements

• Neo4j 5.x (latest patch) with the GDS library (1.6+). The core and library versions must match; see the compatibility matrix on Neo4j’s site.
• Graph size: 1–10 million nodes, 10–100 million relationships. The example below uses a synthetic 1 million‑node graph.
• Community detection algorithm: Louvain (or other GDS community detection routines).
• Expose results via REST or Bolt for downstream services.
• Production use requires a commercial GDS license; a free trial is available for testing.

Smallest Suitable Design

The minimal architecture consists of a single Neo4j 5.x instance with GDS installed. All data and GDS jobs run in the same process, eliminating inter‑service network traffic.

# Install GDS from the Neo4j Desktop or download the plugin
# Place the .jar in $NEO4J_HOME/plugins
# In neo4j.conf add:
# dbms.security.procedures.unrestricted=gds.*
# Restart Neo4j

Once the instance is running, create a graph projection in memory for the target subgraph:

# Create synthetic data
CREATE (p:Person)
WITH p, range(0,9) AS i
UNWIND i AS j
MERGE (q:Person {id: toString(p.id + j)})
MERGE (p)-[:FRIENDS]->(q);

# Project the graph for GDS
CALL gds.graph.project(
  'proj',
  'Person',
  'FRIENDS'
) YIELD graphName, nodeCount, relationshipCount;

After the projection, run Louvain to detect communities:

CALL gds.louvain.stream('proj')
YIELD communityId, nodeCount
RETURN communityId, nodeCount
LIMIT 10;

The projection is created in memory (default) and lives as long as the instance is up. If you need persistence, use the gds.alpha.graph.project variant with a storeNodeIds flag, but note the extra disk usage.

Trust / Data Boundaries

• All graph data resides in a dedicated Neo4j database. If you use multiple databases, run GDS only on the one that contains the target graph.
• GDS runs in the same process as Neo4j, so access control is governed by Neo4j’s role‑based authentication. Create a dedicated role for analytics users:

CREATE ROLE analytics;
GRANT READ ON DATABASE graphdb TO analytics;
GRANT EXECUTE ON PROCEDURE gds.* TO analytics;

Users without the gds.* privilege will receive an authorization error when attempting to run a GDS job, preventing accidental data exposure.

Operational Checks

• Job Status Monitoring: Use the GDS job API to poll status and capture logs.

  CALL gds.job.stream('proj.louvain') YIELD jobId, status, progress, error;
  

• Memory Usage: Configure the heap limit to avoid OOM. In neo4j.conf:

  dbms.memory.heap.max_size=8G
  

  Then monitor with:

  CALL dbms.memory.stats() YIELD name, value
  WHERE name CONTAINS 'heap'
  RETURN name, value;
  

• Resource Limits for GDS: Pass maxIterations or concurrency parameters to the algorithm call to cap CPU usage.
• Logging: Check logs/neo4j.log for entries like Job failed or out of memory. GDS writes detailed job logs to the Neo4j log file.

Failure Modes

• Memory Exhaustion: Large in‑memory projections can exceed the configured heap. Watch the heapUsed metric and consider using a gds.alpha.graph.project with a small sample or apply node/relationship filters.
• Algorithm Timeouts: Set a timeout on the algorithm call or run it asynchronously with gds.beta.job.schedule to avoid blocking other operations.
• Schema Mismatches: If the graph lacks the expected relationship type or node label, the projection will fail. Validate existence before projection:

  CALL db.labels() YIELD label
  WHERE label = 'Person';
  

• Data Corruption: Corrupted properties can lead to incomplete community assignments. Run CALL gds.alpha.graph.list() and verify that node and relationship counts match the source graph.
• Unauthorized Access: Users lacking the gds.* privilege can view projection metadata but not execute algorithms. Ensure role separation.

Change Triggers – When to Scale Out

• Graph Size > 1 billion entities: A single instance cannot hold such a projection in memory. Consider a Neo4j Enterprise cluster with sharding or move to a dedicated analytics cluster.
• Multi‑Tenant Isolation: If different teams need isolated analytics environments, separate databases or separate Neo4j instances are required.
• GPU‑Accelerated Algorithms: GDS offers GPU variants for some algorithms. Switching to a GPU‑enabled Neo4j instance (e.g., with NVIDIA GPUs) may require a different deployment strategy.
• High Throughput Read/Write: Running GDS jobs on a production instance can impact normal traffic. In that case, create a read‑replica dedicated to analytics and run GDS there.

Practical Verification Checklist

1. Verify Neo4j and GDS versions match: CALL dbms.components() and CALL gds.version().
2. Run a projection and confirm it appears in CALL gds.graph.list().
3. Execute CALL gds.louvain.stream('proj') and check that the stream is non‑empty.
4. Inspect dbms.memory.stats() before and after the job to ensure heap usage stays within limits.
5. Attempt to run a GDS job as a user without the gds.* privilege and confirm an authorization error.

When all checks pass, you have a minimal, reliable Neo4j + GDS deployment for community detection that can be audited and scaled only when thresholds are exceeded.