Diagnosing Intermittent Read Latency Spikes from Page Cache Pressure in Neo4j

Problem: Read latency spikes with moderate CPU and rising page faults

In a self-managed Neo4j causal cluster, a common performance degradation pattern involves intermittent p95/p99 read latency spikes while CPU utilization remains moderate. Previously fast read queries suddenly slow down, page cache utilization stays persistently near the configured limit, and OS major page faults per second rise in sync with these latency events. Occasional heap Garbage Collection (GC) pauses often coincide with these spikes.

The primary takeaway is that this pattern indicates the working set (the portion of the data frequently accessed) no longer fits in the page cache, forcing the database to wait on the OS to fault pages in from disk. Page cache is Neo4j's off-heap buffer for database pages; a page fault occurs when a requested page is not in cache. This guide assumes Neo4j 5.x self-managed deployments. Note that metric names and memory defaults differ between 4.x and 5.x, and between self-managed and Aura instances.

Recognizable Condition

• Intermittent p95/p99 read latency spikes without sustained CPU saturation.
• Increased execution time for reads that were historically fast.
• Page cache utilization consistently near the configured limit.
• Spikes in major page faults per second on the host OS.
• Heap GC pauses coinciding with the latency events.

Cause and Diagnostic Table

Observed Signal	Likely Cause	Technical Impact
High page faults, falling cache hit ratio	Working set exceeds page cache size	Hot pages are evicted; reads hit physical disk
Spikes during batch ETL or large scans	Concurrent scans evict hot pages	Scan working set displaces read working set
Latency spikes coincide with full GC	Insufficient heap or heap pressure	Stop-the-world pauses stall page cache access
Cache size at container limit	Misconfigured dbms.memory.pagecache.size	OS cannot allocate memory; high OOM kill risk

Ordered Diagnostic Checks

Check 1: Correlate Cache Pressure with Latency

Run these checks on each core member host. You will need sudo permissions for OS counters and Neo4j admin access for internal metrics.

# Run on host to monitor major page faults
vmstat 1 60

Expected Result: Look for increases in major faults per second that align with read latency spikes. Risk: vmstat is a read-only tool and poses no risk to the system.

Next, query your Prometheus scrape or Neo4j metrics for the page cache hit ratio and page faults per second. Overlay these against read latency percentiles. If latency spikes occur exactly when the hit ratio drops and faults rise, the issue is cache pressure rather than CPU saturation.

Check 2: Inspect Active Work and Query Plans

Connect to a read replica using a user with system procedure permissions to verify current memory allocations.

# Run in Cypher shell
:use system
CALL dbms.listConfig() YIELD name, value WHERE name STARTS WITH 'dbms.memory'

Expected Result: Confirm the current dbms.memory.pagecache.size and dbms.memory.heap.max_size. Risk: This is a read-only configuration check.

Review the query log for long scans, missing indexes, or large ORDER BY and SKIP operations. Identify planner misestimates by comparing planning time versus actual DB hits. For example, if a hot read query shows a NodeByLabelScan with high DB hits despite a filter property that has an index, the planner is ignoring the index, causing unnecessary page evictions.

Fixes Tied to Findings

Increase Page Cache

If the cache hit ratio is low and major faults are high, increase the page cache size. Ensure you reserve enough RAM for the JVM heap and OS overhead to avoid swapping.

# Edit neo4j.conf on each member (Requires Restart)
dbms.memory.pagecache.size=8G
dbms.memory.heap.max_size=4G

Risk: Changes require a restart. In containerized environments, ensure the container memory limit is higher than the sum of page cache and heap, otherwise the OS will trigger an OOM kill. Rollback: Restore previous values in neo4j.conf and restart.

Reduce Scan Pressure

If large scans or ETL jobs are evicting hot pages:

• Add or optimize indexes to reduce the number of scanned nodes.
• Rewrite pagination to use keyset pagination (filtering by the last seen ID) instead of large SKIP offsets.
• Throttle or reschedule batch ETL jobs to off-peak windows.

Tune Heap to Avoid Full GC Pauses

If latency spikes coincide with full GC, adjust the heap size or GC algorithm to minimize stop-the-world pauses. Review debug logs for GC pause durations and memory allocation warnings post-adjustment.

Escalation Criteria

Escalate to vendor support if any of the following occur:

• Persistent high page fault rates remain after a safe page cache increase.
• Repeated Out-Of-Memory (OOM) errors or container kills despite correct settings.
• Data corruption warnings appear in the debug.log.
• Latency degradation occurs across all read replicas simultaneously, indicating a cluster-wide orchestration issue.

Verification and Limitations

This guide assumes page cache is sized independently of the heap and that OS memory limits are not being silently enforced by a container orchestrator. To verify the fix:

1. Monitor Neo4j metrics for a sustained increase in the page cache hit ratio.
2. Compare major page faults per second against read latency percentiles over a full workload cycle.
3. Validate query plans for hot queries to confirm index usage and that estimated row counts match actuals.
4. Review debug.log to ensure GC pause durations have decreased.