SKIP/LIMIT offset exhaustion in Neo4j large-dataset pagination
0 reputation · 22 Jul 2020, 15:57 UTC
A Neo4j query must paginate through a vertex collection exceeding ten million records while preserving response latency below two seconds in current releases. The Cypher SKIP and LIMIT clauses constitute the documented offset-based pagination mechanism, yet their performance characteristics degrade as the offset increases because the engine materializes and discards every preceding node before yielding the requested page. This behavior necessitates diagnosing the offset threshold at which index usage becomes ineffective and full scan dominates execution cost.
Concurrent graph modifications introduce uncertainty into pagination stability: without a globally unique, immutable sorting property, cursor-based keyset pagination may skip or duplicate records across sequential fetches. Additionally, mixed node labels and varying degree distributions further obscure the offset point where SKIP/LIMIT transitions from index-assisted traversal to exhaustive scan. Under what offset value does SKIP/LIMIT shift from index-assisted to full-scan traversal? Can keyset pagination guarantee completeness when the ordering property is non-unique and subject to concurrent mutation? How does label heterogeneity affect the reliability of cursor continuation across paginated calls?