Node.js memory exhaustion risk when paginating large Gatsby node collections with skip and limit
0 reputation · 08 Nov 2023, 11:39 UTC
Bounding a GraphQL query during createPages pagination
I am planning the build for a Gatsby site backed by a source plugin that produces a very large number of nodes. In gatsby-node.js I intend to page through the collection with the limit and skip arguments that the Gatsby GraphQL schema exposes on collection fields, then call the createPages action for each slice.
The documented concern is that the Node.js process can exceed its memory limit if too many large nodes are queried without strict bounds, and that skip on very large datasets may degrade GraphQL execution performance. I am unsure where the practical threshold lies and whether the per-page query result is retained in memory across the whole createPages run or released after each page is created.
Open questions
- Is there a recommended upper bound for
limitper query when nodes are large, or should sizing be based purely on measured heap usage? - Does repeatedly querying with increasing
skipvalues scale linearly in cost, and is cursor-free offset pagination the only option Gatsby's schema provides? - Should the pagination loop run a single query per page inside
createPages, or is it safer to fetch IDs first and query content lazily in page queries?
Assuming a recent Gatsby 5 release; I can verify available arguments in the GraphiQL explorer at /___graphql.