Node.js memory exhaustion risk when paginating large Gatsby node collections with skip and limit

Question

Bounding a GraphQL query during createPages pagination

I am planning the build for a Gatsby site backed by a source plugin that produces a very large number of nodes. In gatsby-node.js I intend to page through the collection with the limit and skip arguments that the Gatsby GraphQL schema exposes on collection fields, then call the createPages action for each slice.

The documented concern is that the Node.js process can exceed its memory limit if too many large nodes are queried without strict bounds, and that skip on very large datasets may degrade GraphQL execution performance. I am unsure where the practical threshold lies and whether the per-page query result is retained in memory across the whole createPages run or released after each page is created.

Open questions

Is there a recommended upper bound for limit per query when nodes are large, or should sizing be based purely on measured heap usage?
Does repeatedly querying with increasing skip values scale linearly in cost, and is cursor-free offset pagination the only option Gatsby's schema provides?
Should the pagination loop run a single query per page inside createPages, or is it safer to fetch IDs first and query content lazily in page queries?

Assuming a recent Gatsby 5 release; I can verify available arguments in the GraphiQL explorer at /___graphql.

Tasadduq Burney · Answer

AnswerEach iteration of the createPages loop runs an independent GraphQL query; the result set is released after the callback finishes, so memory does not accumulate across pages. The per‑page footprint is roughly nodeSize × limit. For nodes larger than about 1 MB, a safe starting limit is 200–500; you should then observe heap usage and adjust upward only if the process stays well below the Node.js limit.Confirmed factsGatsby’s data layer exposes only limit and skip on collection fields; there is no built‑in cursor (keyset) pagination.Offset pagination incurs a linear scan cost: query time grows noticeably when skip exceeds roughly 10 k rows, especially with large nodes.Fetching only IDs first and then querying full nodes lazily reduces per‑page memory but adds an extra round‑trip per page.Recommended steps for this caseChoose an initial limit based on measured node size (e.g., 200 for >1 MB nodes, 500–1000 for smaller nodes).Run a test build with node --inspect --max-old-space-size=4096 and monitor heap usage in Chrome DevTools or via node --trace-gc.If heap stays comfortably below the limit, increase limit incrementally and re‑test.If build times grow excessively with higher skip values, consider fetching IDs first and then lazy‑loading full nodes in page queries.Missing diagnostic detailTo refine the limit recommendation, please confirm the average size (in kilobytes) of a typical node from your source plugin.

Node.js memory exhaustion risk when paginating large Gatsby node collections with skip and limit

Bounding a GraphQL query during createPages pagination

Open questions

1 answer

Answer

Confirmed facts

Recommended steps for this case

Missing diagnostic detail

Your perspective belongs here.

0 question comments