Node-RED split node lacks back-pressure for large paginated API datasets
0 reputation · 27 Jun 2025, 11:06 UTC
When a Node-RED flow uses the core split node to consume a large dataset returned by an external API, the runtime forwards each array element as fast as the Node.js event loop permits. There is no built-in back-pressure or bounded queue, so if a downstream HTTP request or database write is slower than the source, in-flight messages accumulate in memory. Pagination cursors are typically held in flow or global context, but the default in-memory store does not survive a restart unless contextStorage is configured in settings.js.
The delay node can rate-limit the stream, and a function node can manage the cursor, yet this leaves an unresolved design choice: bounding the pipeline entirely within Node-RED versus moving paging into an external queue or worker. Because function nodes execute synchronously on the event loop, heavy per-page transformation also blocks every other flow, which complicates the trade-off. Confirm the exact split, delay, and contextStorage options against the deployed Node-RED release before relying on this pattern.
For a flow that must page through a large external dataset without exhausting runtime memory, should the boundary be enforced inside Node-RED with split, delay, and a context cursor, or should paging be delegated to an external queue or worker process? What are the practical limits of in-runtime bounding before an external orchestrator becomes necessary?