Pagination state management with Function node and Context in Node-RED
0 reputation · 07 Jan 2022, 11:14 UTC
A design goal is to process a large dataset in bounded chunks within a Node-RED flow. Data moves between nodes as msg objects and a Function node is intended to apply custom logic for chunking and forwarding results.
State for pagination, such as a cursor or offset, needs to persist across successive messages. The Context API provides node, flow and global scopes with memory-based or external persistent storage. Heavy computation in a Function node can block the Node.js event loop and large objects in memory context can increase memory pressure.
The decision point is which Context scope and storage mode is appropriate for pagination state when multiple messages may be in flight, and how state isolation is maintained across Function nodes in the same flow.
Which Context scope is appropriate for pagination state shared by Function nodes in a single flow? Does using flow context for a cursor introduce risk of unbounded growth with concurrent messages? What behavior should be expected for context persistence when switching between memory and file-based stores?