Question
Context Storage transition from memory to persistent stores in Node-RED
Neel Maple
0 reputation · 08 Aug 2020, 04:33 UTC
14.5K views1
Loading…
0 reputation · 08 Aug 2020, 04:33 UTC
Node-RED utilizes a Context API to manage state across node, flow, and global scopes. By default, this data is stored in local memory, providing high-speed access for asynchronous message flows.
When transitioning to persistent storage—such as file-based or Redis stores via settings.js—the system introduces I/O overhead for every read and write operation. In environments with high-frequency message flows and concurrent requests, this shift from memory-resident data to external I/O can impact overall throughput.
There is a need to determine the performance boundary where the reliability of persistent state outweighs the latency introduced by I/O operations during concurrent execution.
34775 reputation · 08 Aug 2020, 06:54 UTC
Transitioning Node-RED context storage from memory to persistent stores (file or Redis) shifts the performance profile from nanosecond-scale RAM access to millisecond-scale I/O operations. This transition introduces latency because the Node.js event loop must wait for disk acknowledgments or network round-trips to ensure data durability.
The impact on message throughput is determined by the frequency of context operations. In a memory-resident setup, context.get() and context.set() are non-blocking and extremely fast. When switching to persistent stores:
The "performance boundary" is typically reached when the frequency of context writes exceeds the IOPS (Input/Output Operations Per Second) capacity of your storage medium or the bandwidth/latency of your Redis instance.
To maintain persistence without reverting to volatile memory, use these configuration strategies to minimize I/O blocking:
noderedcontext.set() on every message, use a function node to buffer state changes and write to context at a set interval or after specific conditions.node scope whenever possible. global and flow scopes often involve more overhead and potential contention than node-level context.settings.js.To provide a specific optimization recommendation, what is the average number of context writes per second in your most active flow?
Use comments to ask for clarification. Post a solution as an answer.
6,260 reputation · 08 Aug 2020, 08:40 UTC
To further mitigate the I/O overhead mentioned, it is worth noting that Node-RED's settings.js allows for a hybrid configuration. You can define multiple storage modules and assign them specifically to different scopes (flow vs global). This allows high-frequency, transient state to remain in the memory store while critical configuration or long-term state is mapped to localfilesystem or redis.
When using persistent stores, be mindful that data is typically serialized to JSON. For large objects, the CPU overhead of serialization/deserialization can become a secondary bottleneck alongside disk I/O. To verify the actual impact in your environment:
context.set() and the subsequent message processing step.