Question
PM2 Master and Worker Processes: IPC Overhead During High-Frequency Metric Emission
Dev Maple
0 reputation · 12 May 2021, 11:37 UTC
63K views1
Loading…
0 reputation · 12 May 2021, 11:37 UTC
When running Node.js applications in PM2 cluster mode, the master process coordinates worker instances and receives telemetry such as logs and custom metrics via an internal IPC channel.
The goal is to understand whether the volume of IPC messages generated by high-frequency logging or metric emission introduces measurable event-loop lag in the individual workers, even when overall CPU usage stays below saturation thresholds.
Constraints include the default round-robin load balancer, the inability to disable IPC without losing real-time monitoring, and the lack of a documented knob to throttle or batch these messages.
How does the IPC message rate correlate with observed event-loop delay in a worker under concurrent request load? Is there a configuration option to limit or batch IPC traffic while preserving essential process visibility? Are there known work-arounds that separate telemetry transport from the request-handling path?
35275 reputation · 12 May 2021, 20:14 UTC
High‑frequency IPC traffic from workers to the PM2 master can add measurable event‑loop delay even when CPU utilization looks low.
Confirmed facts
pm2
nodejschild_process.send) to transmit logs, custom metrics and internal telemetry as JSON‑serialized messages.NODE_DISABLE_IPC=1) stops all master‑worker communication, breaking PM2’s ability to restart, scale or report status.When workers emit thousands of metric messages per second, the cumulative IPC overhead adds extra work to the worker's event loop. This work appears as increased latency for setImmediate, setTimeout or I/O callbacks, even though overall CPU usage stays below saturation because the overhead is spread across many small tasks rather than a single heavy computation.
pm2 start app.js -i 4).pm2.metrics.point('test', value) inside a setInterval.setImmediate loop that records the difference between expected and actual fire times (or use async_hooks if you need finer granularity).autocannon -c 100 -d 30 http://localhost:3000) while you vary the metric emission rate (low, medium, high).prom-client) exposing an HTTP endpoint instead of emitting via pm2.metrics.point. Compare latency; a lower variance indicates IPC overhead as the dominant factor.PM2 itself does not provide a knob to throttle or batch the internal metrics stream. The only way to reduce IPC frequency is to:
prom-client and scrape them with Prometheus; keep PM2 for process management only.pino-http with pino-destination) that batches logs before sending them via IPC, or write logs to a file and have a separate shipper.To decide whether throttling IPC is necessary, we need to know the actual IPC message size and rate you are observing. What is the average payload size and messages‑per‑second emitted by each worker? This detail changes the recommendation because if the rate is already low (< 100 msg/s) the overhead is negligible; if it is in the thousands, mitigation steps become worthwhile.
Use comments to ask for clarification. Post a solution as an answer.
5,750 reputation · 12 May 2021, 13:14 UTC
PM2’s master process communicates with each worker via Node’s built‑in IPC socket, queuing JSON messages for logs and custom metrics. When workers emit thousands of metrics per second, the serialization and queueing add micro‑tasks that increase the worker’s event‑loop lag, even though CPU usage stays low. To verify, use process.hrtime.bigint() to timestamp each metric send and compute per‑worker delay, then compare latency distributions while varying emission rates under a constant load. No native PM2 flag exists to throttle or batch IPC; the only practical approach is to modify the worker code to batch messages or replace the default logger with a throttled variant.