PM2 cluster worker never signals readiness after reload
0 reputation · 14 Aug 2020, 13:37 UTC
0 reputation · 14 Aug 2020, 13:37 UTC
In PM2 cluster mode, a worker that does not emit the ready signal during a pm2 reload causes the reload to hang or roll back. The reload process relies on listen_timeout to detect readiness and kill_timeout to terminate the old worker. However, the interaction between these timeouts and the worker’s shutdown hooks is not fully documented.
Understand the conditions under which PM2 fails to transition to a new worker, and how the kill and listen timeouts influence this behavior.
app.on('listening', …) during reload?listen_timeout be configured to trigger a rollback earlier, or does PM2 always wait the full interval before aborting the reload?When a PM2 cluster worker does not signal readiness during pm2 reload, PM2 does not infer health from CPU or memory. It starts a listen_timeout timer—30000 ms by default in current PM2 releases, though you should verify against your installed version—and waits for one of two IPC events: the Node.js 'listening' event emitted when a net.Server binds, or an explicit process.send('ready') message when wait_ready is enabled. If neither arrives before the timer expires, PM2 marks the replacement worker as failed and rolls back to the previous process set. The timeout is a hard ceiling, not an adaptive check: PM2 will not abort earlier unless the worker exits on its own or kill_timeout forces termination.
The reload lifecycle follows a fixed order:
SIGINT or SIGTERM to the old worker and waits up to kill_timeout for a graceful exit.'listening' or 'ready' within listen_timeout.Shutdown hooks on the old worker affect step 1 only. They do not shorten or extend the readiness window for the new worker.
listen_timeout can be shortened in an ecosystem file or via the CLI. Lowering it makes rollback happen sooner, but it does not create an early-abort path for a worker that is merely slow to initialize.
// ecosystem.config.js
module.exports = {
apps: [{
name: 'api',
script: 'server.js',
instances: 2,
exec_mode: 'cluster',
listen_timeout: 8000,
kill_timeout: 5000,
wait_ready: true
}]
}
You can also override per invocation: pm2 reload api --listen-timeout 8000.
To keep reloads fast while avoiding false failures:
server.listen(), and keep that work asynchronous so the event loop is not blocked.net.Server, no extra code is needed—the 'listening' event fires automatically when the socket binds.wait_ready, send the ready message in the listen callback:const server = require('http').createServer(app);
server.listen(3000, () => {
process.send('ready');
});
kill_timeout; they do not affect the readiness timer for the new worker.To confirm the behavior locally, start a minimal app that never signals readiness and observe the rollback:
pm2 start app.js -i 2 --listen-timeout 3000
pm2 reload app
pm2 logs
You should see the reload stall for roughly the configured interval, then PM2 restores the previous workers. Adding process.send('ready') after listen and enabling wait_ready should make the same reload complete in seconds.
One caution: a very low listen_timeout can cause false failures for applications that legitimately need extra startup time, turning a slow reload into a repeated rollback loop. Size the timeout to your worst-case initialization, not your average case.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.