PM2 graceful reload vs Node.js SIGTERM handling across versions

Answer to the question

Use a single, version‑agnostic shutdown routine that listens for SIGINT, SIGTERM and SIGHUP. Inside the handler:

1. Stop accepting new traffic – call server.close().
   • On Node 18+ also call server.closeIdleConnections() if the method exists.
   • On Node 12 keep a Set of open sockets and destroy the idle ones manually.
2. Drain in‑flight requests – let the existing callbacks finish.
3. Guard the whole sequence with a hard deadline – set a setTimeout that fires after kill_timeout - 200 ms and calls process.exit(0).
4. When the drain completes before the deadline, call process.exit(0) immediately.

This pattern works on Node 12 (where server.closeIdleConnections is missing) and on Node 18+ (where the method is available). The explicit process.exit() guarantees that the process does not hang if the event loop is still busy when the handler returns.

Why this pattern works across Node 12 and 18+

• PM2’s cluster reload sends SIGINT to workers; listening to SIGTERM alone can be bypassed.
• Node 12’s server.close() does not automatically drop idle keep‑alive sockets, so a manual cleanup is required.
• Node 18+ adds closeIdleConnections() (from 18.2) and improves server.close() to terminate idle sockets (from 19). The handler uses whichever API is present.
• Calling process.exit() after the drain ensures the runtime does not wait for the default termination flow, which would otherwise ignore the handler’s asynchronous work.

Tuning kill_timeout relative to request latency

PM2’s kill_timeout (default ~1.6 s) is the maximum time the orchestrator will wait for a worker to exit before sending SIGKILL. To avoid premature termination:

1. Measure the p99 latency of the slowest request path.
2. Choose a drain deadline of p99 + 1 s (allowing a margin for request processing and socket cleanup).
3. Set kill_timeout to drain_deadline + 500 ms.

Example: if the longest request takes 3 s, set kill_timeout to ~3.5 s. This gives the handler enough time to finish draining and exit cleanly, preventing PM2 from issuing a SIGKILL that would drop the response.

Effect of wait_ready on the signal sequence

The wait_ready flag only affects startup: the worker must send process.send('ready') before PM2 routes traffic to it. It does not alter the shutdown sequence. Therefore the same signal handler is still required for reloads and graceful shutdowns, even when wait_ready is enabled.

Practical steps for your fleet

1. Check the effective kill_timeout and wait_ready values:

   pm2 describe <app> | grep -E 'kill_timeout|wait_ready'

2. Update your ecosystem.config.js to set kill_timeout as calculated above.
3. Add the following shutdown routine to your Node entry point:

   const http = require('http');
   const server = http.createServer(app);
   const sockets = new Set();
   
   server.on('connection', socket => sockets.add(socket));
   server.on('close', socket => sockets.delete(socket));
   
   function gracefulShutdown(signal) {
     console.log(`Shutdown triggered by ${signal}`);
     server.close();
     if (typeof server.closeIdleConnections === 'function') {
       server.closeIdleConnections();
     } else {
       for (const socket of sockets) socket.destroy();
     }
     const deadline = setTimeout(() => {
       console.log('Hard exit after timeout');
       process.exit(0);
     }, process.env.KILL_TIMEOUT - 200);
     server.on('listening', () => {
       // No new connections will be accepted
     });
     server.on('close', () => {
       clearTimeout(deadline);
       process.exit(0);
     });
   }
   
   ['SIGINT', 'SIGTERM', 'SIGHUP'].forEach(sig => process.on(sig, () => gracefulShutdown(sig)));
   

4. Run a test: start the app, trigger pm2 reload, and observe that in‑flight requests finish before the worker exits.

Missing diagnostic detail

To fine‑tune kill_timeout we need the current p99 latency of your application. Please provide the value (in milliseconds) of the longest request that completes under normal load.