PM2 cluster worker crash during shutdown handler: restart or abandon?

PM2 does not restart a worker that exits with a non-zero code during its shutdown handler, provided the exit occurs during a pm2 reload or pm2 stop operation. If a worker crashes during this window, the master continues waiting for remaining workers to exit or until the kill_timeout expires; it does not immediately escalate to SIGKILL for the entire process group.

The Shutdown Lifecycle

When you trigger a pm2 reload, PM2 transitions the cluster into a state where it expects workers to exit. This is the 'graceful shutdown' phase. In this state, the standard auto-restart policy is effectively suspended because:

• The Signal: PM2 sends a SIGINT (or IPC 'shutdown' message) to workers.
• The Handler: The application begins its cleanup logic. If the worker crashes here (e.g., an unhandled rejection), the process master sees the termination as part of the requested lifecycle change.
• The Timeout: The master waits for the duration of kill_timeout (default 1600ms). Only if workers are still alive after this period does PM2 issue a hard SIGKILL.

Behavioral Summary

To write defensive cleanup logic, assume these confirmed PM2 policy behaviors:

1. No Restart on Crash: Because the master is already in a "stopping/reloading" state, a non-zero exit code is not treated as an unexpected failure that would trigger a restart.
2. Parallel Waiting: If one worker crashes during shutdown, the master remains waiting for the others. It does not abort the entire operation or kill siblings prematurely.
3. SIGKILL Escalation: The kill_timeout is a global timer for the worker group. Once the timer hits, any surviving workers are forcibly terminated.

Recommended Approach for Defensive Orchestration

Since you cannot guarantee the shutdown handler will complete if it crashes, you should implement the following strategies:

• Idempotent Cleanup: Ensure your database pool closures and file handle releases can be called multiple times without side effects, in case the process is interrupted by a SIGKILL later.
• Increase kill_timeout: If your cleanup involves heavy network operations (like flushing a large buffer), increase the timeout in your ecosystem.config.js to prevent the master from SIGKILL before your logic finishes.
• External Monitoring: For critical resources (like distributed locks), use an external TTL (Time-to-Live) so that resources are released even if the PM2 worker crashes and fails to clean up.

Note: This behavior has been consistent across PM2 4.x and 5.x versions, though specific IPC message handling may vary based on how your app listens for the 'shutdown' event.