PM2 cluster worker crash during shutdown handler: restart or abandon?
0 reputation · 23 Sept 2023, 07:17 UTC
Context
PM2's graceful shutdown sends SIGINT to the master, waits kill_timeout (default 1600 ms), then issues SIGKILL. In cluster mode, workers only receive a graceful notice when the application explicitly handles the 'shutdown' IPC message during pm2 reload; plain pm2 stop does not broadcast that message.
Unresolved behavior
The documentation does not specify what happens if a worker process crashes inside its shutdown handler (for example, an unhandled rejection while closing a database pool) before the master's kill_timeout expires. It is unclear whether PM2 will restart that worker to complete cleanup, leave it dead and continue waiting for other workers, or immediately escalate to SIGKILL for the entire process group.
Goal
Determine the exact policy so that cleanup logic can be written defensively—either by making shutdown handlers idempotent and crash-proof, or by adding external orchestration that guarantees resource release.
Does PM2 restart a worker that exits with a non-zero code during its shutdown handler before the master's kill_timeout elapses? If not, does the master continue waiting for remaining workers or proceed directly to SIGKILL? Which PM2 version introduced or changed this behavior?