Zero‑downtime reloads with PM2 cluster mode and graceful shutdown

Problem: Reloading a Node.js service drops in‑flight requests

When you need to push a code change or update environment variables, a naïve restart of a Node.js process kills the worker while it might still be handling HTTP requests. Clients see errors or incomplete responses, which is unacceptable for production services that require high availability.

Thesis: PM2’s cluster mode combined with a graceful‑shutdown hook lets you reload workers without dropping connections, provided you externalize any in‑memory state.

How cluster mode works

PM2 can start multiple worker processes that all listen on the same port. Internally, Node’s cluster module (or PM2’s internal load balancer) distributes incoming connections among the workers. This gives you:

• Load distribution across CPU cores.
• Fault isolation – if one worker crashes, others keep serving.
• The ability to stop and start workers individually while the service continues to accept traffic.

Graceful shutdown mechanism

When you tell PM2 to reload or stop a worker, it sends a SIGINT (or SIGQUIT) to the process. The worker can catch this signal, stop accepting new connections, finish any requests already in progress, then exit. PM2 waits for the worker to exit; if it does not exit within a configurable timeout (default 30 s), PM2 sends a SIGKILL to force termination.

Worked example: configuring an ecosystem file for zero‑downtime reloads

Assume a simple Express app that uses a PostgreSQL client. The app opens a database connection when it starts and must close it cleanly before the worker terminates.

1. Application code (server.js)

const express = require('express');
const { Pool } = require('pg');

const app = express();
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

app.get('/', async (req, res) => {
  const result = await pool.query('SELECT NOW()');
  res.send(`Time: ${result.rows[0].now}`);
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Worker ${process.pid} listening on ${PORT}`);
});

// Graceful shutdown hook
function shutdown() {
  console.log(`Worker ${process.pid} received shutdown signal`);
  pool.end(() => {
    console.log(`Worker ${process.pid} closed DB pool`);
    process.exit(0);
  });
}

process.on('SIGINT', shutdown);
process.on('SIGTERM', shutdown);

2. ecosystem.json

{
  "apps": [
    {
      "name": "api-service",
      "script": "server.js",
      "instances": "max",          // spawn one worker per CPU core
      "exec_mode": "cluster",
      "env": {
        "NODE_ENV": "development",
        "PORT": 3000
      },
      "env_production": {
        "NODE_ENV": "production",
        "PORT": 3000
      }
    }
  ]
}

3. Starting the service

Run the command as a user that can bind to the chosen port (usually no special privileges needed for ports >1024):

pm2 start ecosystem.json --env production

You should see log lines similar to:
[PM2] Starting /path/to/server.js in cluster mode (4 instances)
[PM2] App [api-service] launched (4 instances)

4. Triggering a reload
When you need to apply a code change or update an environment variable, run:
pm2 reload api-service --env production

PM2 will:

Send SIGINT to the first worker.
Wait for that worker to finish active requests and exit (or timeout after 30 s).
Start a fresh worker with the new code/env.
Repeat for each worker until all are replaced.

During this window, the load balancer continues to route new requests to any workers that are still accepting connections, so overall error rates stay at zero.
Verification steps
Log inspection
After issuing pm2 reload, watch the logs:
pm2 logs api-service

You should see each worker emit the shutdown hook message, close the DB pool, and then a new worker start.
Load test while reloading
Using a tool like hey or ab, generate steady traffic and observe error rates:
hey -z 2m -c 10 http://localhost:3000/

In another terminal, trigger the reload. The hey output should show 0 % failed requests and latency staying within normal bounds.
Check worker count
pm2 list

The instances column should match the number of CPU cores both before and after the reload.
Trade‑off and limitation
Cluster mode improves throughput and fault tolerance, but it introduces complexity around shared state:

Any data kept only in a worker’s memory (e.g., local caches, WebSocket connections, in‑memory session stores) becomes private to that worker. After a reload, a new worker will not have that state, leading to inconsistent behavior if your application assumes it is shared.
To avoid this, externalize state to a datastore (Redis, a database, or a dedicated cache) that all workers can access.
If you need sticky sessions (e.g., for WebSocket handshakes), you must enable PM2’s sticky‑session feature or use an external load balancer that supports it.

Another practical limit is the graceful‑shutdown timeout. If a worker cannot finish its work within the default 30 seconds, PM2 will force‑kill it, potentially aborting in‑flight requests. You can adjust this timeout in ecosystem.json with the kill_timeout property, but setting it too high delays the reload and may hinder rapid scaling.
Actionable closing
To achieve zero‑downtime reloads in production:

Wrap your app in PM2’s cluster mode (instances: "max", exec_mode: "cluster").
Add a shutdown handler that closes all external resources (DB pools, message‑broker connections, etc.).
Test the reload under realistic load to verify zero error rates.
Externalize any in‑memory state or design your service to be stateless per worker.
Monitor the kill_timeout and adjust it only after measuring your longest‑running request.

With these steps, you can push updates confidently, knowing that active requests are allowed to complete and your service remains available.