Non-zero exit code restart loop in PM2 process manager
0 reputation · 22 Mar 2023, 17:08 UTC
0 reputation · 22 Mar 2023, 17:08 UTC
PM2 is configured to automatically restart Node.js processes that exit with a non-zero code. Because this mechanism re-executes the entry point script without preserving in-memory state, applications performing asynchronous write operations may face challenges regarding data duplication.
If a process crashes after a database write has been initiated but before the operation is fully acknowledged or recorded in a local state, the subsequent PM2 restart will trigger the logic again. PM2 provides max_restarts and restart-delay to manage the frequency of these cycles, but it does not offer transactional guarantees across process lifecycles.
Given these constraints, how can an application distinguish between a fresh start and a PM2-initiated restart to avoid duplicate writes? Is there a documented way to detect the restart trigger within the application logic to ensure idempotency?
34775 reputation · 22 Mar 2023, 23:35 UTC
PM2 exposes a PM2_JSON environment variable containing a restart_count field that increments on each restart. A fresh start shows restart_count: 0; a PM2-initiated restart shows restart_count > 0. However, this mechanism is undocumented, version-dependent, and unreliable in containerized deployments. The robust solution is application-level idempotency (idempotency keys, unique constraints, or deduplication logic) rather than relying on process-manager signals.
On each process launch, PM2 injects a JSON string into process.env.PM2_JSON. Typical structure includes:
{
"name": "my-app",
"pm_id": 0,
"restart_count": 3,
"status": "online",
"exit_code": 1
}
In your entry script you can parse it:
let restartCount = 0;
try {
const meta = JSON.parse(process.env.PM2_JSON || '{}');
restartCount = typeof meta.restart_count === 'number' ? meta.restart_count : 0;
} catch { /* ignore parse errors */ }
const isFreshStart = restartCount === 0;
console.log(isFreshStart ? 'Fresh start' : `Restart #${restartCount}`);
This lets you skip non-idempotent initialization on restarts, but only if the variable survives your deployment pipeline.
pm2 restart, config change).Design the write path to be safely repeatable regardless of how the process started:
hash(userId + action + timestampMinute)) and store it with a unique constraint in the database. Duplicate attempts become no-ops.INSERT ... ON CONFLICT DO NOTHING (PostgreSQL) or INSERT IGNORE (MySQL) with a unique index on the business key.While not a substitute for idempotency, these settings make restart loops observable and bounded:
module.exports = {
apps: [{
name: 'my-app',
script: 'index.js',
max_restarts: 5, // stop after 5 crashes
restart_delay: 5000, // 5s backoff between attempts
min_uptime: 10000, // must stay alive 10s to reset counter
watch: false, // disable in production
env: {
NODE_ENV: 'production'
}
}]
};
With max_restarts set, PM2 will eventually mark the process as errored instead of looping indefinitely, surfacing the root cause in pm2 logs and monitoring.
What is your PM2 version (pm2 --version) and deployment target (bare metal, Docker, Kubernetes, PM2-runtime)? The exact PM2_JSON schema and whether the variable reaches your process depend on both.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.