Elixir Release Handler: Determining Safe Rollback Path After Upgrade Failure

Answer the Question First

The only documented, safe way to return to the pre‑upgrade code base when a :release_handler callback raises an exception is to terminate the node and restart it with the previous release. OTP does not provide an automatic rollback hook, and purging individual modules can leave the VM in an inconsistent state.

Why a Full Restart is Required

• During an upgrade the VM may load new modules, start processes that depend on them, and the exception can leave some processes in the new code while others remain in the old code.
• OTP 27 introduced :app_upgrade callbacks that run only on successful upgrades; they are not invoked when the upgrade aborts, so they cannot help undo a failure.
• Purging a single module with :code.purge/1 followed by :code.load_file/1 does not guarantee that all references to the new code are removed, especially if any process was started with the new module.

Distinguishing the Scenario

Before proceeding, confirm whether the node is running:

• In a single‑node deployment or as part of a cluster.
• With state stored only in ETS, only in Mnesia, or in both.

These details influence the minimal steps you need to take after a restart.

Minimal Recovery Steps

1. Detect the Failure
   When the upgrade callback throws, the node will terminate automatically. Verify the termination by checking the OTP logs or using ps -ef | grep beam.
2. Stop the Node Gracefully (if still running)
   Run mix release --stop or, if the node survived the exception, mix release --stop to ensure no processes are left in the new code.
3. Restart the Old Release
   Navigate to the directory of the pre‑upgrade release and start it:

   cd /path/to/old_release
   ./bin/app_name start
   

4. Restore Persisted State (if applicable)
   If you had taken a snapshot of ETS tables (e.g., :ets.tab2file/2) or performed a Mnesia backup before the upgrade, load those snapshots back into the restarted node.

Cluster‑Aware Considerations

In a clustered environment, stop and restart all affected nodes in a coordinated fashion to avoid split‑brain situations. After the old release is running again, re‑join the cluster using the normal --join options or by re‑executing the cluster join command.

Missing Diagnostic Detail

To tailor the exact commands you need, please confirm:

• Whether the node is part of a cluster.
• Which state stores (ETS, Mnesia, or both) you rely on.

Knowing these will allow me to suggest the precise snapshot/restore steps for your setup.