Performing a Zero‑Downtime Hot Code Upgrade in Erlang/OTP with Release Handler

Desired Outcome

Replace the running code and configuration of an OTP application while a node is live, without terminating processes or dropping in‑flight messages. After the upgrade, all processes should continue with the new module versions and the system should report the new release number.

Prerequisites

• OTP 25+ (or the specific OTP release you target).
• An application built as a release using relx or OTP’s releases tool. The release directory must contain releases/RELEASE with a RELEASES file.
• Versioned modules: the old and new module files must be compiled with distinct vsn attributes in the .app file.
• An appup file for each application that describes the code change steps. A typical appup contains a list of tuples: {old_vsn, new_vsn, […] }.
• A supervision tree that implements code_change/3 for each process that may be upgraded. If a process doesn’t support code_change, OTP will terminate it during the upgrade.
• Manual migration logic for ETS tables or persistent_term if your application stores state there, as OTP does not move these automatically.
• Root or user permissions to read the release directories and write to the running node’s code path.

Focused Procedure

1. Prepare the new release. Build the new release in a separate directory, e.g. /opt/app/releases/2.0.0. Verify that the RELEASES file lists the new version and that the appup file is present.

   # Build with relx
   $ relx -c relx.config
   

2. Start the old release. Launch the node from the current release directory.

   $ cd /opt/app/releases/1.5.0
   $ ./bin/app start
   

3. Verify the current state. On the running node, check the loaded module and application version.

   1> code:which_is_loaded(my_app_mod).
   2> application:which_applications().
   

   Expected output: module my_app_mod with vsn {1,5,0}, application my_app at version {1,5,0}.
4. Invoke the release handler. From the node shell or a separate terminal, run:

   1> release_handler:upgrade(\"/opt/app/releases/2.0.0\", [], [])
   

   Parameters:
   • ReleaseDir – path to the new release.
   • Options – empty list for default behaviour.
   • AppUpList – empty list lets OTP read appup files automatically.
   The handler will:
   • Load the new code into the code server.
   • Apply each application’s appup instructions.
   • Call code_change/3 on all supervisors and workers.
   • Update persistent_term entries if specified in the appup.
5. Confirm the upgrade. After the call returns with ok, run the same checks as before:

   1> code:which_is_loaded(my_app_mod).
   2> application:which_applications().
   

   You should see vsn {2,0,0} for the module and {2,0,0} for the application.
6. Validate process continuity. Inspect a long‑running process to ensure it is still alive:

   1> whereis(my_worker).
   2> sys:get_state(whereis(my_worker)).
   

   The state should reflect the new code’s structure. If the process crashed, OTP would have restarted it; check the supervisor’s restart count.
7. Check message queues. Ensure no messages are lost:

   1> erlang:process_info(whereis(my_worker), message_queue_len).
   

   The queue length should be reasonable; a sudden spike indicates a backlog during the upgrade.

Expected Checks

• Module Version: code:which_is_loaded(Module) returns the new vsn.
• Application Version: application:which_applications() shows the upgraded vsn.
• Process State: sys:get_state/1 on a worker returns a term matching the new code.
• No Crashes: supervisor:which_children/1 shows all children alive; supervisor:restart_count/1 unchanged.
• Message Queue: Length below a threshold (e.g., < 1000) indicates no backlog.

Recovery Options

Hot upgrades are atomic from the node’s perspective, but you still need fallback plans.

Abort the Upgrade

If the handler reports an error or you notice a problem early, abort:

1> release_handler:abort()

This rolls back to the previous code state. Verify with code:which_is_loaded/1.

Rollback to a Previous Release

Keep a copy of the old release directory. If the upgrade fails, stop the node and start the old release:

$ cd /opt/app/releases/1.5.0
$ ./bin/app start

OTP will load the old code automatically.

Restart Supervisors Manually

If a code_change fails for a particular process, OTP may terminate it. Restart the supervisor to recover:

1> supervisor:restart_child(my_sup, my_worker).

The process will be reinitialized with the new code.

Limitations & Caveats

• External API changes (e.g., new function signatures) break compatibility unless you provide a code_change that adapts existing state.
• ETS tables and persistent_term contents are not migrated automatically; you must write migration logic in the appup or a separate upgrade script.
• If a process does not implement code_change/3, OTP will terminate it during the upgrade.
• Large ETS tables may cause a temporary slowdown; monitor CPU and memory during the upgrade.

Practical Verification Checklist

Check	Command	Expected Result
Module vsn	code:which_is_loaded(my_mod)	vsn {2,0,0}
App vsn	application:which_applications()	my_app {2,0,0}
Process alive	whereis(my_worker)	pid
Process state	sys:get_state(whereis(my_worker))	new structure
Message queue	erlang:process_info(whereis(my_worker), message_queue_len)	< 1000
Restart count	supervisor:restart_count(my_sup)	unchanged

Conclusion

By following the steps above—ensuring proper appup files, implementing code_change/3, and verifying state post‑upgrade—you can safely perform zero‑downtime hot code upgrades in a production Erlang/OTP system. Remember to test the procedure in a staging environment first, and keep a backup of the previous release for quick rollback.