Seamless In‑Place OTP Application Upgrade with release_handler

Desired Outcome

Upgrade a live Erlang OTP node from release 1.0 to 2.0 without stopping the node or losing any client connections. After the upgrade, all running processes should continue executing with the new BEAM binaries and any state‑ful modules must migrate cleanly.

Prerequisites

• OTP 26.x or later – release_handler is part of OTP since R16B.
• Both releases share the same application name and cookie. The .app files in releases/1.0 and releases/2.0 must list the same applications and modules.
• All modules that maintain state must implement the -module attribute with a code_change/3 callback. This function is invoked automatically during the upgrade.
• The node must be started with the same name and cookie as the running instance. You can verify the name with erlang:system_info(node) and the cookie with erlang:system_info(cookie).
• Write the new release to a directory that mirrors the old one: /opt/app/releases/2.0 contains the same ebin and releases/vsn files as /opt/app/releases/1.0.

Focused Procedure

1. Prepare the new release
   • Compile the new code:

     cd /opt/app
     rebar3 release --overwrite
     

   • Verify that releases/2.0/ebin/*.app exists and matches the old release’s module list.
2. Start the node with the new release directory as the current code path
   • From the running node’s console or via SSH, run:

     erl -name myapp@127.0.0.1 -setcookie MYSECRET -pa /opt/app/releases/2.0/ebin -s myapp start
     

   • Use -pa to prepend the new ebin so the loader finds the updated binaries first.
3. Trigger the upgrade
   • In the node’s shell, call:

     release_handler:upgrade({release, "2.0"}).
     

   • The function blocks until all processes have executed their code_change/3 callbacks. A long‑running process may delay the upgrade; monitor the console for progress messages.
4. Verify the upgrade
   • Check the log for the success message:

     release_handler: upgraded successfully
     

   • Run observer:start() and confirm that all process trees still exist and that the module binaries point to releases/2.0/ebin.

     code:which(myapp_worker).
     

   • Run erlang:system_info(release) to see the current release version.
5. Persist the new release
   • Once the node is stable, delete the old release directory or move it to a backup location.

     mv /opt/app/releases/1.0 /opt/app/releases/backup_1.0
     

Expected Checks

• All processes should still be alive after the upgrade. Use erlang:processes() or observer.
• Modules that implement code_change/3 must return {ok, NewState} or {error, Reason}. A return of error aborts the upgrade and rolls back automatically.
• The node’s load average should not spike significantly during the upgrade. If it does, consider splitting the upgrade into smaller steps or using release_handler:upgrade/1,2 with a force flag.
• After the upgrade, run erl -noshell -eval "code:which(myapp_worker)." -s init stop to ensure the module resolves to the new path.

Relevant Recovery Options

• Abort on error: If release_handler:upgrade/1 returns {error, Reason}, the node automatically rolls back to the previous release. Check the console for the error message.
• Manual rollback: If the node hangs or a process crashes during the upgrade, stop the node with init:stop(), then restart it pointing to the old release directory:

  erl -name myapp@127.0.0.1 -setcookie MYSECRET -pa /opt/app/releases/1.0/ebin -s myapp start
  

• Hot code reload only: If the upgrade is only a code change (e.g., a bug fix) and no state migration is needed, you can skip the release_handler step and simply reload the modules:

  code:load_file(myapp_worker).
  

Limitations and Caveats

• Backward compatibility is mandatory. New modules that the old release doesn’t know about will be ignored until the next full restart.
• Stateful processes must implement code_change/3. Failure to do so will result in crashes or inconsistent state.
• The upgrade blocks the node until all processes finish migrating. A single long‑running process can delay the entire upgrade.
• If the new release has a different OTP version or different application dependencies (e.g., a new library version), the upgrade may fail with a badmatch error.

Practical Check After Upgrade

Run the following one‑liner to confirm that the node is using the new release and that a key module is loaded from the new directory:

erl -noshell -eval "io:format(\"Release: ~p~nModule path: ~p~n\", [erlang:system_info(release), code:which(myapp_worker)])." -s init stop

If the output shows Release: "2.0" and the module path ends with releases/2.0/ebin/myapp_worker.beam, the upgrade succeeded.

Conclusion

Using OTP’s release_handler module, you can replace a running node’s code with a newer release while keeping the node alive. The key to a smooth upgrade is ensuring backward compatibility, implementing code_change/3 for stateful modules, and verifying the node’s state post‑upgrade. When in doubt, perform a dry‑run on a staging node before applying the upgrade to production.