Implementing Hot Code Upgrades in Erlang OTP Services

The Problem: Updating Services Without Downtime

In high-availability systems, restarting a node to deploy a bug fix or a new feature creates a service gap. Erlang solves this through Hot Code Upgrading (HCU), allowing you to replace the logic of a running process without stopping it. The challenge lies in managing the transition of the process state from the old version of the code to the new version without crashing the process.

Prerequisites

• OTP Release: Your application must be packaged as an OTP release (using rebar3 or reltool) rather than run as a standalone script.
• Supervisor Hierarchy: A supervisor must manage your gen_server to ensure that if a state transition fails, the process is restarted according to a defined strategy.
• Module Versioning: A clear versioning scheme for your releases (e.g., v1.0.0, v1.1.0).

Step 1: Preparing the gen_server for State Transition

To support hot upgrades, your gen_server must implement the code_change/3 callback. This function is called by the OTP framework immediately after the new code is loaded but before the first message is processed by the new version.

-module(my_service).
-behaviour(gen_server).

%% The code_change callback handles state migration
code_change(OldVsn, State, NewData) ->
    case OldVsn of
        undefined -> 
            %% First time the module is loaded
            {ok, State};
        {1, 0, 0} -> 
            %% Migrate state from v1.0.0 to v1.1.0
            NewState = migrate_state_to_v110(State),
            {ok, NewState};
        _ -> 
            {ok, State}
    end.

migrate_state_to_v110(State) ->
    %% Example: Adding a new field to the state record or map
    State #{new_feature_enabled => true}.

Step 2: Defining the Supervisor Strategy

The supervisor ensures the gen_server is monitored. For a single-service upgrade, a one_for_one strategy is typically used. If the code_change function crashes, the supervisor will restart the process using the new code, though this may result in the loss of the current in-memory state.

%% In the supervisor's init/1 function
init(_Args) ->
    ChildSpecs = [
        #{id => my_service,
           start => {my_service, start_link, []},
           restart => permanent,
           shutdown => 5000,
           type => worker}
    ],
    {ok, {#{strategy => one_for_one}, ChildSpecs}}.

Step 3: Executing the Upgrade

Hot upgrades are managed by the release_handler. You must first build a relup file, which contains the instructions for the node on how to transition from the current release to the next.

1. Build the new release: Use rebar3 release to generate the new version tarball.
2. Deploy the relup: Copy the relup file to the releases/ directory of the running node.
3. Trigger the upgrade: Run the following command in the Erlang shell on the target node:

   %% Run as an administrator/owner of the node
   release_handler:upgrade('my_app/v1.1.0').

Verification and Diagnostics

To verify that the upgrade was successful and the state was preserved, perform the following checks:

Check	Command / Method	Expected Result
Current Release	release_handler:which_releases().	The new version (e.g., v1.1.0) is listed as current.
State Persistence	sys:get_state(my_service).	The state contains both old data and the new migrated fields.
Stability	Check logger or sasl reports.	No badarg or crash_report entries related to code_change.

Limitations and Risks

• NIFs and Drivers: Native Implemented Functions (NIFs) cannot be hot-swapped. If you change a C/Rust NIF, you must perform a full node restart.
• State Mismatch: If code_change/3 is omitted or returns an error, the process will crash. This is a critical failure point during deployment.
• Binary Compatibility: Ensure that any external ports or linked-in drivers remain binary compatible across versions.

Rollback Procedure

If the new version exhibits unstable behavior, you can revert to the previous release. This will trigger the code_change logic in the old version of the code to handle the downgrade of the state.

Run the following in the Erlang shell:

release_handler:downgrade('my_app/v1.0.0').

Warning: Downgrading requires that the old version of the code also implements code_change/3 to handle state coming from a newer version, or it will result in a process crash.