Implementing Zero-Downtime Updates with Erlang Hot Code Loading

The Problem: Updating Logic Without Process Restarts

In high-availability systems, restarting a node to apply a patch can cause unacceptable downtime or drop active client connections. The BEAM virtual machine solves this via "Hot Code Loading," allowing you to replace the code of a module while the system is running. However, simply loading a new file is not enough; processes must be designed to "migrate" from the old version to the new one.

Prerequisites

• Compiled BEAM files: The new version of the module must be compiled to a .beam file and placed in the code path.
• State Compatibility: If the module manages state (e.g., via a record), the new version must be able to handle the data structure produced by the old version. Changing a record definition without a migration strategy typically results in badarg errors.
• Fully Qualified Calls: To trigger a version check, processes must call functions using the module:function/arity syntax. Local calls (calling a function within the same module without the module prefix) bypass the BEAM's version check and will keep the process locked into the old code version.

Procedure for Hot Loading

1. Deploy the New Beam File

   Place the updated .beam file in the directory where the VM expects modules. If you are using a release, this is typically handled by the release tool, but for manual testing, ensure it is in the code_path().

2. Load the Module

   Run the following command in the Erlang shell (or via a remote shell) to load the new version into memory:

   code:load_file("beam_file:my_module").

   The BEAM now holds two versions of the module: current and old.

3. Trigger Process Migration

   Processes currently executing a function will finish that function using the old version. The next time they make a fully qualified call to that module, the VM will switch them to the current version.

4. Purge the Old Version

   Once you have verified that all processes have migrated, remove the old version from memory to reclaim space:

   code:purge(my_module).

   This returns true if the old version was successfully removed. If it returns false, some processes are still executing the old code.

Implementation Example: The Hot-Upgradable Loop

To ensure a process updates its logic, the recursive loop must call itself using a fully qualified name. In this example, we change the increment value from 1 to 10.

Version 1 (Initial)

-module(counter).
-export([start/0, loop/1]).

start() ->
    spawn(counter:loop, [0]).

loop(Count) ->
    receive
        increment ->
            NewCount = Count + 1,
            io:format("Count is ~p~n", [NewCount]),
            counter:loop(NewCount); %% Fully qualified call triggers update
        get_val ->
            io:format("Current: ~p~n", [Count]),
            counter:loop(Count)
    end.

Version 2 (Updated)

Change NewCount = Count + 1 to NewCount = Count + 10, recompile, and load.

-module(counter).
-export([start/0, loop/1]).

start() ->
    spawn(counter:loop, [0]).

loop(Count) ->
    receive
        increment ->
            NewCount = Count + 10, %% Updated logic
            io:format("Count is ~p~n", [NewCount]),
            counter:loop(NewCount);
        get_val ->
            io:format("Current: ~p~n", [Count]),
            counter:loop(Count)
    end.

Diagnostics and Verification

To verify the update without relying on output logs, use the following checks in the Erlang shell:

• Check Module Path: code:which(my_module). returns the path to the current version of the module.
• Verify Migration: If a process is stuck in the old version, code:purge(my_module) will return false.
• Check Process State: Use erlang:process_info(Pid, current_function) to see which function the process is currently executing.

Recovery and Rollback

If the new code introduces a bug, you can roll back by loading the previous .beam file:

1. Load the original version: code:load_file("beam_file:my_module_v1").
2. The VM marks the buggy version as old and the original as current.
3. Processes will revert to the original logic upon their next fully qualified call.

Limitations

• Local Calls: If you call loop(NewCount) instead of counter:loop(NewCount), the process will never migrate to the new code until it is killed and restarted.
• Blocking Calls: A process blocked on a receive or a sleep will not migrate until it wakes up and makes a new fully qualified call.
• Memory: Keeping two versions of a module increases memory usage until code:purge/1 is called.