Building a Fault‑Tolerant Erlang Service with a Simple OTP Supervision Tree

Requirements

The service must continue operating despite node crashes, isolate faulty workers so that a failure in one does not cascade, and support hot code upgrades without taking the system offline. These goals translate into three concrete needs:

1. Automatic restart of crashed workers.
2. Containment of failures within a supervision boundary.
3. Visibility into restart frequency to detect pathological loops.

Smallest Suitable Design

A single supervisor using the one_for_one strategy manages one gen_server worker that holds the core business logic. The supervisor is started as part of an OTP application and is linked to the application controller, so that if the supervisor itself terminates the controller can decide to restart the whole application or shut down the node.

Supervisor Specification (Erlang/OTP 26)

-module(my_app_sup).
-behaviour(supervisor).
%% API
-export([start_link/0, init/1]).

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    % MaxR = 5, MaxT = 60 seconds => allow 5 restarts per minute
    SupFlags = #{strategy => one_for_one, intensity => 5, period => 60},
    ChildSpec = #{id => my_worker,
                  start => {my_worker, start_link, []},
                  restart => permanent,
                  shutdown => 5000,
                  type => worker,
                  modules => [my_worker]},
    {ok, {SupFlags, [ChildSpec]}}.

Minimal gen_server Worker

-module(my_worker).
-behaviour(gen_server).
%% API
-export([start_link/0]).
%% gen_server callbacks
-export([init/1, handle_call/3, handle_cast/2, handle_info/2,
         terminate/2, code_change/3]).

start_link() ->
    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

init([]) ->
    {ok, #{}}.

% Example business logic: echo casted messages
handle_cast({echo, Msg}, State) ->
    io:format("Worker echoed: ~p~n", [Msg]),
    {noreply, State};
handle_cast(_Msg, State) -> {noreply, State}.
% Other callbacks can be kept minimal for this example
handle_call(_Req, _From, State) -> {reply, ok, State}.
handle_info(_Info, State) -> {noreply, State}.
terminate(_Reason, _State) -> ok.
code_change(_OldVsn, State, _Extra) -> {ok, State}.

Place both files in an application directory, add them to src/, and include the supervisor in the application resource file (my_app.app) as the mod entry.

Trust and Data Boundaries

The supervisor and its child run on the same Erlang node and share the same cookie, meaning they trust each other implicitly. All external interaction—whether via TCP, HTTP, or distributed RPC—should happen outside the supervision tree. A typical pattern is to have a separate acceptor process (also supervised) that receives raw input, validates it, and then sends a validated message to the gen_server worker. This keeps untrusted data from reaching the supervised core.

Operational Checks

To detect when the supervision loop is stressed, monitor the restart intensity exposed by the supervisor.

• Query restart counts: Run supervisor:which_children(my_app_sup) on the node (requires shell access with the same cookie). The output includes the number of restarts per child.
• Process‑level info: Use erlang:process_info(Pid, recent_stacktrace) to see why a worker terminated.
• Metrics export: Hook a simple Prometheus exporter (e.g., prometheus.erl) that reads supervisor:which_children/1 every 15 seconds and exposes a gauge erlang_supervisor_restarts_total. Alert when the gauge exceeds a threshold (e.g., 4 restarts in the last minute).
• Observer tool: Start observer:start/0 from an attached shell to visualise the supervision tree and see live restart counts.

Example Verification Steps

1. Start an Erlang node with the application: erl -sname test_node -setcookie mycookie -s my_app start.
2. In the node shell, obtain the worker PID: Pid = whereis(my_worker).
3. Simulate a fault: exit(Pid, kill).
4. Check that the supervisor restarted the worker: supervisor:which_children(my_app_sup) should show restarts incremented by 1.
5. Repeat the kill five times within a minute; on the sixth kill the supervisor should terminate (you will see the application controller log a shutdown).
6. Confirm via Observer or the Prometheus gauge that the restart intensity spiked and then dropped after the supervisor exited.

These steps require only shell access to the node; no special privileges beyond the ability to run exit/2 on a process you own are needed. The risk is that repeatedly killing the worker may cause the supervisor to shut down the entire application, which is intentional for testing the intensity limit.

Failure Modes

If the worker crashes more often than the supervisor’s intensity allows within the period, the supervisor itself terminates. Because the supervisor is linked to the application controller, the controller receives a 'EXIT' signal and, according to its mod specification, either restarts the whole application or stops the node. This cascade prevents a tight restart loop from consuming CPU indefinitely.

Other failure modes include:

• Mis‑behaving gen_server callbacks that violate the OTP contract (e.g., returning a bad state from code_change/3) can leave the worker in an inconsistent state after a hot code upgrade.
• Network partitions in a distributed deployment are not detected by local supervision; the node may appear healthy while it cannot reach peers.

When the Design Should Change

Revisit the supervision topology when any of the following conditions arise:

1. Heterogeneous worker types: If you need different restart strategies (e.g., temporary for a cache worker, transient for a connection pool), replace the single one_for_one supervisor with a one_for_all or rest_for_one supervisor that can assign distinct restart values per child.
2. Cross‑node fault tolerance: When the service must survive an entire node failure, move from local supervision to a distributed application using global name registration or a raft‑based consensus layer (e.g., raft library) so that supervisors can be started on peer nodes.
3. Strict isolation of untrusted inputs: Introduce a gateway process that does strict message validation and runs under its own supervisor. The gateway then forwards only sanitized messages to the core gen_server. This adds an extra supervision layer but keeps the core trust boundary small.

Each change should be evaluated against the original requirements: does it improve fault containment, upgrade safety, or observability without adding unnecessary complexity?