Choosing an Erlang Supervisor Restart Strategy for Stateful Workers

A crashed stateful worker loses its memory

A gen_server keeps request state in its loop state argument. When the process exits, that state disappears. If a caller uses gen_server:call/2 without a timeout, it can block indefinitely waiting for a reply from a process that no longer exists. The supervisor will start a replacement, but the replacement begins with the state returned by init/1, not the state the crashed worker held.

Takeaway: choose a restart strategy that isolates worker failures, always time out caller requests, and treat process memory as volatile.

Match the strategy to how workers are coupled

OTP offers three supervisor restart strategies. The right choice depends on whether workers share resources or depend on startup order.

Strategy	What restarts	When to use it
one_for_one	Only the child that terminated.	Independent workers with no shared state or startup dependency.
one_for_all	All children.	Children share a resource or must restart together to stay consistent.
rest_for_one	The terminated child plus any child started after it.	Ordered startup where later children depend on earlier ones.

For a pool of independent stateful workers, one_for_one is usually the safest default. A single bad worker cannot force its siblings to restart repeatedly.

Worked example: supervise a counter worker

This example runs locally on one Erlang/OTP node. It does not cover distributed node failure.

%% stateful_worker.erl
-module(stateful_worker).
-behaviour(gen_server).

-export([start_link/0, init/1, handle_call/3, handle_cast/2,
         handle_info/2, terminate/2, code_change/3]).

start_link() ->
    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

init([]) ->
    {ok, #{count => 0}}.

handle_call({get_count, Timeout}, _From, State) ->
    {reply, maps:get(count, State), State};
handle_call(_Request, _From, State) ->
    {reply, unknown, State}.

handle_cast({increment, N}, State) ->
    {ok, State#{count => maps:get(count, State) + N}};
handle_cast(_Msg, State) ->
    {noreply, State}.

handle_info(_Info, State) ->
    {noreply, State}.

terminate(_Reason, _State) ->
    ok.

code_change(_OldVsn, State, _Extra) ->
    {ok, State}.

%% worker_sup.erl
-module(worker_sup).
-behaviour(supervisor).

-export([start_link/0, init/1]).

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    Worker = {stateful_worker, {stateful_worker, start_link, []},
              permanent, 5000, worker, [stateful_worker]},
    {ok, {{one_for_one, 5, 10}, [Worker]}}.

Compile and run in an Erlang shell:

erlc stateful_worker.erl worker_sup.erl
erl -pa .

Where to run: a local development shell. No special permissions are required. Placeholders: Sup is the supervisor reference, Worker is the child pid.

1. Start the supervisor: {ok, Sup} = worker_sup:start_link().
2. Start a child: {ok, Worker} = supervisor:start_child(Sup, []).
3. Call with a timeout: gen_server:call(Worker, {get_count, 5000}). The 5000 is milliseconds; without it, a dead worker blocks the caller.
4. Crash the worker: exit(Worker, kill).
5. Inspect the supervisor: supervisor:which_children(Sup). Expect a new pid different from Worker.
6. Read the new state: sys:get_state(NewPid). Expect the initial map from init/1, not the pre-crash value.

Risk: exit(Pid, kill) is immediate and cannot be trapped. Do not run it against a production worker handling live traffic unless you have drained connections first. Also, {one_for_one, 5, 10} allows five restarts in ten seconds; a worker that crashes during init/1 can exhaust that intensity and shut down the supervisor.

Limits and how to verify recovery

• Persist anything that must survive a restart outside the process heap or process dictionary.
• Avoid trapping exits in ordinary workers unless you need cleanup; trapping changes exit semantics and can block supervisor restarts.
• Tune {maxR, maxT} for transient faults, not permanent bugs.
• Verify in staging by killing the worker and confirming sys:get_state/1 returns init/1 state before deploying.
• This guidance targets local nodes. Remote node loss requires separate connection and supervision handling.

Closing

Start with one_for_one for independent workers, wrap every gen_server:call/2 in a timeout, and treat process state as disposable. If workers share a resource, consider one_for_all or rest_for_one, but test the restart cascade in staging first.