Let It Crash: Building Resilient Erlang Services with Supervision Trees

The problem: a single faulty worker can bring down a node

In a typical Erlang application, many small, lightweight processes handle business logic. If one of them receives malformed data and throws an exception, the entire node can terminate unless the error is isolated. Defensive programming—adding guards and try/catch blocks everywhere—quickly turns the codebase into a maze of error handling and obscures the core logic.

Thesis: let the worker crash and let a supervisor restore it

Erlang’s “Let it Crash” philosophy moves error handling out of the worker and into a dedicated supervisor process. A supervisor watches its children, detects abnormal termination, and restarts them according to a strategy. This keeps the rest of the system running while still allowing the worker to fail fast.

How supervision trees work

A supervisor is a process that implements the supervisor behaviour, not a gen_server. It defines a list of child specifications and a restart strategy:

• one_for_one – only the crashed child is restarted.
• one_for_all – all children are stopped and restarted.
• rest_for_one – the crashed child and any that started after it are restarted.

Each child spec includes the module, start function, restart type (permanent, transient, temporary), shutdown timeout, and child type (worker or supervisor). If a child exits with a reason other than normal, shutdown, or {shutdown, _}, the supervisor counts a restart. If the number of restarts exceeds max_restarts within max_seconds, the supervisor terminates itself, propagating the failure upward.

Worked example: a crashing worker and its supervisor

Below is a minimal application that demonstrates a worker that crashes on bad input and a supervisor that restarts it. The example uses rebar3 to create the project.

# rebar3 new app crash_demo
cd crash_demo

Worker module src/crash_worker.erl:

-module(crash_worker).
-behaviour(gen_server).

%% API
start_link() -> gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

init([]) -> {ok, #{}}.

handle_call({parse, Json}, _From, State) ->
    case Json of
        <<"{\"ok\":true}">> -> {reply, {ok, parsed}, State};
        _ -> %% pattern match failure will crash the process
            {error, badarg}
    end.
handle_cast(_Msg, State) -> {noreply, State}.
handle_info(_Info, State) -> {noreply, State}.
terminate(_Reason, _State) -> ok.
code_change(_OldVsn, State, _Extra) -> {ok, State}.

Supervisor module src/crash_sup.erl:

-module(crash_sup).
-behaviour(supervisor).

start_link() -> supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    RestartStrategy = {one_for_one, 5, 10}, % 5 restarts / 10 seconds
    ChildSpec = {
        crash_worker,
        {crash_worker, start_link, []},
        permanent,
        5000,
        worker,
        [crash_worker]
    },
    {ok, {RestartStrategy, [ChildSpec]}}.

Application callback src/crash_demo_app.erl:

-module(crash_demo_app).
-behaviour(application).

start(_StartType, _StartArgs) -> crash_sup:start_link().
stop(_State) -> ok.

Update src/crash_demo.app.src to list the modules and register the application. Build and run:

rebar3 compile
rebar3 shell

In the Erlang shell:

application:start(crash_demo).
% Normal request
gen_server:call(crash_worker, {parse, <<"{\"ok\":true}">>}).
% Malformed request – the worker will crash
gen_server:call(crash_worker, {parse, <<"bad">>}).

After the malformed call, you’ll see the worker terminate and the supervisor immediately start a fresh instance. Subsequent normal calls succeed again, proving the restart loop works.

Trade‑off and limitation

Supervisors are great for transient faults, but they can mask permanent problems. If a worker keeps crashing on the same input, the supervisor will keep restarting it until the intensity limits are hit, at which point the supervisor itself terminates. Therefore:

• Use meaningful restart strategies (e.g., transient for workers that should only restart on abnormal exits).
• Set max_restarts and max_seconds to reasonable values that allow quick recovery but prevent infinite loops.
• Log each termination reason with error_logger or a monitoring tool so you can investigate recurring failures.

Actionable closing

When building Erlang services, isolate business logic into small processes, let them fail fast, and supervise them with a clear strategy. Keep the supervisor definition simple, set sensible intensity limits, and ensure termination reasons are logged. This pattern keeps the node alive while allowing individual workers to recover automatically, embodying the “Let it Crash” principle.