Let It Crash: Building Fault‑Tolerant Apps with Erlang Supervision Trees

Why Let It Crash?

In most languages you guard every call with try/catch, if/else, or other defensive checks. That boilerplate can grow faster than the business logic and still leave hidden edge cases. Erlang flips the script: processes are allowed to fail, and the system is designed to recover from those failures automatically. The result is a leaner codebase and a more robust runtime.

OTP Supervision Trees – The Core of Reliability

At the heart of Erlang’s fault tolerance is the supervisor, a special process that monitors a set of child processes (workers). When a child terminates unexpectedly, the supervisor receives a EXIT signal and applies a pre‑defined restart strategy. This separation of concerns keeps the worker code focused on its job while the supervisor handles recovery.

Restart Strategies

• :one_for_one – Only the crashed child is restarted.
• :one_for_all – All children are restarted when any one fails.
• :rest_for_one – The failed child and all children started after it are restarted.
• :simple_one_for_one – Used for dynamically spawned workers; the supervisor keeps a template.

Each strategy is paired with a *restart intensity* configuration: a tuple {MaxR, MaxT} where MaxR is the maximum number of restarts allowed in MaxT milliseconds. Exceeding the limits causes the supervisor itself to terminate, preventing infinite crash loops.

Hands‑On Example – One‑for‑One Supervisor

The following modules illustrate a minimal worker that intentionally crashes and a supervisor that restarts it. The code is written for OTP 27 and assumes you have the Erlang compiler available.

%% worker.erl – a simple worker that crashes on a specific message
-module(worker).
-behaviour(gen_server).
-export([start_link/1, init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]).

start_link(Args) ->
    gen_server:start_link({local, ?MODULE}, ?MODULE, Args, []).

init(Args) ->
    {ok, Args}.

handle_call(_Req, _From, State) ->
    {reply, ok, State}.

handle_cast({crash}, State) ->
    %% Intentionally exit to trigger supervisor restart
    exit(crash).

handle_info(_Info, State) ->
    {noreply, State}.

terminate(_Reason, _State) ->
    ok.

code_change(_OldVsn, State, _Extra) ->
    {ok, State}.

%% my_supervisor.erl – supervisor that restarts the worker
-module(my_supervisor).
-behaviour(supervisor).
-export([start_link/0, init/1]).

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    %% Strategy: one_for_one, max 5 restarts in 10 seconds
    RestartStrategy = {one_for_one, 5, 10000},
    %% Child spec: {ID, StartFunc, Restart, Shutdown, Type, Modules}
    ChildSpec = {worker,
                 {worker, start_link, [[]]},
                 permanent,
                 5000,
                 worker,
                 [worker]},
    {ok, {RestartStrategy, [ChildSpec]}}.

To observe the restart behavior, follow these steps in the Erlang shell (erl):

1. c(worker). – compile the worker module.
2. c(my_supervisor). – compile the supervisor module.
3. my_supervisor:start_link(). – start the supervisor.
4. WorkerPid = whereis(worker). – get the current worker PID.
5. gen_server:cast(worker, {crash}). – send the crash message.
6. Check the shell output: you should see an EXIT log and a new PID for the worker.
7. whereis(worker). – confirm the PID has changed.

All of the above actions require normal user permissions; no superuser rights are necessary. The restart intensity of 5 restarts per 10 seconds protects against runaway crashes. If the worker were to crash more frequently, the supervisor would terminate, allowing you to investigate the root cause.

Trade‑Offs and Practical Limits

• State Loss – A process’s memory is discarded on crash. Persist critical state to disk or a database before crashing, or pass the state to the supervisor via the restart_info mechanism.
• Restart Loops – Misconfigured MaxR/MaxT can lead to a supervisor that keeps restarting a failing child. Monitor the observer or use supervisor:count_children/1 to watch restart rates.
• Complex Hierarchies – Deep supervision trees can hide where a failure originates. Flatten the tree or add intermediate supervisors only when necessary.
• Debugging Overhead – Since crashes are expected, logs can become noisy. Use Erlang’s error_logger filters or third‑party tools like observer to focus on relevant events.

When to Use Let It Crash

Let It Crash shines in environments where:

• Processes are short‑lived or stateless.
• Failures are isolated and do not corrupt shared memory.
• Restarting a process is inexpensive compared to the cost of defensive checks.
• You have a reliable persistence layer for any critical state.

If your application requires complex transaction management or shared mutable state, you may need additional layers (e.g., a database or a distributed lock manager) to preserve consistency across restarts.

Takeaway

By embracing Erlang’s OTP supervision trees and the Let It Crash philosophy, you can write cleaner business logic and let the runtime handle the heavy lifting of fault recovery. A well‑configured supervisor with appropriate restart strategies and intensity limits turns transient failures into self‑healing behavior, making your distributed system more resilient and easier to maintain.