Diagnosing Erlang OTP Supervisor Restart Cascades

The problem and what to do first

An Erlang node that repeatedly logs SUPERVISOR REPORT entries is usually not suffering from one broken worker. It is suffering because the supervisor above that worker is treating a local fault as a shared one. When the wrong restart strategy is chosen, a single crashing gen_server can drag down its healthy siblings, then its parent, and eventually the whole node.

Takeaway: match the restart strategy to the failure domain. Use one_for_one for independent workers, reserve one_for_all for children that genuinely depend on each other, and set max_restart and max_seconds so a broken child stops restarting instead of looping forever.

Terms used below: BEAM is the Erlang virtual machine. OTP is the standard library and design framework that includes supervisors. A supervisor is a process that starts, watches, and restarts other processes. A child specification is the data structure that tells a supervisor how to start a child and whether to restart it. Restart intensity is the number of restarts allowed inside a time window. A link is a bidirectional bond that propagates exit signals; a monitor is a one-way watch that does not.

Recognizable condition

You are likely in a restart cascade when you see one or more of the following:

• The error log contains repeated SUPERVISOR REPORT lines with child_terminated or restarted for the same child.
• A child starts, exits, and restarts within seconds, over and over.
• Healthy sibling processes terminate even though their own code did not fail.
• The supervisor itself exits with reason shutdown, taking its parent and the rest of the tree down with it.

Cause and diagnostic table

Observed condition	Likely cause	Where to check
Only the failed child restarts, but it never stays up	Child is permanent and keeps crashing on startup	Child specification restart field; crash logs
All siblings restart when one child dies	Strategy is one_for_all or children are linked	Supervisor init/1 return value; link or monitor usage
Supervisor terminates after several quick restarts	Restart intensity limit exceeded	Supervisor strategy tuple; SASL or error log
Unrelated processes die across the tree	Links propagate exits to parent and other supervisors	Process links; erlang:process_info/2

Ordered checks

1. Identify the supervisor and read its strategy. In the Erlang shell attached to the node, call supervisor:which_children(Pid) or use Observer to locate the supervisor, then inspect its source in init/1. The strategy is the first element of the value returned by init/1, for example {one_for_one, 3, 5} or {one_for_all, 3, 5}.
2. Read each child specification. Confirm the restart atom: permanent always restarts, transient restarts only on abnormal exit, and temporary never restarts. A permanent child that crashes on every start is a restart-loop candidate.
3. Check restart intensity. The strategy tuple includes MaxRestarts and MaxSeconds. If a child restarts MaxRestarts times within MaxSeconds, the supervisor terminates all children and then itself.
4. Distinguish links from monitors. Code that calls erlang:link/1 creates a bidirectional exit bond. Code that calls erlang:monitor/2 receives a 'DOWN' message but is not killed by the target's exit. Links between siblings can turn one crash into many.
5. Inspect the error log and SASL reports. Look for SUPERVISOR REPORT and PROGRESS REPORT entries around the time of the failures. Note whether the same child repeats and whether siblings appear in the same report.

Fixes tied to findings

Finding	Fix	Effect
Strategy is one_for_all; failures are independent	Change to {one_for_one, MaxRestarts, MaxSeconds}	Only the failed child is restarted; siblings stay up
Child is permanent but should not restart on abnormal exit	Change restart to transient	Normal exits are not restarted; abnormal exits still are
Child never needs restarting	Change restart to temporary	The supervisor ignores the child's exit entirely
Intensity limit is too tight for a flaky but acceptable child	Increase MaxRestarts or MaxSeconds	More restarts allowed before the supervisor gives up
Linked siblings die together	Replace erlang:link/1 with erlang:monitor/2, or move volatile children under their own supervisor	Exit signals no longer cascade through links

Example supervisor configuration

The following is a classic tuple-style child specification. It starts one worker under a one_for_one strategy with an intensity of three restarts in five seconds.

-module(my_sup).
-behavior(supervisor).

-export([start_link/0, init/1]).

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    Workers = [
        {worker1, {worker, start_link, []}, permanent, 5000, worker, [worker]}
    ],
    Strategy = {one_for_one, 3, 5},
    {ok, {Strategy, Workers}}.

To make the strategy less aggressive, replace {one_for_one, 3, 5} with {one_for_one, 1, 10} so the supervisor gives up after one restart in ten seconds. To protect siblings, ensure no child code calls erlang:link/1 on another child unless termination of both is intended.

Commands and where to run them

• Compile: erlc my_sup.erl worker.erl — run in the project directory as the OS user who owns the BEAM node. No root privileges are required.
• Start a test node: erl -pa ebin -s my_sup start_link — run from the same directory. This starts the BEAM and the supervisor.
• Inspect running children: in the Erlang shell attached to the node, run supervisor:which_children(whereis(my_sup)).
• Check process status: sys:get_status(Pid). where Pid is the supervisor or a child.
• Open Observer: observer:start(). — a graphical tool that shows the supervision tree and process links.

Risks: Changing a supervisor's strategy or child specifications requires recompiling and restarting the supervisor, which stops and starts all children under it. Do not make this change on a production node during peak traffic unless the tree is already failing.

Escalation criteria

• If a child hits the intensity limit and the supervisor terminates after you have already tuned MaxRestarts and MaxSeconds, the child has a persistent logic, resource, or dependency bug. Escalate to the team that owns the child code.
• If the failure climbs the tree—supervisor after supervisor terminates—review the entire supervision topology instead of patching one leaf.
• If one_for_all is required because children share state or a connection pool, consider isolating the shared resource in its own supervisor so a pool failure does not restart unrelated workers.

Limitations

• A restart strategy contains failures; it does not fix them. A child with a bug will keep crashing no matter how it is supervised.
• one_for_one does not protect against shared-resource exhaustion. If every worker crashes because the database is unavailable, restarting each worker individually will not help.
• Supervisor configuration cannot be changed on a running supervisor. The new strategy takes effect only after the supervisor restarts.

Practical verification

After applying a fix, restart the supervisor and confirm:

• supervisor:which_children/1 lists the expected children with the expected restart counts.
• Killing one child with exit(Pid, kill) restarts only that child under one_for_one; siblings remain in the running state.
• The error log stops showing repeated SUPERVISOR REPORT entries for the same child within the observation window.

If the supervisor still terminates, revisit the intensity tuple and the child's crash reason before changing the strategy again.

Rollback

Because editing the supervisor module changes the running system only after a restart, rollback means reverting the source file to the previous strategy, recompiling with erlc, and restarting the supervisor. Confirm rollback by checking sys:get_status/1 and the error log after the restart.