Elixir Supervision Trees as Fault Isolation Boundaries

A crash in a single request handler should not restart your database pool or your message consumer. In production Elixir systems the common failure is not the crash itself, but restart propagation that couples unrelated subsystems. The useful takeaway is to treat a supervision tree as a trust boundary: a supervisor owns the lifecycle of its children and decides which failures are correlated enough to restart together.

Requirements for fault isolation

Fault isolation requires three properties. First, failure containment: a bug in one worker must not kill workers that share no state. Second, restart policy clarity: the strategy must match failure correlation. Third, observable intensity: you need to see when a supervisor is restarting too fast.

OTP supervisors provide those properties via strategy and intensity limits. The default intensity is max_restarts: 3, max_seconds: 5. If a supervisor restarts more than three children in five seconds it terminates itself, which is the signal that a design boundary is wrong.

Smallest suitable design

Start with one top-level supervisor per independent subsystem, not one tree for the whole app. A minimal shape for a service with HTTP, DB and a consumer is:

defmodule MyApp.Application do
  use Application

  def start(_type, _args) do
    children = [
      {MyApp.ApiSupervisor, []},
      {MyApp.DBSupervisor, []},
      {MyApp.ConsumerSupervisor, []}
    ]

    opts = [strategy: :one_for_one, name: MyApp.TopSupervisor]
    Supervisor.start_link(children, opts)
  end
end

Each subsystem supervisor owns only its own children. A crash in the API pool does not restart the DB pool because they are siblings under a :one_for_one top supervisor.

Child specifications since Elixir 1.5/OTP 21 are maps with :id, :start, :restart, :shutdown, :type and :modules. Using the map form makes restart intent explicit and supports hot code upgrades.

Supervisor.child_spec({MyApp.Repo, []}, id: :repo, restart: :permanent, shutdown: 5_000, type: :worker)

Keep nesting shallow. More than five levels makes restart propagation hard to reason about and increases shutdown latency during releases.

Trust and data boundaries

A supervisor is a trust boundary. Children inside it are assumed to be related enough that restarting one may require restarting others.

• :one_for_one assumes failures are independent. Use for pools of stateless workers.
• :rest_for_one assumes ordering dependency, e.g., a cache that must be restarted after its source.
• :one_for_all assumes strong correlation. Use sparingly, typically for a small group that shares in-memory state.

Do not trap exits in a supervisor. Trapping defeats the isolation guarantee and can mask bugs by converting crashes into silent messages. If you need custom restart logic, implement it in a separate process, not in the supervisor itself.

Supervision does not provide backpressure. Pair supervision with GenStage or Broadway for flow control in data pipelines, otherwise a restarted consumer can be overwhelmed by a backlog.

Operational checks

Monitor restart frequency and child health, not just process uptime.

In an iex session connected to the node you can inspect the tree:

iex -S mix
:observer.start()

The Applications tab visualizes the tree. Trigger a controlled exit to verify isolation, e.g.:

{sup_pid, child_pid} = ... # obtain from your test harness
Process.exit(child_pid, :kill)

Check supervisor state:

:supervisor.count_children(sup_pid)
:sys.get_status(sup_pid)
Supervisor.get_child_spec(sup_pid, child_id)

These calls show current child count, restart intensity, and the effective :restart and :shutdown values. Alert when restart intensity approaches the configured :max_restarts/:max_seconds threshold.

For deeper inspection, :recon can report memory growth per process and restart hotspots. Review child specs in code review to ensure non-critical workers use :temporary or :transient restart instead of :permanent.

Failure modes

Aggressive restarts cause thundering herd. A supervisor that restarts a pool of workers too quickly can spike CPU and connection usage. Mitigate with :temporary for fire-and-forget tasks and by adding backoff in the child start function, not in the supervisor.

Deep nesting amplifies shutdown latency. During a release, a top supervisor will wait for each child’s :shutdown timeout before moving down the tree. Long timeouts in leaf workers can delay the whole release.

Elixir 1.11+ deprecated :simple_one_for_one in favor of DynamicSupervisor. Legacy code should migrate to avoid warnings and gain better introspection via DynamicSupervisor.which_children/1.

Conditions that change the design

Introduce a stateful singleton, such as a Registry or a named GenServer, and the tree needs a dedicated supervisor with :one_for_one and :permanent restart, isolated from pools.

Cross-node supervision changes boundaries. Use :global or :pg for process groups, but note that network partitions turn supervision into a distributed failure domain. Prefer local supervision with explicit RPC boundaries.

Regulatory requirements for zero-downtime deployments may require separate supervision roots for blue/green releases, allowing independent shutdown and start sequences.

Limitations to remember: supervision trees restart processes, they do not heal corrupted state. Pair supervision with idempotent workers and external health checks. Verify the design by creating a minimal mix project with two children, one crashing, one stable, and asserting the stable child continues after killing the crashing child.