Elixir Supervisor Restart Strategy Diagnostic Guide: Recognize, Check, Fix, Escalate

Recognizable Condition

Frequent `:shutdown` messages in the supervisor log together with sudden CPU spikes indicate that child processes are terminating and being restarted faster than the supervisor’s intensity limits allow.

Cause / Diagnostic Table

Possible Cause	What to Look For
Mis‑chosen restart strategy (e.g., `:one_for_one` when children are tightly coupled)	Check the child spec and see if children share state or depend on each other’s success.
Unhandled exception in a child process	Examine crash logs (`logger` output or `:sys.get_state/1`) for reason terms like `:badarg`, `:undef`, or custom error tuples.
Incorrect intensity limits (`max_restarts`/`max_seconds`)	Review the supervisor options; low values cause the supervisor to terminate the whole tree after a few restarts.
Resource starvation (mailbox overflow, CPU saturation)	Monitor message queue length with `:erlang.process_info(pid, :message_queue_len)` and observe CPU usage via `:observer` or OS tools.

Ordered Checks

1. Examine supervisor logs – run the application locally or in staging and tail the log file (e.g., `tail -f log/elixir_app.log`). Look for lines containing `Supervisor ... child ... terminated` and the reason `:shutdown`.
2. Inspect each child’s start_link/function – open the module that implements the child (often a `GenServer`) and verify that the `init/1` and `handle_*` callbacks do not raise. You can temporarily add `:try ... rescue` or use `Sys.get_state/1` after start to see the state.
3. Verify restart strategy and intensity – locate the supervisor definition (usually in `lib/app/application.ex` or a dedicated supervisor module). Note the `:strategy` and `:intensity` values (`:max_restarts`, `:max_seconds`).
4. Check system resources – from an IEx session run `:observer.start()` and go to the Applications tab, or from the command line run `mix run --no-halt` and in another terminal execute:

   :erlang.process_info(self(), :message_queue_len)

   to see the mailbox length of the current process, and use `:erlang.system_info(:schedulers_online)` for CPU count.
5. Test the child in isolation – start the child manually with `Supervisor.start_child(MyApp.Supervisor, [arg])` in an IEx session and send it typical messages. Observe whether it crashes or stays alive.

Fixes Tied to Findings

If the strategy is too aggressive for coupled children

Change the supervisor’s strategy to `:one_for_all` or `:rest_for_one` depending on the dependency graph. Example configuration:

children = [
  worker(MyApp.Parser, []),
  worker(MyApp.Validator, []),
  worker(MyApp.Storage, [])
]

opts = [strategy: :one_for_all, intensity: [max_restarts: 5, max_seconds: 60]]
Supervisor.start_link(children, opts)

This ensures that when one child fails, all (or the rest) are restarted together, preventing a cascade of isolated restarts.

If children crash due to unhandled exceptions

Locate the offending clause and either fix the bug or add a rescue/catch. For a GenServer:

def handle_info(:timeout, state) do
  # original code that could raise
  {:noreply, state}
end

def handle_info(_msg, state) do
  {:noreply, state}
end

Or wrap the risky call:

try do
  risky_operation()
rescue
  exception -> 
    Logger.error\\"Child failed: #{inspect(exception)}\\"
    {:stop, :error, state}
end

If intensity limits are too low

Increase the allowed restarts or lengthen the window. Example:

opts = [strategy: :one_for_one, intensity: [max_restarts: 20, max_seconds: 300]]

Remember that this only hides the symptom; use it temporarily while you investigate the root cause.

If resource starvation is observed

Apply back‑pressure or reduce work per message. For a GenServer that processes batches:

def handle_info({:process_batch, data}, state) do
  # Process a limited chunk
  chunk = Enum.take(data, 100)
  # …process chunk…
  remaining = Enum.drop(data, 100)
  if remaining != [] do
    self() |> send(:process_batch, remaining)
  end
  {:noreply, state}
end

Also consider increasing the VM’s process limit with `+P` flag if you truly need many concurrent processes.

Escalation Criteria

• After applying a fix, restart frequency remains > 5 restarts per minute (check with `logger` timestamps or a simple script that counts `:shutdown` lines).
• CPU usage stays elevated (> 80 % of a core) or memory grows unbounded.
• The supervision tree still produces cascading failures despite strategy changes.

When any of these criteria are met, consider:

1. Redesigning the supervision tree: move the problematic child to its own supervisor with a tailored strategy.
2. Extracting the child into a separate OTP application to isolate its failure domain.
3. Replacing the GenServer with a `Task` or `Agent` if the workload is simple, or using a `GenStage` flow with explicit demand‑based back‑pressure.

Verification

After changes, observe the supervisor log: the `:shutdown` messages should drop to baseline (e.g., less than one per minute) and CPU usage should return to normal levels. You can also run a load test:

mix run --no-halt
# In another terminal:
while true do
  curl -s http://localhost:4000/endpoint > /dev/null
  sleep 0.1
done

Check that no unexpected shutdowns appear and that response times stay within your SLA. Optionally, use `:observer.start()` or `:sys.get_state/1` to inspect child process state and mailbox sizes to ensure they remain bounded.