Building Fault‑Tolerant State with GenServer and OTP Supervision

The problem

Elixir services often need a piece of mutable state that survives process crashes, handles concurrent requests, and can be upgraded without downtime. Using a plain process or an ETS table gives you either no isolation or no built‑in restart logic. The OTP combination of GenServer plus a supervision tree solves exactly this: a single‑threaded state holder that the supervisor restarts automatically, while callers interact only through a well‑defined API.

Requirements

• Isolated, recoverable state that can be read and written concurrently.
• Automatic restart on crash with configurable intensity.
• Live code upgrade support (hot reload) without losing state.
• No shared memory between callers and the state holder.

Smallest suitable design

A single GenServer module implementing the required callbacks, supervised by a Supervisor using :one_for_one (or :rest_for_one) strategy. The child specification declares restart type (:permanent for critical services, :transient for optional ones) and restart intensity limits.

Minimal GenServer

defmodule KVStore do
  use GenServer

  # Client API
  def start_link(opts \\ []) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  def put(key, value) do
    GenServer.call(__MODULE__, {:put, key, value})
  end

  def get(key) do
    GenServer.call(__MODULE__, {:get, key})
  end

  # Server callbacks
  @impl true
  def init(_opts), do: {:ok, %{}}

  @impl true
  def handle_call({:put, k, v}, _from, state), do: {:reply, :ok, Map.put(state, k, v)}
  @impl true
  def handle_call({:get, k}, _from, state), do: {:reply, Map.get(state, k), state}

  @impl true
  def handle_cast(_msg, state), do: {:noreply, state}

  @impl true
  def handle_info(_msg, state), do: {:noreply, state}

  @impl true
  def terminate(_reason, _state), do: :ok

  @impl true
  def code_change(_old_vsn, state, _extra), do: {:ok, state}
end

Supervisor child spec

children = [
  %{
    id: KVStore,
    start: {KVStore, :start_link, [[]]},
    restart: :permanent,
    shutdown: 500,
    type: :worker
  }
]

Supervisor.start_link(children, strategy: :one_for_one, max_restarts: 3, max_seconds: 5)

The max_restarts/max_seconds tuple caps restart frequency; tune it if the child starts slowly.

Trust and data boundaries

The GenServer owns its state map. Callers never touch it directly; they only invoke call (synchronous) or cast (asynchronous). Validation belongs in the client functions (put/2, get/1) before the message crosses the process boundary. Internal transitions are pure functions, making reasoning and testing straightforward.

Operational checks

1. Process health – In an IEx session attached to the node run :sys.get_state(KVStore) to snapshot the current map.
2. Mailbox and memory – Start :observer.start() or use :recon (:recon.proc_window/3) to watch mailbox length and heap size.
3. Restart counters – Process.info(pid, :restart_count) (replace pid with the child’s PID) shows how many times the supervisor has restarted the GenServer.
4. Supervisor strategy verification – Kill the child with Process.exit(pid, :kill) and observe the log: a :one_for_one supervisor restarts only that child.

Risk: :sys.get_state/1 blocks the caller until the GenServer replies; avoid calling it from latency‑sensitive paths.

Failure modes

• Crash → restart – Any unhandled exception in a callback triggers the supervisor’s restart policy.
• Mailbox overflow – If handle_call or handle_cast runs slower than incoming requests, the mailbox grows and can OOM the VM.
• State corruption on upgrade – code_change/3 must transform every prior state version; missing clauses cause the upgrade to fail and the process to crash.
• Deadlock cycles – Synchronous calls that recursively call back into the same GenServer (or another that calls back) deadlock the caller.

When the design must change

Verification checklist

1. Start the application: iex -S mix (run in project root, no special permissions).
2. Call KVStore.put(:foo, 1) and KVStore.get(:foo) – expect :ok and 1.
3. Simulate crash: Process.exit(Process.whereis(KVStore), :kill) – supervisor logs a restart.
4. Check restart count: Process.info(Process.whereis(KVStore), :restart_count) – should be 1.
5. Enable tracing for a load test: :sys.trace(KVStore, true) then run concurrent calls; inspect the trace log for message order and state transitions.
6. Release a new version with a transformed state (e.g., convert map keys to strings) and invoke the release handler; verify code_change/3 returns {:ok, new_state}.

Limitations

GenServer processes are single‑threaded; CPU‑bound work blocks the mailbox. Offload heavy computation to a Task pool or a separate GenServer. The default restart intensity (3 restarts in 5 seconds) can be too aggressive for children that need longer boot times—adjust max_restarts/max_seconds accordingly. Distributed GenServer introduces split‑brain risk; always pair it with a consensus layer if linearizability matters.

Practical way to confirm correctness

Run a property‑based test (e.g., with StreamData) that fires random put/get sequences, kills the process at random intervals, and asserts that every successful put is eventually readable after the restart. This exercises the supervision loop, state persistence, and code_change path in one automated suite.