Building Fault‑Tolerant Chat Services with OTP Supervision Trees in Erlang

The problem: a chat server that dies when a single user misbehaves

Imagine a simple Erlang chat service where each connected user runs in its own process. If one user sends a malformed message that triggers an exception, that process crashes. Without a restart mechanism, the crashed user disappears and the rest of the system keeps running, but you have no visibility into why the failure happened and you must manually intervene to restore the user’s session.

Thesis: OTP supervision trees give you automatic, bounded restarts that keep the service alive while surfacing failures for later investigation

By organizing user processes under a supervisor, you let OTP handle restarts according to a chosen strategy. The supervisor isolates faults, prevents them from cascading, and logs each restart so you can diagnose the root cause without losing the rest of the chat room.

Designing a supervision tree for a chat server

OTP separates concerns: workers do the actual work (here, handling a socket and broadcasting messages); supervisors only monitor workers and decide what to do when they die. A typical hierarchy looks like this:

• Application supervisor – starts the chat server and ensures it is running.
• Chat supervisor – manages all user‑connection workers.
• User worker – a gen_server (or plain process) that reads from a socket, decodes messages, and publishes them to a room.

Because user workers are stateless with respect to the supervision tree (they hold only socket state), a one_for_one strategy is appropriate: if a single worker crashes, only that worker is restarted, leaving other users untouched.

Worked example: starting the tree and observing a restart

The following steps illustrate how to create, compile, and test the supervision tree. Replace chat_app with your own application name.

1. Create a new OTP application:

   # Run in a shell with normal user privileges
   mix new chat_app
   cd chat_app

2. Add the supervision boilerplate to lib/chat_app/application.ex:

   defmodule ChatApp.Application do
     use Application
   
     def start(_type, _args) do
       children = [
         {ChatApp.ChatSupervisor, []}
       ]
   
       opts = [strategy: :one_for_one, name: ChatApp.Supervisor]
       Supervisor.start_link(children, opts)
     end
   end

3. Define the chat supervisor in lib/chat_app/chat_supervisor.ex:

   defmodule ChatApp.ChatSupervisor do
     use Supervisor
   
     def start_link(_) do
       Supervisor.start_link(__MODULE__, :ok, name: __MODULE__)
     end
   
     def init(:ok) do
       children = [
         worker(ChatApp.UserWorker, [])
       ]
   
       Supervisor.init(children, strategy: :one_for_one)
     end
   end

4. Implement a minimal user worker that crashes on purpose after receiving the string "crash":

   defmodule ChatApp.UserWorker do
     use GenServer
   
     # client API
     def start_link(_) do
       GenServer.start_link(__MODULE__, :ok, name: via_tuple())
     end
   
     def via_tuple() do
       {:via, Registry, {ChatApp.Registry, self()}}
     end
   
     # server callbacks
     def init(:ok) do
       {:ok, nil}
     end
   
     def handle_info({:msg, content}, state) do
       if content == "crash" do
         # Simulate a bug
         exit(:kill)
       else
         IO.puts("Received: #{content}")
         {:noreply, state}
       end
     end
   end

5. Start a local registry (needed for the via_tuple): add to lib/chat_app/application.ex inside the children list:

   {Registry, [name: ChatApp.Registry, keys: :unique]}

6. Compile and launch the application:

   # Still inside the chat_app directory
   mix compile
   iex -S mix

7. Inside the IEx session, start a user worker:

   {:ok, pid} = ChatApp.UserWorker.start_link()
   # pid is the registered process

8. Send a normal message – you should see it printed:

   send pid, {:msg, "hello"}
   # => Received: hello

9. Now send the crash trigger:

   send pid, {:msg, "crash"}
   # The process exits; the supervisor logs something like:
   # ** (EXIT from #PID<0.123.0>) normal
   # Supervisor ChatApp.ChatSupervisor restarted UserWorker at ...
   # A new worker is automatically started.

10. Verify the restart by sending another message to the same registered name:

    send pid, {:msg, "after crash"}
    # => Received: after crash (new worker)

All commands above assume you have Erlang/OTP 25+ and Elixir 1.14+ installed. No special privileges are required; just a regular user account on your development machine.

Trade‑offs and limitations

While automatic restarts improve availability, they can hide bugs if you rely on them without investigating why a worker dies. Repeated crashes of the same worker may indicate a logic error that should be fixed rather than masked by restarts.

Choosing the wrong restart strategy also has costs. A one_for_all strategy would restart every user worker when one fails, causing unnecessary churn in a busy chat room. Conversely, a one_for_one strategy is safe for isolated faults but will not protect you from a supervisor‑level bug that takes down the whole tree.

Practical way to check the result: after each crash, look at the console output (or use :observer.start() in IEx) to confirm that exactly the expected number of restarts occurred and that the total number of live user workers matches the number of active connections.

Actionable next steps

1. Replace the simulated exit(:kill) with real error handling (e.g., catching malformed JSON) and log the offending payload.
2. Experiment with different strategies (rest_for_one, one_for_all) and observe the impact on worker counts using the observer tool.
3. Add a max_restarts and max_seconds to the supervisor configuration to prevent infinite restart loops.
4. Consider persisting user state (e.g., nickname, rooms) outside the worker process so that a restart does not lose user‑specific data.