Building Fault-Tolerant State with Gleam and OTP GenServers

The Fragility of In-Memory State

In most typed languages, managing a shared, mutable state across concurrent threads involves complex locking mechanisms or risk of race conditions. Even when using actors, a single unhandled exception in a state-holding process often means that state is lost forever, and the process remains dead until the entire application is manually restarted.

The solution is to move from simple processes to Supervised GenServers. By leveraging the gleam_otp library, you can combine Gleam's strict type system with the Erlang VM's (BEAM) "let it crash" philosophy. The takeaway is simple: don't try to catch every possible error inside your logic; instead, define a clear state boundary and let a Supervisor restore it when things go wrong.

The GenServer Pattern in Gleam

A GenServer (Generic Server) is a process that serializes access to a piece of state. It handles three primary types of communication:

• Calls: Synchronous requests that expect a response.
• Casts: Asynchronous "fire-and-forget" messages.
• Info: Messages sent from other processes or system events.

In Gleam, these are implemented via callback functions. Unlike Erlang, where these callbacks are loosely typed, gleam_otp encourages the use of Result types, forcing you to explicitly handle the failure cases of your state transitions before the process crashes.

Implementing a Supervised Counter

To implement this, you need the gleam_otp package. Add it to your gleam.toml and run gleam fetch. The following example demonstrates a simple counter that can be called to increment or retrieve a value, wrapped in a supervisor to ensure it stays alive.

import gleam/otp/process
import gleam/otp/gen_server

// Define the messages the server can handle
pub type Message {
  Increment
  GetCount
}

type State {
  State(count: Int)
}

// The callback implementation
fn handle_call(msg: Message, state: State) {
  case msg {
    Increment -> {
      let new_state = State(count: state.count + 1)
      // Return: {response, new_state}
      {:ok(Nil), new_state}
    }
    GetCount -> {
      {:ok(state.count), state}
    }
  }
}

pub fn main() {
  // Start the server
  let assert Ok(server) = gen_server.start(
    init: fn() { State(count: 0) },
    handle_call: handle_call,
    handle_cast: fn(_, state) { state },
    handle_info: fn(_, state) { state },
  )
  
  // Example usage
  gen_server.call(server, Increment)
  let assert Ok(count) = gen_server.call(server, GetCount)
}

Running and Verifying

Run this on your local machine using gleam run. To verify the fault tolerance, you would typically wrap this gen_server.start call inside a gleam_otp.supervisor child specification. If the handle_call function were to trigger a panic (e.g., through an explicit panic call), the Supervisor would detect the process termination and restart the GenServer with the initial state defined in the init function.

Trade-offs: State Loss vs. Availability

The primary limitation of the GenServer/Supervisor pattern is that restarts reset the state. If your GenServer crashes, the Supervisor restores the process, but the init function runs again, wiping any in-memory data accumulated since the last start.

To mitigate this, you have two architectural choices:

1. External Persistence: Save the state to a database (like PostgreSQL or Redis) during handle_call and reload it during init.
2. State Mirroring: Use a separate process to keep a backup of the state, though this increases complexity and synchronization overhead.

Practical Checklist for OTP in Gleam

When deploying GenServers, follow these safety checks:

• Pin your versions: Ensure gleam_otp is pinned in gleam.toml to avoid breaking changes between minor releases.
• Check BEAM Version: Ensure your Erlang/OTP version is >= 24, as newer gleam_otp features rely on recent BEAM capabilities.
• Avoid Blocking: Never perform long-running synchronous I/O inside handle_call, as this blocks the process mailbox and can lead to timeouts for other callers.