gen_server mailbox bloat impact on connection pool availability

To prevent a single blocked gen_server from exhausting a connection pool, you must move the blocking I/O operation out of the main gen_server process loop or implement a manual timeout mechanism. There is no native mechanism in OTP to enforce a timeout on a handle_call callback; if the function blocks, the process remains occupied and cannot process the next message in its mailbox.

Strategy 1: The Worker Process Pattern (Recommended)

The most robust strategy is to treat the gen_server strictly as an orchestrator. Instead of performing the I/O directly within handle_call, the server spawns a temporary worker process to handle the specific request.

• The gen_server receives the call and spawns a worker using spawn/1.
• The worker performs the blocking I/O.
• The worker sends the result back to the gen_server or directly to the caller.
• This keeps the gen_server loop free to continue consuming messages from its mailbox while workers wait on downstream latency.

Strategy 2: Manual Timeout with send_after

If you cannot change the architecture, you can manually implement a timeout using erlang:send_after/3. This allows the process to "recover" if the downstream dependency takes too long.

handle_call({fetch_data, From}, State) ->
  Timeout = 5000, % 5 seconds
  Ref = erlang:send_after(Timeout, self(), :timeout)
  
  % Start the blocking operation in a separate process or async task
  % or use a state machine to track the pending request.
  
  {noreply, State#{waiting_for: From, timeout_ref: Ref}}

hand_info(:timeout, State#{waiting_for = From, timeout_ref = Ref}) ->
  erlang:cancel_timer(Ref)
  reply(From, {error, timeout})
  {noreply, State#{waiting_for: undefined}}

Understanding the Diagnostic Gap

While process_info(Pid, message_queue_len) identifies that the mailbox is growing, it does not explain why. To distinguish between high load and a stalled process, you must correlate the queue length with process state metrics:

• High Load: High queue length + High CPU usage (the process is working but busy).
• Stalled: High queue length + Zero CPU usage (the process is blocked on a synchronous I/O call or a lock).

Are you using a specific pooling library like poolboy, or is this a custom-built gen_server manager? This would change the specific implementation recommendation.