Erlang Process Mailbox Overflow: Diagnostic Guide for Unbounded Queues

Problem: an Erlang process mailbox that never drains

An Erlang process mailbox is the process's message queue. When producers send messages faster than the process can receive them, `message_queue_len` climbs. The VM continues to schedule the process, but the queue consumes heap, garbage collection slows, and the node can run out of memory. Do not treat this as a generic "Erlang is slow" issue. The useful takeaway is to isolate the exact process holding the backlog, determine whether the consumer is slow, the producer is bursting, or a receive clause is dropping messages, then apply the matching fix.

Recognizable condition

You are likely looking at mailbox overflow when:

• `erlang:process_info(Pid, message_queue_len)` returns a value that increases over repeated samples instead of fluctuating around a small baseline.
• The same process shows rising `total_heap_size` in `process_info/2`.
• Calls to a `gen_server` or `gen_statem` owned by that process take seconds instead of milliseconds.
• The node approaches its memory limit or emits out-of-memory warnings.

Cause and diagnostic table

Observed pattern	Likely cause	First diagnostic signal
Queue grows while process is scheduled and running	Slow consumer	High reductions with little queue reduction
Sudden queue spike after traffic increase	Burst producer	Correlate with `erlang:statistics(receive)`
Old messages sit at the head of the queue	Selective receive mismatch	Missing catch-all `_ ->` clause
High-priority process starves other queues	Priority inversion	`process_flag(priority, high)` set on suspect
Many processes queue at once on one node	Distribution congestion	`net_kernel` link churn

Ordered runtime checks

1. Identify the top queue holders. Run in an Erlang shell attached to the target node (local shell or `erl -remsh `):

   recon:info(processes, 20, message_queue_len).

   This requires the `recon` application to be loaded. It is read-only and safe on production. Look for Pids with queue lengths far above the rest of the system.
2. Inspect each suspect process. Replace `` with the identifier from step 1:

   erlang:process_info(, [message_queue_len, memory, status, current_function, stacktrace]).

   Run in the same shell. No special OS permissions beyond Erlang node access are required. A `status` of `waiting` with a receive in `current_function` means the process is alive but not matching messages.
3. Measure consumer liveness over a window.

   recon:proc_window(, 5000, [reductions, run_queue, io]).

   This samples the process for five seconds. If reductions are high but the queue still grows, the consumer is processing work but not draining mail; if reductions are low, the process may be blocked on I/O or a scheduler.
4. Audit receive clauses in source. Search the module for `receive` blocks. A clause such as `receive ... after 0 -> ok end` or a block without a catch-all `_ ->` can leave messages unmatched indefinitely. This is a code review step, not a shell command.
5. Check allocator and GC pressure. Sample `erlang:statistics(gc)` and, if `recon` is available, `recon:allocator_memory()`. Carrier migration or fragmentation can make queue growth look worse than it is.
6. Correlate with application metrics. Match queue growth to request rates, TCP accept counts, or downstream dependency latency. A burst that coincides with a deploy or traffic spike points to the producer side.

Fixes tied to findings

Slow consumer

Optimize the hot path in `current_function`, split work across multiple parallel consumer processes, or tune scheduler count at VM startup with `+S`. Example startup command:

erl +S 4 -s my_app

This changes VM scheduling capacity; restart the node to apply.

Burst producer

Apply backpressure. For TCP sockets, use `{active, once}` so the process explicitly requests the next message instead of having the socket push continuously:

{ok, Socket} = gen_tcp:connect(Host, Port, [binary, {active, once}]),

When data is processed, call `inet:setopt(Socket, [{active, once}])` to arm the next message. Kernel buffers absorb short bursts; ensure `sndbuf` and `recbuf` are sized for your expected burst. Application-level rate limiting or a bounded queue before the Erlang process is another option.

Selective receive mismatch

Add a catch-all clause that logs unmatched messages:

receive\n  {expected, Payload} -> handle(Payload);\n  Msg ->\n    error("unmatched message ~p", [Msg]),\n    handle(Msg)\nend.

Better yet, refactor ad-hoc `receive` blocks into `gen_server` or `gen_statem` callbacks so OTP handles the dispatch. This is a code change; redeploy the module and roll back by reverting the deploy if queue behavior does not improve.

Priority inversion

If a low-priority worker is being starved, raise its priority only after profiling confirms the inversion:

process_flag(priority, high).

Return it to `normal` once the backlog clears. Raising priority can starve other processes, so treat it as temporary.

Distribution congestion

Tune `net_kernel` buffer settings or add nodes and shard work so no single node carries all distribution traffic. Persistent `nodedown` and `nodeup` events indicate an unstable link that should be fixed before tuning mailbox behavior.

Escalation criteria

Escalate to on-call when any condition persists:

• One process queue above 100,000 messages for more than 30 seconds.
• Total VM memory above 80 percent of its limit, with more than 30 percent attributable to process heaps.
• Scheduler utilization above 95 percent on all cores with run queue depth above two times the core count.
• Garbage-collection pause above 50 ms, observed by tracing `garbage_collection` with `erlang:trace/3`.
• Repeated `nodedown` or `nodeup` events.

Collect a `recon:save_state/1` snapshot before restarting, and generate a crashdump only if the VM is unresponsive.

Limitations and verification

`message_queue_len` is a snapshot, not a rate. Poll at one-second intervals for at least 30 seconds to separate a transient burst from a leak. Off-heap message data (`message_queue_data = off_heap`) reduces GC pause but increases resident memory; monitor both queue length and RSS.

After applying a fix, verify by sampling the suspect PID every second for one minute. The queue should stabilize or decline. Run a representative load test while watching `erlang:statistics(gc)` to confirm GC pause time drops. If the queue continues to grow under normal load, return to step 2 and re-inspect the process stacktrace.