GenServer.call latency measurement vs Telemetry instrumentation: selecting an accurate approach

GenServer.call latency measurement vs Telemetry instrumentation

When a GenServer receives a high‑frequency synchronous request, the round‑trip time of GenServer.call/3 can become a performance bottleneck. Two documented ways to capture this latency are :timer.tc/1 (or :erlang.monotonic_time/0) and Telemetry events emitted by the BEAM runtime. Each technique has subtle interactions with the scheduler, message queue, and measurement overhead.

Using :timer.tc gives a raw time span in microseconds but adds a function call for every measurement, which can itself influence the observed latency. Telemetry, on the other hand, emits events that can be collected by handlers; while the event payload is lightweight, the handler code and any downstream processing may introduce additional delay. Moreover, the BEAM scheduler flags (+P, +Qw, +sbwt) can change preemption behavior, affecting both measurement methods independently of the GenServer logic.

Deciding which method to use before optimising the GenServer is therefore non‑trivial. The goal is to obtain a trustworthy baseline that reflects the true cost of a call under realistic load, without the measurement itself becoming the bottleneck.

Unresolved questions

• Under what conditions does the overhead of :timer.tc skew the latency measurement for a GenServer handling thousands of calls per second?
• How does the presence of Telemetry handlers (e.g., logging or monitoring) impact the observed call latency compared to a bare :timer.tc approach?
• Which BEAM scheduler flags provide the most stable latency readings when using each measurement technique?