Zig std.Thread.Pool server: p99 latency spikes only under concurrent requests

Direct Answer

Vary allocator strategy first (default page_allocator vs. per‑request ArenaAllocator), then queue boundedness (unbounded MPSC vs. custom bounded queue with backpressure), then worker count. A fixed‑size pool with a bounded queue is not provided by std.Thread.Pool; you must implement backpressure yourself. Allocator contention on the shared page_allocator is a documented bottleneck—per‑request arenas are the safer default for predictable tail latency.

1. Allocator contention (highest impact, easiest to test)

The default std.heap.page_allocator uses a global lock. Under concurrent requests, every task allocation/deallocation serializes on that lock, producing latency spikes that grow with concurrency but disappear in single‑request tests. Switching each task to an ArenaAllocator backed by the same page_allocator moves the lock contention to arena creation/destruction only, which happens once per request instead of per allocation.

2. Unbounded queue growth (structural, requires custom code)

std.Thread.Pool uses an unbounded MPSC queue. Burst traffic causes unbounded memory allocation for queue nodes, amplifying allocator pressure and adding GC‑like pauses. The std library does not provide a bounded queue or backpressure mechanism. You must add a std.Semaphore (or bounded channel) in front of the pool to reject or delay new tasks when the queue reaches a configured limit (e.g., return 503).

3. Worker count (tuning, not a root cause)

Excess workers increase context‑switch overhead; too few underutilize CPU. For I/O‑bound HTTP handlers, start with 2–4× logical cores. This factor modulates the symptom but rarely creates the spike alone.

Minimal Benchmark Plan (One Variable at a Time)

1. Pin the Zig version (e.g., 0.13.0 or 0.14.0-dev) and read std/thread/pool.zig to confirm the queue implementation and allocator usage.
2. Baseline: zig build -Doptimize=ReleaseFast, fixed request payload, constant rate (e.g., wrk -t4 -c100 -d30s --rate=5000), record p99 latency histogram.
3. Test A: Replace default allocator with per‑request ArenaAllocator (reset after each request). Re‑run identical load.
4. Test B: Add a std.Semaphore (initial count = queue bound) acquired before spawnWg, released after task completion. Compare p99 with/without bound at same concurrency.
5. Test C: Vary worker count (cores, 2×, 4×) while keeping allocator and queue bound fixed.
6. Profile: Use perf record -g or heaptrack on the server process to confirm allocation rates and lock contention hotspots.

One Missing Diagnostic Detail

What is the exact Zig version/commit hash you are pinning? The std.Thread.Pool queue implementation (unbounded vs. bounded, allocation strategy) has changed between pre‑1.0 releases, and the allocator contention profile may differ. Confirming the source lets you know whether the unbounded‑queue behavior described here matches your build.