Nim httpclient connection pool exhaustion under high async concurrency

Answer the exact question first

The Nim stdlib httpclient keeps a per‑host pool of idle sockets. The pool size is hard‑coded (8 in Nim 1.6.x, 16 in Nim 2.0.0) and there is no public API to change it. When your async dispatch spawns more concurrent requests than that number, the client blocks waiting for a free socket. This manifests as latency spikes or occasional request failures that look like server‑side limits.

Diagnostic approach (no stdlib changes)

1. Measure active sockets. While the benchmark runs, run:

   lsof -p $PID | grep ESTABLISHED | wc -l
   # or
   ss -tanp | grep $PID | wc -l
   

   This counts the TCP connections that the Nim process has open. If the count plateaus at the hard‑coded limit, you have pool exhaustion.
2. Record request latency. Instrument your code to log the timestamp before client.send and after the response is received. A histogram that shows a sharp increase in the 90th‑percentile latency when you exceed the pool size is a clear sign of blocking.
3. Correlate with pending async tasks. In Nim, you can inspect the number of queued tasks via asyncdispatch.getPoolSize() or by adding a counter that increments on asyncproc start and decrements on finish. If the counter grows without bound while the socket count is capped, the pool is saturated.
4. Optional wrapper. Wrap client.send in a small module that logs when a connection is borrowed or returned. This gives you an internal view without touching the stdlib source.

Observable signals before requests block

• Latency distribution shifts: the tail of the histogram spikes.
• All sockets in the process reach the pool cap and stay in ESTABLISHED state.
• Async task queue length grows steadily while throughput plateaus.

Should you cap application concurrency?

Yes, as a pragmatic mitigation. Set the maximum number of concurrent async requests to the pool size minus a safety margin (e.g., 70 % of the hard‑coded limit). This keeps the client from blocking while still giving you throughput. If you need higher concurrency, consider:

• Using a third‑party HTTP client that exposes a tunable pool.
• Running multiple Nim processes and balancing the load across them.

Verifying the hard‑coded limit for your Nim version

1. Open the source file stdlib/httpclient.nim in your toolchain directory and locate the constant maxIdlePerHost. Confirm the value matches the one you observed.
2. Alternatively, compile a tiny program that prints httpclient.maxIdlePerHost if the constant is exported (for newer releases it may be).
3. Run the socket‑count diagnostic above while issuing a large burst of requests. If the count never exceeds the source value, you can trust that the limit is still enforced.

Because future Nim releases might change the constant or expose a configuration API, automate this check as part of your CI pipeline: run the benchmark, assert that the socket count does not exceed the source‑defined limit, and fail the build if it does.

Missing diagnostic detail we need to refine the recommendation

Which exact Nim compiler version and toolchain are you using in production? Knowing the exact version ensures that the hard‑coded limit we reference matches your environment.