Increase Netty workerThreads or accept acceptor‑queue‑induced latency bursts under high concurrency?

Answer the question first

There is no single workerThread count that works for every Ktor application. The optimal size depends on the number of CPU cores, the blocking‑ness of your request handlers, and the load pattern. A practical rule of thumb is to start with two worker threads per CPU core for I/O‑bound workloads and then adjust based on measured p99 latency and context‑switching activity. Raising the pool beyond the point where latency stops improving will only increase context switches and memory pressure.

The acceptor backlog (the OS TCP listen queue) is not exposed as a stable Ktor or Netty API. When it overflows, new connections are dropped or delayed, which shows up as latency spikes. Because you can’t change it through Ktor, you need to monitor it with OS‑level tools and let the results guide whether you need a larger backlog or a smaller worker pool.

Likely explanation of the trade‑off

• WorkerThreads > saturation → more context switches → higher tail latency.
• WorkerThreads < saturation → worker threads idle while the OS accept queue builds → bursts when the backlog fills.
• Both extremes can be mitigated by measuring latency, CPU usage, and the backlog depth.

Steps to find the right workerThreads count

1. Determine the number of available logical CPUs on the host. On Linux, you can use nproc or lscpu | grep CPU(s).
2. Set the initial workerThreads value in NettyApplicationEngineConfig to twice that number.

       engine { workerThreads = 2 * Runtime.getRuntime().availableProcessors() }
       

3. Run a representative load test that mimics the real traffic pattern. Capture:
   • p50 and p99 request latency (in milliseconds)
   • CPU utilization per core
   • Context‑switches per second (e.g., vmstat -s or pidstat -w -p <pid>)
4. Increase workerThreads by 10‑20% increments and repeat the test until the p99 latency plateaus or starts to rise again.
5. Stop when you observe:
   • Latency improvement stops or worsens.
   • Context‑switch rate rises noticeably.
   • CPU utilization remains high but not saturated.

Observing the acceptor queue depth

Because the backlog is managed by the OS, you can’t read it from Netty. Use these commands to get an indirect view:

• ss -ltn state listening -i – shows the backlog size and the number of queued connections for each listening socket.
• netstat -tanp | grep LISTEN – older but still useful on many systems.
• On Linux, cat /proc/net/tcp contains backlog information in the sixth column of each line.
• If you see frequent “listen queue drops” or a backlog that reaches its limit, the OS is dropping or delaying new connections.

Correlate these metrics with spikes in request latency from your load test. If the backlog is often full, consider raising the OS backlog value (e.g., sysctl -w net.core.somaxconn=65535) or reducing the rate of new connections by throttling clients or adding a connection pool on the client side.

When to consider alternatives

• If your handlers perform blocking I/O (e.g., JDBC, file reads) and you already have a worker pool that saturates, move the blocking work to a dedicated thread pool or use non‑blocking libraries.
• Enable backpressure in Ktor by setting pipeline.intercept(HttpSendPipeline.After) { ... } to avoid overwhelming the event loop.
• Limit the maximum concurrent connections at the server or client level to keep the backlog from filling too quickly.

Missing diagnostic detail that can change the recommendation

Could you tell me how many logical CPU cores are on the machine running the Ktor server and whether your request handlers perform blocking operations? This information will help fine‑tune the workerThreads count and decide whether a larger backlog or a smaller pool is more appropriate.