Tuning Tomcat’s NIO Connector for High Concurrency

Problem: Connections are refused even though CPU is idle

During a traffic spike you notice HTTP 503 or "connection refused" errors in the access logs, while top shows plenty of free CPU and memory. The application code is not blocking, yet Tomcat stops accepting new sockets. This symptom usually points to the NIO connector’s internal limits being hit.

Thesis: Properly sized thread pool, accept queue, and keep‑alive timeout eliminate refusals without over‑provisioning resources

Tomcat’s NIO connector uses a small set of worker threads to handle many sockets via non‑blocking I/O. Three knobs control its behavior under load:

• maxThreads – maximum request‑processing threads.
• acceptCount – length of the socket accept queue when all workers are busy.
• keepAliveTimeout – how long an idle HTTP keep‑alive socket stays open.

Setting them too low causes refused connections; setting them too high wastes memory and can increase context‑switch overhead. The goal is to find values that match the expected concurrent request rate while keeping the JVM heap within safe bounds.

Understanding the NIO connector’s moving parts

Thread pool and maxThreads

Each worker thread processes a request from start to finish. If your application’s average request handling time is T seconds and you expect R concurrent requests, a rough lower bound is maxThreads ≥ R × T. However, because NIO lets a thread switch between sockets while waiting for I/O, you can often run with fewer threads than the raw concurrency number.

Accept queue and acceptCount

When all maxThreads are busy, incoming TCP connections are queued up to acceptCount. Once the queue fills, the OS rejects new SYNs, leading to "connection refused" errors seen by clients. A large queue can hide starvation by increasing latency, so it should be sized to absorb short bursts, not to mask chronic overload.

Keep‑alive timeout and keepAliveTimeout

HTTP/1.1 keep‑alive sockets consume a file descriptor and a slot in the NIO poller. If the timeout is too high, idle connections accumulate under steady load, eventually exhausting file descriptors even when request processing is light. A typical value is a few seconds (e.g., 15000 ms) for high‑traffic sites.

Worked example: Tuning a front‑end Tomcat instance

Assume:

• Expected peak concurrent requests: 800
• Average request processing time (including DB calls): 0.05 s (50 ms)
• Desired headroom for bursts: 20 %
• JVM heap: 4 GB, aiming to keep thread stacks under 500 MB.

Step 1 – Estimate thread need: 800 × 0.05 s = 40 threads to keep the pipeline full. Adding 20 % headroom gives about 50 threads. We’ll set maxThreads to 75 to allow for occasional longer requests.

Step 2 – Size the accept queue: Choose a value that can absorb a short spike of, say, 100 ms of extra arrivals at peak rate. At 800 req/s, that’s 80 extra connections. We’ll set acceptCount to 150.

Step 3 – Keep‑alive timeout: To limit idle descriptors, we choose 15000 ms (15 s). This balances the benefit of keep‑alive for browsers with the risk of descriptor exhaustion.

Step 4 – Share a thread pool across HTTP and HTTPS connectors (optional but reduces duplicate threads). Define a named Executor and reference it from both connectors.

The resulting server.xml snippet (replace placeholders with your actual paths):

<Executor name='tomcatThreadPool'
          namePrefix='tomcat-'
          maxThreads='75'
          minSpareThreads='25'
          maxIdleTime='60000'/>

<Connector port='8080'
           protocol='org.apache.coyote.http11.Http11NioProtocol'
           connectionTimeout='20000'
           redirectPort='8443'
           executor='tomcatThreadPool'
           acceptCount='150'
           keepAliveTimeout='15000'/>

<Connector port='8443'
           protocol='org.apache.coyote.http11.Http11NioProtocol'
           connectionTimeout='20000'
           executor='tomcatThreadPool'
           acceptCount='150'
           keepAliveTimeout='15000'
           scheme='https'
           secure='true'/>

Note: The protocol attribute must explicitly reference the NIO implementation; omitting it falls back to the blocking BIO connector.

Trade‑offs and limitations

Increasing maxThreads reduces the chance of thread starvation but consumes memory: each thread’s stack (default 1 MB on many Linux JVMs) multiplies by the thread count. With 75 threads, stack usage is ~75 MB, well within a 4 GB heap. Pushing maxThreads into the hundreds can quickly approach heap limits, especially if your application uses large thread‑local buffers.

A large acceptCount masks overload by letting connections sit in the queue, raising latency. Monitor queue depth via JMX (Connector.acceptCount vs. Connector.currentThreadCount) to ensure it stays low during normal operation.

The NIO connector does not fix blocking code inside your servlet or service. If a request holds a thread for seconds due to a synchronous DB call, the effective concurrency drops, and you may need to increase maxThreads or, better, make the call asynchronous.

Verification and monitoring

After restarting Tomcat with the new configuration, check that the connector is indeed using NIO:

• In catalina.out look for a line containing "Initializing ProtocolHandler [\"http‑nio‑8080\"]".
• Via JMX (e.g., jconsole or VisualVM), navigate to Catalina → Global → RequestProcessor → maxThreads and currentThreadCount to see active vs. idle threads.
• The Tomcat Manager app (http://host:8080/manager/text/threads) returns a plain‑text dump of thread pool statistics.

To validate under load, run a short test with a tool like k6 or JMeter targeting a static resource. Gradually increase the virtual user count and observe:

• Response time latency – should stay flat until near the configured concurrent limit.
• Connection refused errors – should be zero.
• Thread pool usage – currentThreadCount should approach but not exceed maxThreads.

If you see refusals, increase acceptCount first; if latency climbs sharply, consider raising maxThreads or investigating application‑level blocking.

Actionable closing

Start with the calculated baseline (maxThreads ≈ expected concurrent requests × average request time, plus headroom), set acceptCount to absorb short bursts, and keep keepAliveTimeout in the low‑to‑mid‑seconds range. Verify via JMX or the Manager app that thread usage stays healthy and that no connection refusals appear under realistic load. Adjust incrementally, watching both memory consumption and latency, and you’ll achieve a Tomcat instance that handles high concurrency without refusing connections or needlessly wasting resources.