Tuning Tomcat NIO: Balancing Threads and Connections for High Concurrency

The Thread Exhaustion Trap

A common failure pattern in Apache Tomcat deployments occurs when a server stops accepting new requests despite low CPU usage. This usually happens because the maxThreads limit has been reached, and the TCP backlog queue (acceptCount) has filled up. When this occurs, the operating system rejects new connection attempts, leading to "Connection Refused" errors for the end user.

The takeaway is that concurrency is not the same as throughput. Increasing your thread count blindly often degrades performance due to context switching—the overhead the CPU incurs when swapping between different execution threads.

Understanding the NIO Connection Lifecycle

Modern Tomcat versions use the NIO (Non-blocking I/O) connector by default. Unlike the legacy BIO (Blocking I/O) model, where one thread was tied to one connection for its entire duration, NIO separates the connection from the processing.

• maxConnections: The total number of TCP connections the server will maintain. This includes "Keep-Alive" connections that are idling and not currently requesting data.
• maxThreads: The maximum number of worker threads available to actually execute the request logic (the Servlet code).
• acceptCount: The OS-level queue for incoming connection requests when all maxConnections are full.

In an NIO setup, maxConnections can be significantly higher than maxThreads. This allows Tomcat to hold thousands of open sockets while only using a few hundred threads to process the active requests within those sockets.

Decoupling with the Executor

By default, each Connector manages its own internal thread pool. However, if you run multiple connectors (e.g., one for HTTP and one for HTTPS), it is more efficient to use a shared Executor. This centralizes resource management and prevents one connector from starving the other of JVM memory.

Worked Example: High-Concurrency Configuration

Assume a scenario where you expect high bursts of traffic but have a limited heap size. You want to maintain 2,000 concurrent connections but only allow 200 active processing threads to prevent CPU thrashing.

Edit your conf/server.xml with the following configuration. This requires administrative permissions to modify the Tomcat installation directory.

<!-- Define a shared thread pool -->
<Executor name="tomcatThreadPool" 
          namePrefix="catalina-exec-" 
          maxThreads="200" 
          minSpareThreads="20" />

<!-- Apply the executor to the HTTP Connector -->
<Connector executorName="tomcatThreadPool"
           port="8080"
           protocol="HTTP/1.1"
           connectionTimeout="20000"
           maxConnections="2000"
           acceptCount="100" 
           redirectPort="8443" />

Expected Behavior: Tomcat will accept up to 2,000 simultaneous TCP connections. If 200 threads are already busy processing requests, the remaining 1,800 connections will wait in a non-blocking state. If the 2,000 limit is exceeded, the OS will queue up to 100 more requests before rejecting them.

Trade-offs and Limitations

While increasing maxConnections prevents immediate connection drops, it introduces a risk of latency masking. If your backend database or API is slow, requests will pile up in the NIO poller. Users won't see a "Connection Refused" error, but they will experience extreme timeouts because their request is sitting in a queue rather than being processed.

Additionally, every open connection consumes a file descriptor in the operating system. If maxConnections exceeds the OS limit (often 1,024 by default on some Linux distributions), Tomcat will fail to accept new sockets regardless of the server.xml settings.

Verification and Diagnostics

To verify if your tuning is working, you can monitor the active thread count using JConsole or JVisualVM. Look for the catalina-exec- thread prefix defined in the Executor.

To check if the OS is dropping connections due to the acceptCount being exceeded, run the following command on the host machine (requires sudo/root):

ss -nlt

Check the Send-Q column for your Tomcat port. If the Send-Q is consistently equal to the Listen-Q (the backlog), your acceptCount is too low or your maxThreads are fully saturated, causing a bottleneck.

Rollback Procedure

If the server becomes unstable or encounters OutOfMemoryError after these changes, revert the server.xml to the default values: remove the <Executor> block and remove the executorName, maxConnections, and acceptCount attributes from the <Connector> element. Restart the Tomcat service to apply changes.