Diagnosing Tomcat Connector Thread Exhaustion and Accept-Queue Saturation

The problem: 503s and connection timeouts under load

When a Tomcat 9 or 10 instance begins returning HTTP 503 responses, or browsers hang until they time out, the HTTP connector is a common suspect. The NIO and NIO2 connectors ship with a bounded worker thread pool (maxThreads, default 200) and a bounded accept queue (acceptCount). When every worker thread is busy and the queue is full, the operating system refuses new TCP connections or the client gives up. The useful takeaway: determine whether you are seeing thread pool exhaustion (all workers busy) or accept-queue saturation (backlog full) before changing configuration, because the remedies differ.

Recognizable condition

Externally, the failure is load-dependent. At idle the application responds normally; under concurrent load, some users receive a 503, a connection reset, or a timeout, while others succeed. Internally, access logs show gaps—requests that never reached a servlet—or a spike in 503 entries. catalina.out and localhost.*.log may contain acceptor errors or SocketTimeoutException messages. A thread dump taken during the incident typically shows many worker threads in socketRead or waiting on a downstream resource, with acceptor threads blocked in ServerSocket.accept.

Cause and diagnostic table

Symptom	Likely cause	Diagnostic signal
503s under sustained load; recovery after restart	Thread pool exhaustion: requests are too slow or maxThreads is too low	JMX activeCount equals poolSize and maxThreads; threads stuck in socketRead or waiting on a database
Immediate connection refused; no 503 in access log	Accept-queue saturation: acceptCount too low or acceptor stalled	Missing access-log entries; OS-level connection refused; acceptor thread blocked
Gradual slowdown followed by collapse	Keep-alive connections holding threads; maxConnections too low relative to maxThreads	Many threads in keep-alive state; maxConnections reached before maxThreads
Constant errors regardless of load	Application deadlock or downstream pool exhaustion	Thread dump shows BLOCKED or WAITING on monitors or connection pools

Ordered checks

1. Correlate timing. Compare access-log gaps, catalina.out, and monitoring dashboards to confirm the failure is load-dependent and note the exact window.
2. Capture thread dumps. During the incident, run jcmd <pid> Thread.print > /tmp/tomcat-thread-dump.txt on the Tomcat host (requires the same OS user or root). Take two or three dumps several seconds apart. Look for clusters of threads in socketRead, ServerSocket.accept, or waiting on database pools.
3. Inspect connector metrics via JMX. Connect a JMX client such as JConsole or jvisualvm to the Tomcat JVM. For the ThreadPoolExecutor MBean, compare activeCount, poolSize, and queue size against the configured maxThreads and acceptCount. If activeCount is pinned at maxThreads, the pool is exhausted.
4. Check OS limits and logs. Search catalina.out and localhost.*.log for Too many open files, bind errors, or acceptor exceptions. On Linux, verify file descriptor limits with ulimit -n and the listen backlog with sysctl net.core.somaxconn.
5. Review server.xml. Locate the <Connector> element and record maxThreads, acceptCount, maxConnections, connectionTimeout, and keep-alive settings.

Fixes tied to findings

Finding: thread pool exhaustion with slow downstream calls

If thread dumps show workers waiting on a database or upstream HTTP call, raising maxThreads only delays the failure and adds context-switching overhead. Fix the downstream latency first. If the workload is genuinely I/O-bound and CPU utilization remains moderate, a modest increase may help, but load-test before committing. A representative NIO connector in Tomcat 10 looks like this:

<Connector port="8080" protocol="HTTP/1.1"
           connectionTimeout="20000"
           maxThreads="200"
           acceptCount="100"
           maxConnections="8192"
           keepAliveTimeout="15000"
           maxKeepAliveRequests="100"
           redirectPort="8443" />

Changes to server.xml require a Tomcat restart. JMX tuning of the ThreadPoolExecutor MBean may not persist across restarts and is limited to exposed attributes.

Finding: accept-queue saturation

If connections are refused before reaching a worker, increase acceptCount and ensure maxConnections is at least maxThreads plus headroom. Remember that acceptCount is an operating-system hint, not a guaranteed backlog size; on Linux the effective limit is also governed by net.core.somaxconn. Verify the current value with sysctl net.core.somaxconn and coordinate with infrastructure teams before changing kernel parameters.

Finding: keep-alive connections holding threads

If many threads are held open by idle keep-alive connections, tune keepAliveTimeout and maxKeepAliveRequests. A shorter timeout releases threads faster but may increase connection setup overhead for active clients. Test with a load generator that mirrors real client behavior.

Finding: file descriptor limits

If logs show Too many open files, raise the process limit. For a systemd-managed Tomcat, add LimitNOFILE=65536 to the service unit and reload with systemctl daemon-reload before restarting Tomcat. For shell-started instances, adjust ulimit -n in the startup script.

Escalation criteria

• Application team: thread dumps show application frames in BLOCKED or WAITING state, or a downstream connection pool is exhausted.
• Database or upstream owners: workers are consistently waiting on external calls and latency cannot be reduced by application changes.
• Infrastructure/OS team: file descriptor limits, somaxconn, or network saturation are implicated.
• Escalate further if connector tuning plus downstream fixes do not restore service under a controlled load test.

Verification and limitations

After any change, reproduce the boundary condition with a load generator such as wrk, ab, or JMeter against an endpoint that exercises the same downstream path. Confirm that refusals begin only when concurrent requests exceed maxThreads plus acceptCount, and that recovery is clean when load drops. Limitations: acceptCount behavior varies by platform and connector implementation (APR/native may differ); JMX may not expose queue size directly in all versions; and thread dumps are snapshots, so a single dump can miss transient states. Tomcat 9 uses javax.* package names while Tomcat 10 uses jakarta.*, but the connector element names and diagnostic approach are materially the same across both.