Diagnosing Jetty Thread Pool Exhaustion in Dropwizard Applications

Recognizable condition

Your Dropwizard service starts returning high latency or timeout errors on the /healthcheck endpoint, and clients see "Connection Refused" or prolonged wait times. Under load, the admin metrics show the Jetty thread pool staying near its configured limit while the request queue grows.

Useful takeaway

By checking Jetty‑specific metrics, examining thread states, and reviewing blocking I/O in resource methods, you can confirm thread‑pool exhaustion and apply a targeted fix—either tuning the pool, off‑loading blocking work, or adjusting the queue size—before considering larger architectural changes.

Cause / diagnostic table

Observed symptom	Likely cause
/healthcheck latency spikes, occasional 502/504	Jetty worker threads all busy (jetty.threads.busy ≈ maxThreads)
Steady growth of jetty.queue.size under constant load	Incoming requests exceed processing capacity; queue acting as back‑pressure
Increased heap usage, many request objects retained	Requests queued longer than usual, holding onto buffers and context
Thread dump shows many threads in BLOCKED or WAITING on JDBC/I/O calls	Blocking operations inside resource methods starve the pool

Ordered checks

1. Verify Jetty metrics

   Access the admin port (default 8081) and pull the metrics:

   curl -s http://:/admin/metrics | grep -E 'jetty\.(threads\.busy|queue\.size)'

   Look for jetty.threads.busy approaching the configured maxThreads and a non‑zero jetty.queue.size. No special privileges are needed beyond network access to the admin endpoint.

   Risk: None; read‑only.

2. Capture a thread dump

   Identify the Java process (e.g., via ps -ef | grep java) and run:

   jstack -l  > /tmp/dropwizard-threaddump.txt

   Search for threads with state BLOCKED or WAITING and note the stack trace (often JDBC, file I/O, or external service calls). Requires permission to attach to the JVM (usually the same user running the process).

   Risk: Minimal; dumping threads pauses the VM briefly.

3. Inspect resource methods for blocking I/O

   Review any JAX‑RS resources that invoke synchronous database drivers, HTTP clients, or file operations without off‑loading to an asynchronous executor. Look for calls like Connection.prepareStatement or InputStream.read directly in the request thread.

4. Run a controlled load test

   Using a tool such as k6 or JMeter, generate a steady request rate (e.g., 100 rps) for a few minutes while observing the same metrics. Note the point where jetty.queue.size begins to rise.

   Ensure the test runs from a separate host to avoid contaminating the server’s resources.

   Risk: May exacerbate the symptom; stop the test if error rates climb >5 %.

Fixes tied to findings

If metrics show busy threads ≈ maxThreads and queue growing

• Increase maxThreads cautiously: edit the server block in your Dropwizard YAML:

server:
  type: simple
  applicationContextPath: /
  connector:
    type: http
    port: 8080
  adminConnector:
    type: http
    port: 8081
  maxThreads: 250   # was 150
  minThreads: 20
  acceptQueueSize: 1024
After changing, restart the service. Verify that under the same load jetty.threads.busy stays comfortably below the new limit (e.g., < 70 %).
Limitation: More threads increase memory usage and context‑switch overhead; monitor heap and CPU.
If thread dump reveals blocking on external resources

Off‑load blocking work: wrap the call in a Managed ExecutorService or use Dropwizard’s Async support.

@Path("/orders")
@Produces(MediaType.APPLICATION_JSON)
public class OrderResource {
    private final ExecutorService executor;
    public OrderResource(ExecutorService executor) { this.executor = executor; }

    @GET
    @Path("/{id}")
    public CompletableFuture getOrder(@PathParam("id") String id) {
        return CompletableFuture.supplyAsync(() -> {
            // blocking JDBC call
            return dao.findById(id);
        }, executor).thenApply(Response::ok);
    }
}
Configure the executor size based on the expected I/O latency (e.g., a pool of 20‑30 threads for DB calls). This keeps Jetty threads free to accept new requests.
Verification: After redeploy, repeat the thread dump; the previously blocking threads should now appear in TIMED_WAITING on the executor’s queue, not in BLOCKED on JDBC.
If queue size remains high even after thread increase

Reduce acceptQueueSize to make failures visible earlier, prompting clients to back‑off or retry.

acceptQueueSize: 64   # lower from 1024
This trades a higher early error rate for lower latency spikes. Monitor client error rates to ensure they stay within SLA.
Escalation criteria
If after applying the above adjustments you still observe:

jetty.threads.busy consistently at the new maxThreads under normal load,
Growing heap due to queued request objects, or
Frequent timeouts despite increased threads and off‑loaded I/O,

consider:

Switching to an asynchronous server engine (e.g., Netty) via Dropwizard’s server.type: async.
Evaluating whether the workload is truly CPU‑bound; if so, add more instances behind a load balancer rather than enlarging a single thread pool.
Reviewing garbage‑collection logs for pauses that could be inflating perceived latency.

At this stage, involve the platform or performance‑engineering team for deeper profiling (async-profiler, Java Flight Recorder) and capacity‑planning exercises.