Freeing Tomcat Worker Threads with Servlet 3.0 Asynchronous Processing

Problem: Blocking Servlets Consume Worker Threads

Under heavy load, a classic servlet that performs a long‑running operation (e.g., calling an external API or processing a large file) holds a Tomcat worker thread for the entire duration of that work. While the thread is blocked, it cannot accept new requests, which reduces throughput and increases latency even if the actual CPU work is minimal.

Thesis: Detach the Request from the Worker Thread

Servlet 3.0 introduced asynchronous request processing. By marking a servlet as asyncSupported=true and calling request.startAsync(), the original worker thread is returned to the pool immediately. A separate thread (or a CompletableFuture) performs the long‑running task and later calls AsyncContext.complete() to send the response. This frees Tomcat’s worker threads while still allowing the background work to finish.

Worked Example: An Async Servlet with CompletableFuture

The following servlet simulates a 5‑second delay using a background thread pool. It can be placed in any web application targeting Tomcat 9.0+.

package com.example.async;

import jakarta.servlet.AsyncContext;
import jakarta.servlet.ServletException;
import jakarta.servlet.annotation.WebServlet;
import jakarta.servlet.http.HttpServlet;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

@WebServlet(urlPatterns = "/async-demo", asyncSupported = true)
public class AsyncDemoServlet extends HttpServlet {
    private final ExecutorService background = Executors.newFixedThreadPool(10);

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws ServletException, IOException {
        // Detach the request from the worker thread
        AsyncContext asyncCtx = req.startAsync();
        asyncCtx.setTimeout(30_000); // 30 s safety timeout

        // Off‑load the long‑running work
        CompletableFuture.runAsync(() -> {
            try {
                // Simulate work – replace with real logic
                Thread.sleep(5000);
                asyncCtx.getResponse().getWriter().write("Done after 5 s\n");
            } catch (Exception e) {
                asyncCtx.getResponse().setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
            } finally {
                asyncCtx.complete(); // Must be called to avoid leaks
            }
        }, background);
    }

    @Override
    public void destroy() {
        background.shutdownNow();
        super.destroy();
    }
}

If you prefer XML configuration, add the following to WEB-INF/web.xml:

<servlet>
    <servlet-name>AsyncDemo</servlet-name>
    <servlet-class>com.example.async.AsyncDemoServlet</servlet-class>
    <async-supported>true</async-supported>
</servlet>
<servlet-mapping>
    <servlet-name>AsyncDemo</servlet-name>
    <url-pattern>/async-demo</url-pattern>
</servlet-mapping>

Monitoring and Verification

To confirm that Tomcat worker threads are not blocked:

1. Start Tomcat with the manager app enabled (${CATALINA_HOME}/conf/tomcat-users.xml must contain a user with the manager-gui role).
2. Deploy the WAR containing the servlet.
3. Open http://localhost:8080/manager/html and navigate to the “Thread Pool” view.
4. Using a tool like curl or ApacheBench, send a burst of requests, e.g.:

ab -n 50 -c 10 http://localhost:8080/your-context/async-demo

While the benchmark runs, observe the “Current threads busy” metric. It should stay close to the configured maxThreads (e.g., 200) rather than climbing to the number of concurrent requests, indicating that worker threads are released quickly.

Additionally, you can monitor via JConsole:

• Connect to the Tomcat JVM.
• Examine java.lang:type=Threading → ThreadCount and DaemonThreadCount.
• Watch for a steady count while the background tasks complete.

Check the Tomcat logs (${CATALINA_BASE}/logs/catalina.out) for any warnings such as “AsyncContext not completed” – their absence indicates proper cleanup.

Trade‑offs and Limitations

While async processing frees Tomcat’s worker threads, it introduces several considerations:

• Code complexity: You must manage the AsyncContext lifecycle and ensure complete() (or a timeout) is always called.
• Thread‑safety: Shared objects accessed from the background thread must be safely published or confined.
• Filter/listener compatibility: Some servlet filters assume a synchronous request and may not behave correctly when the request is dispatched asynchronously. Test each filter in an async scenario.
• Memory overhead: Each active AsyncContext holds references to the request and response objects until completion, increasing heap usage proportional to concurrent long‑running requests.
• Executor sizing: The background work still consumes threads; you must provision an executor (e.g., a fixed pool) sized for the expected workload.

Actionable Closing

If your application experiences thread exhaustion due to blocking I/O or external calls, enable Servlet 3.0 async processing on the affected servlets. Start with a small proof‑of‑concept like the example above, verify thread‑pool behavior via the manager app or JConsole, and then gradually migrate other long‑running endpoints. Remember to size your background executor appropriately, audit filters for async safety, and monitor logs for incomplete async contexts to avoid resource leaks.