Diagnosing and Fixing Vert.x Event‑Loop Starvation

Recognizing Event‑Loop Starvation

In a Vert.x application you’ll notice a sudden, step‑wise increase in request latency, timeouts, or a spike in the vert.x-eventloop-thread-* stack traces. The log line Thread blocked for … ms is a clear indicator that the event‑loop is being held up. The thread pool may be busy while CPU usage is low, and throughput collapses.

Symptom Checklist

• Latency p99 climbs by 200–300 ms in a single step.
• Requests start timing out on a previously healthy endpoint.
• Logs contain Thread blocked for … ms from the Vert.x checker.
• CPU usage is < 10 % while request rate is high.
• Throughput drops dramatically; error rates rise.

Diagnostic Workflow

1. Enable the Blocked‑Thread Checker
   Configure VertxOptions in your main or vertx.conf:

   VertxOptions options = new VertxOptions()
     .setBlockedThreadCheckInterval(1000)   // ms
     .setEventLoopExecuteTime(500);        // ms
   Vertx vertx = Vertx.vertx(options);

   This forces a warning if any event‑loop thread runs longer than 500 ms.
2. Collect Metrics
   During a load test that mimics production traffic, record:
   • p99/p999 latency
   • blocked thread count (via vertx.metrics or JMX)
   • CPU and I/O wait from jstat or sar
3. Take Thread Dumps
   When a stall is observed, run jstack -l <pid> > dump.txt several times. Search for vert.x-eventloop-thread-* frames. The frames should point to your blocking library (e.g., java.sql.DriverManager.getConnection or java.net.SocketInputStream.read).
4. Inspect GC and CPU
   Check the GC logs for long pauses. If GC pauses dominate the event‑loop time, the issue may be memory pressure rather than code. Use -XX:+PrintGCDetails and -XX:+PrintGCDateStamps.
5. Trace the Blocking Call Path
   Map the request flow from the HTTP handler to the library call that appears in the thread dump. Verify that the call is synchronous and that the library has no async API.

Common Causes & Fixes

Cause	Diagnostic Indicator	Fix
Synchronous JDBC/HTTP/HTTP‑DNS calls	Thread dump shows java.sql or java.net frames	Use Vert.x AsyncHttpClient or vertx-jdbc-client; move to executeBlocking or a worker verticle.
Large JSON or payload parsing	Frames in com.fasterxml.jackson or org.json	Stream the body with HttpServerRequest.handler() or use JsonObject streaming; cap body size with maxBodySize.
Cache load on first access	Frames in java.util.concurrent.ConcurrentHashMap or custom loader	Pre‑warm caches; load asynchronously with a timeout; use a bounded thread pool.
Long GC pauses	GC logs show > 200 ms pauses; CPU low but event‑loop blocked	Increase heap, tune GC options, or reduce object churn.
Saturated event‑loop pool (all threads busy)	All vert.x-eventloop-thread-* show blocked frames	Remove blocking calls first; only then adjust eventLoopPoolSize if needed.

Example: Moving a JDBC Call to a Worker Verticle

// Handler that was blocking
router.route("/data").handler(routingContext -> {
  Connection conn = DriverManager.getConnection(url, user, pass); // blocks
  ResultSet rs = conn.prepareStatement("SELECT * FROM table").executeQuery();
  // ... build response
});

// Refactored using a worker verticle
router.route("/data").handler(routingContext -> {
  vertx.executeBlocking(future -> {
    try (Connection conn = DriverManager.getConnection(url, user, pass)) {
      ResultSet rs = conn.prepareStatement("SELECT * FROM table").executeQuery();
      // build result
      future.complete(result);
    } catch (SQLException e) {
      future.fail(e);
    }
  }, res -> {
    if (res.succeeded()) {
      routingContext.response().end((String) res.result());
    } else {
      routingContext.fail(res.cause());
    }
  });
});

Example: Using Vert.x Async HTTP Client

HttpClient client = vertx.createHttpClient();
client.getAbs("https://api.example.com/data").send(ar -> {
  if (ar.succeeded()) {
    HttpClientResponse resp = ar.result();
    resp.bodyHandler(body -> routingContext.response().end(body.toString()));
  } else {
    routingContext.fail(ar.cause());
  }
});

When to Escalate

• Blocked‑thread warnings persist after all blocking calls have been removed.
• Stalls correlate with GC or OS metrics rather than application frames.
• Clustered event bus or HA failover is implicated (e.g., vertx.clusteredClient stalls).
• Fix requires a Vert.x or library upgrade that cannot be tested locally.

Verification Checklist

1. Re‑run the load test with the checker enabled.
2. Confirm that p99 latency returns to baseline and no blocked‑thread warnings appear.
3. Validate that the event‑loop thread dump no longer shows the offending frames.
4. Check that GC pauses are within acceptable limits (< 100 ms).
5. Ensure that the worker pool usage stays below 80 % to avoid exhaustion.

Version Sensitivity

Defaults for blockedThreadCheckInterval, eventLoopExecuteTime, and worker semantics differ between Vert.x 4.x and 5.x. Verify the VertxOptions values for your pinned version and adjust thresholds accordingly. Virtual threads in Vert.x 5.x offer a different trade‑off; they still block the event‑loop if used directly, so keep the checker enabled.

Limitations & Caveats

• Disabling the checker or raising thresholds in production hides the root cause and can mask regressions.
• Wrapping every handler in executeBlocking is a quick fix but can exhaust the worker pool and break ordering guarantees.
• Thread dumps and profilers can alter timing; take multiple samples and compare to a baseline.
• Increasing eventLoopPoolSize rarely resolves starvation; it often reduces throughput due to context switching.