Architectural Note: Using Virtual Threads for High‑Concurrency I/O in Java

Requirements

For I/O-bound services that need to handle many concurrent connections, the goal is to keep the thread model simple while avoiding the memory and scheduling cost of OS threads. The design must preserve Java Memory Model guarantees, allow existing blocking APIs to work, and keep heap usage predictable.

Smallest Suitable Design

Replace a fixed-size ThreadPoolExecutor with a VirtualThreadPerTaskExecutor. This lets the JVM mount millions of virtual threads onto a small carrier pool (default ForkJoinPool common pool). No pool size tuning is needed for pure I/O waits.

import java.util.concurrent.Executors;
import java.time.Duration;

// Run in JDK 21+ with standard JVM permissions
try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    for (int i = 0; i < 50_000; i++) {
        executor.submit(() -> {
            // Simulate a blocking I/O call
            Thread.sleep(Duration.ofSeconds(1));
            return \"Done\";
        });
    }
}

Trust and Data Boundaries

Virtual threads follow the Java Memory Model, so visibility rules are unchanged. The main shift is that thread stacks live in the Java heap instead of native memory. Therefore:

• ThreadLocal: each virtual thread gets its own copy; large ThreadLocal values can cause heap exhaustion when scaling to hundreds of thousands of threads. Prefer ScopedValue for short‑lived data sharing.
• Blocking operations: the JVM unmounts the virtual thread during standard java.io, java.net, and java.nio calls. Third‑party native libraries or code inside a synchronized block can pin the carrier thread.

Operational Checks and Failure Modes

The most common failure mode is pinning. A virtual thread becomes pinned when it cannot be unmounted from its carrier thread.

1. Inside a synchronized block or method.
2. Executing a native method via JNI.

If many threads pin, the carrier pool exhausts and new tasks stall, resembling a saturated platform‑thread pool.

Diagnostic Decision Matrix

Symptom	Likely Cause	Diagnostic Action
High CPU, low throughput	CPU‑bound workload	Verify that tasks are doing heavy computation instead of waiting for I/O.
Low CPU, stalled processing	Carrier thread pinning	Run jcmd <pid> Thread.dump_to_file and search for the word pinned.
Rapid heap growth	ThreadLocal bloat	Analyze a heap dump for many ThreadLocalMap instances.

When to Change the Design

Virtual threads give no advantage when the workload does not spend time waiting.

• CPU‑bound tasks (e.g., image processing, cryptographic hashes) – use a fixed‑size platform thread pool tuned to core count.
• Ultra‑low latency paths where the extra heap allocation and scheduler overhead outweigh the benefit – consider a dedicated thread per core or an event‑loop framework.

Verification of Result

To confirm that virtual threads are behaving as expected:

1. Enable JDK Flight Recorder and record the event jdk.VirtualThreadPinned. A low frequency indicates proper unmounting.
2. Compare throughput and memory footprint against a ThreadPoolExecutor with a large queue under a simulated load of 20 000 HTTP client GET requests (using java.net.http.HttpClient).
3. Check carrier thread count with jcmd <pid> VM.info – it should stay close to the number of available cores plus a small surplus.