Worker thread contention in high-concurrency Ballerina runtimes

Answer to the Core Questions

1. Prioritization when workers exceed OS threads

The Ballerina runtime keeps a fixed‑size worker thread pool (by default one thread per CPU core). Active worker strands are mapped to these threads via a work‑stealing queue. When the number of active strands is larger than the pool, the scheduler does not apply explicit priorities. Each thread processes strands from its local deque in FIFO order; if a thread becomes idle it steals work from the deques of other threads. Thus, all strands are treated equally and the scheduling order is driven by the order in which they become ready.

2. Preemption behavior for CPU‑intensive vs. I/O‑resumed strands

Ballerina uses cooperative preemption. A running strand continues until it reaches a yield point – e.g., after a loop iteration, a function call that contains an await, or an explicit yield statement. There is no OS‑level time slice or priority boost for a strand that has just resumed from an asynchronous I/O suspension; it is placed in the global ready queue and will be scheduled like any other ready strand. A CPU‑intensive strand that never yields will hold onto its thread until it finishes or encounters a yield, potentially blocking other strands.

Confirmed Facts (from the runtime design)

• Fixed worker thread pool size (configurable via ballerina.runtime.worker.count).
• Work‑stealing queue with FIFO selection per thread.
• No explicit priority levels or time‑slice preemption.
• Yield points are mandatory for cooperative context switches.
• I/O resumption does not confer higher priority.

Likely Explanation for Observed Latency

If you notice high latency or thread starvation under heavy load, it is usually due to CPU‑bound strands that lack sufficient yield points. Even with many workers, a single non‑yielding strand can monopolize a thread, preventing other strands from progressing.

Steps to Mitigate Thread Contention

1. Review and adjust the worker pool size
   Set ballerina.runtime.worker.count to match the number of logical cores (or a small multiple if you have many lightweight strands). Avoid over‑provisioning beyond the core count unless you have evidence that the workload is primarily I/O‑bound.
2. Insert explicit yield points
   In long CPU loops or heavy computation, add yield or await check statements every few iterations to allow the scheduler to switch strands.
3. Enable observability metrics
   Expose the runtime metrics endpoint (e.g., http://localhost:9090/metrics) and monitor:
   • Number of active strands
   • Worker thread utilization (CPU vs. idle)
   • Yield point hits per strand
4. Profile the application
   Use JVM tooling (e.g., jstack or jcmd) to verify that CPU‑intensive strands are not blocking threads while I/O strands are parked.
5. Iterate
   Adjust the worker count or add more yield points based on the metrics until latency stabilizes.

Missing Diagnostic Detail

Could you confirm the current value of ballerina.runtime.worker.count and the approximate number of active worker strands during peak load? This information will help fine‑tune the recommendations.

Diagram

Component	Description
Worker Thread Pool	Fixed OS threads mapped to Ballerina workers
Work‑Stealing Queue	FIFO deque per thread, with idle threads stealing work
Yield Point	Cooperative context switch trigger
I/O Resumption	Ready‑queue insertion, no priority boost