Resource contention when multiple Ballerina workers access synchronized shared state

Question

Resource contention when multiple Ballerina workers access synchronized shared state
The worker keyword creates separate execution contexts, each with its own event loop, but shared resources require synchronization across workers.
Scheduling Uncertainty
When several workers attempt to lock the same synchronized resource, the runtime scheduler must decide how to allocate execution time, and the underlying JVM configuration may influence this decision.
The precise behavior of the scheduler under high contention and the effect of JVM thread‑priority or heap settings remain unclear, creating a need to verify whether worker count, JVM options, or lock implementation dictate priority.
What is the scheduler's policy for prioritizing workers when they compete for the same synchronized resource? How does JVM configuration such as thread priority or heap size affect worker scheduling under contention? Does increasing the number of workers worsen context‑switch overhead and degrade performance?

Tasadduq Burney · Answer

What the runtime actually does when workers fight for a lock

In Ballerina 2201.x and newer, the worker keyword is a lightweight abstraction that the runtime maps to a Java Thread. The worker scheduler itself is a simple round‑robin dispatcher over the pool of available Java threads; it does not impose any priority or fairness policy on the workers. When several workers try to enter a synchronized block or acquire a ReentrantLock, the JVM's thread scheduler decides which Java thread wins the lock. The Ballerina runtime therefore has no built‑in mechanism to favour one worker over another.

Effect of JVM configuration

Thread priority – The -XX:ThreadPriorityPolicy or the Java thread priority set via Thread.setPriority influences the operating‑system scheduler. If a worker's underlying Java thread is given a higher priority, the OS may schedule it more aggressively, but this is an OS‑level effect and not a Ballerina‑level guarantee.
Heap size – Increasing the heap (e.g., -Xmx4G) does not change lock acquisition order. It can reduce GC pauses, indirectly making lock wait times more predictable, but it does not alter the scheduler's policy.
Thread‑pool size – The property ballerina.runtime.worker.threadpool.size controls how many Java threads the runtime can use. If the number of workers exceeds this pool size, the runtime queues excess workers, adding context‑switch overhead and potentially worsening throughput under contention.

Does more workers always mean worse performance?

Not in isolation. Adding workers can increase parallelism when the workload is I/O‑bound or contains non‑shared work. However, when the shared resource is a synchronized hotspot, every extra worker that tries to acquire the same lock increases the number of threads racing for the monitor. The JVM's lock contention algorithm (biased locking, lightweight locking, or heavyweight locking) will then pay a higher cost per acquisition, and the round‑robin scheduler will spend more time context‑switching between blocked and runnable threads. In practice, you usually see a performance plateau or even a regression once the worker count exceeds the thread‑pool size.

Assumptions and uncertainty

This analysis assumes Ballerina 2201.x or later, where the worker scheduler is round‑robin; earlier versions may differ. It also presumes the lock is a standard Java synchronization primitive—custom lock implementations may have different semantics. OS‑level scheduling policies (Linux CFS vs. Windows) can further influence observed priorities, so results may vary across platforms.

How to verify the policy on your system

Enable tracing – Run the Ballerina program with ballerina.runtime.logging.level=TRACE. Log entries that show worker lock acquisition will reveal the actual order.
Inspect thread states – Use Java's ThreadMXBean (e.g., jcmd <pid> Thread.print) to see which threads are BLOCKED on the same monitor.
Measure context switches – On Linux, pidstat -w or perf stat can report context‑switch counts. Compare runs with 2 workers vs. 8 workers.
Experiment with thread priority – Start the JVM with -XX:ThreadPriorityPolicy=2 and observe any change in the ordering of lock acquisition in the trace logs.
Check pool size – Print ballerina.runtime.worker.threadpool.size at runtime and ensure that workerCount <= poolSize to avoid queuing.

What to do next

If you find that lock contention is a bottleneck, consider:

Replacing the synchronized block with an AtomicInteger or a lock‑free data structure.
Reducing the number of workers that touch the same resource, or partitioning the state so each worker has its own slice.
Tuning the thread‑pool size to match the number of CPU cores and the expected parallelism.

Missing diagnostic detail

Could you tell me the exact number of workers you are creating relative to the runtime thread‑pool size? This will help determine whether you are already queuing workers and incurring extra context switches.

Resource contention when multiple Ballerina workers access synchronized shared state