Worker exhaustion during blocking I/O calls in Swan Lake runtime

Worker Pool Saturation and Queuing

In the Swan Lake runtime, there is no single, static numerical threshold that triggers request queuing across all environments. Instead, queuing begins the moment the available worker pool is fully saturated by active strands. Because blocking I/O operations tie up a worker thread for the duration of the call, the threshold is dynamically defined by the number of configured workers relative to the number of concurrent blocking calls.

Request Prioritization during Saturation

When all available workers are blocked by synchronous I/O, the runtime does not employ a complex priority-based scheduling system for new incoming requests. Instead, new requests are placed into a queue. These requests remain pending until a worker is released by a completing I/O operation. This results in a first-in, first-out (FIFO) behavior for the queue, which manifests as increased latency (tail latency spikes) rather than immediate connection refusals or runtime exceptions.

Likely Explanation vs. Confirmed Behavior

Confirmed Behavior: Blocking I/O in a worker-based model degrades the efficiency of the event loop. If the worker pool is exhausted, the system stops processing new strands until existing ones terminate or yield.

Likely Explanation: The "silent" nature of this exhaustion occurs because the network layer may still accept TCP connections, but the runtime cannot assign a worker to the resulting strand, leading to requests timing out at the client level while the server appears "healthy" from a process-monitoring perspective.

Mitigation Steps

To prevent total service unavailability during I/O spikes, follow these configuration and architectural steps:

1. Adjust Worker Limits: Modify the ballerina.toml file to increase the worker pool size if the hardware has sufficient CPU cores to handle the context switching.
2. Implement Timeouts: Ensure all blocking I/O calls have strict timeouts to prevent a single stalled backend from permanently occupying a worker.
3. Offload Blocking Calls: Where possible, refactor synchronous calls to use non-blocking alternatives provided by the Ballerina standard library to allow the worker to yield during I/O wait times.

Verification

To verify the saturation point, you can use a load-testing tool (such as k6 or JMeter) to gradually increase concurrent requests while monitoring the following:

• Latency: A sharp, non-linear increase in response time indicates the transition from active processing to queuing.
• Thread Count: Use OS-level tools (e.g., top -H or jstack if applicable to the underlying JVM) to observe if the worker thread count has hit the configured limit.

Missing Diagnostic Detail: To provide a more precise ballerina.toml recommendation, please provide the current worker_limit setting and the average CPU core count of your deployment environment.