Async/await latency spikes under concurrent requests in Vala

Likely explanation

The latency spikes are most often caused by serialization inside a shared GMainContext. All Vala async continuations are resumed via GTask callbacks that are dispatched by the same main-loop thread. When many tasks complete at nearly the same time, the context's wake-up pipe/eventfd is written many times, the ready-list is rebuilt, and each callback is run one after another. A single slow or blocking callback (even a short g_usleep or I/O) stalls the dispatcher, making every other pending request wait, which appears as an unpredictable latency spike without any error log.

Confirmed facts about GMainContext contention

• A GMainContext can be owned/iterated by only one thread at a time; ownership is acquired with g_main_context_acquire and released after each iteration.
• Attaching or removing sources, updating the priority queue, and writing to the wake-up pipe/eventfd all require the context's internal lock.
• When many worker threads push completions into the same context, these operations become serialized, and the dispatch of the ready list is also single-threaded.

Allocator contention (secondary)

Frequent allocations/frees for GTask, closures, GObject instances, or GVariant can stress g_malloc/g_slice and the system allocator, especially when objects are allocated on one thread and freed on another. This usually shows as CPU time in malloc/free or futex waits, not as async-specific latency, unless G_SLICE forces a global lock or arena limits are hit.

Concurrent GObject signal emission

Signal emission adds latency proportional to the number of connected handlers, the complexity of marshalling, any accumulator or return-value handling, and re-entrancy. If handlers do only quick work, emissions scale linearly and rarely cause spikes. Problems arise when handlers block, emit signals re-entrantly, or when classes are being modified while the load runs.

Steps to isolate the cause for this case

1. Give each worker thread its own GMainContext, acquire it, set it as thread-default before creating any async tasks, and run the same load.
2. Compare latency (p50/p99) with the original shared/default context run.
3. If spikes disappear, the shared context's wake-up/dispatch path is the culprit.
4. If spikes remain, profile allocator usage (e.g., G_SLICE=always-malloc, MALLOC_ARENA_MAX) and check signal handlers for blocking work.

One missing diagnostic detail

Are the async callbacks performing any blocking I/O, mutex waits, or lengthy computations? Knowing this changes the recommendation: if they do, move that work off the main context; if they are already non-blocking, focus on context allocation or allocator tuning.