Unexpected latency in Dart isolate despite low CPU usage in VM Service

Which metric isolates the dominant contributor?

Start with async gap duration, not CPU samples. The three metrics answer different questions, and only one of them measures the thing you actually observe—wall-clock time:

• CPU samples tell you where Dart code spent cycles. You already know this is near zero, so the sampler has done its job: it has ruled out Dart execution as the cause. Low samples plus high latency is itself a diagnosis—the isolate is waiting, not computing.
• Async gap duration measures the wall-clock span between a Dart suspension point (an await) and resumption. This is where I/O waits, platform-channel round trips, timer delays, and event-loop scheduling delays all land. It is the metric that most directly accounts for the missing time.
• GC event count (and pause durations in the GC tab) is a secondary check. GC pauses inflate wall-clock latency without showing up as Dart function time, so a quick look rules memory pressure in or out—but it rarely explains sustained latency on its own.

So the practical order is: confirm CPU samples are low (done), check GC pauses to exclude them, then attribute the remainder to async gaps and break those gaps down by source.

Separating platform-channel callbacks in the timeline

In the DevTools timeline, enable the full event stream rather than the default sampled view, then:

1. Record in profile mode (flutter run --profile or an AOT build with the VM Service enabled) under realistic load. Debug mode inflates async overhead enough to invalidate the comparison.
2. Filter or group timeline events by category. Platform-channel work appears as events on the platform thread and as message-handling events, distinct from Dart execution frames on the isolate. In recent DevTools versions you can select the platform/raster threads separately from the UI isolate, which makes the handoff visible.
3. Instrument the channel boundary yourself if the built-in categories are too coarse: wrap channel invocations with Timeline.startSync('channel:methodName') / Timeline.finishSync() (from dart:developer) so each call appears as a labeled interval you can sum.

Caveat: the exact category names and thread separation vary across DevTools and SDK versions—older versions lump non-Dart work into a generic interval. Verify against your installed version before trusting the labels.

When to target I/O instead of Dart code

There is no authoritative threshold published by the Dart team, so treat this as a heuristic, not a rule: if summed async gap time exceeds roughly 20–30% of total operation latency while Dart CPU time stays negligible, optimization effort belongs on the I/O side—channel batching, fewer round trips, connection reuse, or moving work off a contended thread. Below that, look at scheduling and GC first.

Verifying the attribution

Because sampling frequency and timeline overhead can distort short intervals, cross-check before changing code:

• Replay the identical scenario twice and confirm async gap totals are stable run-to-run; noisy gaps mean your measurement overhead is distorting the result.
• Correlate with OS-level evidence: pidstat -w 1 -p <pid> shows voluntary context switches consistent with I/O waiting, independent of the VM's own accounting.
• Align GC pause timestamps with isolate suspension events; matching intervals confirm GC as a contributor, non-overlapping ones exclude it.

One detail would sharpen this further: whether the latency is uniform per operation or appears as occasional spikes—spikes point toward GC or scheduling, uniform delay points toward channel or I/O round trips.