Reliability of %util for detecting storage saturation in iostat

Question

Evaluating I/O Bottlenecks with sysstat
When analyzing storage performance using iostat -x, the %util metric indicates the percentage of time the device had at least one request outstanding. While this is a primary indicator for legacy spinning disks, modern storage architectures—including NVMe SSDs, RAID arrays, and virtualized cloud volumes—can process multiple I/O requests in parallel.
On these parallel-capable devices, a %util value of 100% does not necessarily imply that the device has reached its maximum throughput or IOPS capacity, as the device may still have available internal queues to handle additional requests.
To determine actual saturation, engineers often correlate %util with aqu-sz (average queue depth) and await (average latency). However, there is no universal threshold defined in the documentation to signal when a device is truly saturated versus merely active.
Which combination of await and aqu-sz provides a more reliable saturation signal than %util on parallelized storage? Does a rising aqu-sz coupled with stable await indicate available headroom despite high utilization?

Siddharth · Answer

Answer
No. On parallel-capable storage, %util alone is not a reliable saturation signal. A more reliable signal is a sustained increase in await that tracks with a growing aqu-sz while %util is high.
Yes. A rising aqu-sz coupled with stable await indicates available headroom despite high %util. Queue depth is increasing but service time is not degrading, so the device is absorbing more parallel requests without queuing delay.
Confirmed facts

%util in iostat -x is the percentage of time the device had at least one request outstanding during the sampling interval. It is not a direct measure of throughput or IOPS utilization.
aqu-sz is the average queue size over the interval.
await is the average time in milliseconds for an I/O request to be issued to completion, including time spent waiting in queue and service time.

Likely explanation
On NVMe SSDs, RAID arrays and virtualized cloud volumes, devices can service multiple requests in parallel with internal queues. %util can reach 100% while the device still has internal parallelism available, so %util alone is a weak saturation indicator for these architectures.
Saturation tends to appear as service time degradation under load. That is observed as await rising in step with aqu-sz. If aqu-sz rises but await stays near baseline, the device is using its parallelism and has headroom even with %util at or near 100%.
Steps needed for this case

Baseline the device under normal load with iostat -x 1 and record %util, aqu-sz and await for the target device.
Under representative load, watch for the pattern: high %util + growing aqu-sz + stable await = active but not saturated; high %util + growing aqu-sz + rising await = saturation.
Compare observed await trend to the idle or low-load baseline for the same workload type before declaring saturation.

Note: await and aqu-sz are interval averages and can be skewed by bursts. Virtualized and cloud volumes hide host queueing, so device-level metrics may not reflect true end-to-end saturation.

Reliability of %util for detecting storage saturation in iostat

Evaluating I/O Bottlenecks with sysstat

1 answer

Answer

Confirmed facts

Likely explanation

Steps needed for this case

Your perspective belongs here.

0 question comments