MMC Request Latency Tracing: Absent Kernel Tracepoint for SDHCI Completion Events

To isolate SDHCI controller latency from block-layer and filesystem overhead on a Raspberry Pi 4, you can use kprobes to dynamically instrument the kernel. Since a native tracepoint for mmc_request_done is absent, kprobes allow you to capture timestamps at the exact moment the MMC core processes a request completion, bypassing the need for an out-of-tree module.

Implementation Strategy

A minimal kprobe approach is preferred over a custom module to avoid altering the SDHCI driver's interrupt path. By attaching probes to the entry and exit of the MMC request completion functions, you can calculate the delta between the hardware interrupt and the block-layer completion signal.

Verification and Setup Steps

1. Eliminate CPU Noise: Set the scaling governor to performance to prevent frequency drops during I/O wait from inflating latency measurements.

   echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

2. Verify Symbols: Confirm the mmc_request_done symbol is available in your current kernel build.

   sudo perf probe -l mmc_request_done

3. Instrument the Path: Use perf probe to add a probe to the completion function. This captures the event without the overhead of a full kernel module.

   sudo perf probe -a mmc_request_done

4. Capture Latency: Run your fio workload while recording the probe events to analyze the distribution of completion times.

Analysis: Controller vs. Card Latency

It is possible to distinguish between host-side scheduler jitter and SD card internal wear-leveling by analyzing the latency distribution (histogram):

• Scheduler Jitter: Typically manifests as a consistent shift in the median latency or small, frequent fluctuations across all requests.
• Wear-Leveling/GC: Characterized by a "long-tail" distribution, where the vast majority of requests are fast, but occasional spikes occur (e.g., 100ms+), regardless of the block-layer queue depth.

Assumptions and Limitations

This approach assumes the mmc_request_done symbol is not inlined by the compiler in your specific kernel version. If perf probe fails to find the symbol, you may need to check /proc/kallsyms. Note that while --direct=1 in fio removes page-cache interference, it does not bypass the MMC driver's internal queue.

Required Diagnostic Detail: Please provide the specific kernel version (e.g., uname -a). If you are using a real-time (PREEMPT_RT) patched kernel, the interrupt latency profile will differ significantly, requiring a different probe placement in the ISR.