perf_event_paranoid and PMU counter trust in virtualized Linux hosts

Short answer: with perf_event_paranoid set to 2 or higher, an unprivileged user loses system-wide and CPU-wide profiling and hardware PMU events, but keeps per-process profiling of their own tasks using software events and user-space-only sampling. Those software and clock-based events are also exactly the classes that remain comparable between bare metal and virtualized hosts, because they are generated by the kernel rather than by a physical PMU whose presence and fidelity in a guest depend on the hypervisor.

What remains at paranoid >= 2

The paranoid level is a tiered boundary. At the restrictive end you can still:

• Profile your own processes: perf stat and perf record scoped to a PID or command you own.
• Use software events: page-faults, context-switches, cpu-migrations, task-clock, cpu-clock. These come from kernel accounting, not the PMU.
• Use tracepoints you have permission for (sched, syscalls), subject to lockdown mode, seccomp, and container policy, which can restrict them independently of the paranoid setting.
• Sample user-space-only cycles where the kernel permits it (exclude_kernel/exclude_hv semantics vary by kernel version and distro).

The supported way to relax this for a monitoring account without full root is CAP_PERFMON (kernel 5.8+; older kernels required CAP_SYS_ADMIN). That is a policy decision for the host owner, not something to work around.

How the kernel exposes PMU availability differences

Two surfaces tell you what is real:

• /sys/bus/event_source/devices/ — if there is no cpu PMU device with an events directory, the guest has no usable hardware PMU and perf silently falls back to software events.
• perf_event_open failure modes — requesting a hardware event that the guest does not back fails with EOPNOTSUPP/ENOENT/ENODEV, and perf stat reports the event as <not supported>. This is distinct from a permission denial, and telling the two apart is the key diagnostic step.

On virtualized hosts, PMU exposure is a hypervisor configuration choice: many guests see no PMU at all, some see a virtualized PMU with a reduced event set, and counters may be multiplexed across vCPUs or emulated. A virtualized PMU can return plausible-looking numbers that are not accurate, so absolute hardware counter values should not be compared across hosts without validating the source (hypervisor vPMU settings, dmesg PMU initialization lines).

Comparable event classes for triage

When PMU fidelity is uncertain, base bottleneck triage on kernel-generated events:

• task-clock vs wall time: separates CPU-bound behavior from blocked/waiting behavior.
• context-switches and cpu-migrations: contention and scheduling problems.
• page-faults: memory pressure signals.
• sched and syscall tracepoints, where permitted.

Treat cycles, instructions, and cache events as trustworthy only when the PMU device exists and the events return real counts rather than <not supported>.

Verification for your environment

cat /proc/sys/kernel/perf_event_paranoid
ls /sys/bus/event_source/devices/
perf stat -e cycles,instructions,page-faults,context-switches -- <your command>

Run the same perf stat on bare metal and in the VM and diff perf list output to see exactly which hardware events the hypervisor exposes. Note that paranoid-level semantics and defaults vary across kernel versions and distributions, so confirm the level on the actual host rather than assuming a distro default.

One caveat: this reflects general upstream kernel behavior, not an audited specific version — if your triage conclusion hinges on a particular event, verify it on the target kernel before acting on the numbers.