Practical eBPF Tracing with bpftrace for Linux System Call Profiling

The tracing gap in production Linux services

When a service becomes unresponsive or CPU usage spikes, reaching for strace often means stopping the process or sifting through noisy, full-system output. Built-in monitors show totals but not which process is making which call. This limits your ability to pinpoint bottlenecks without invasive debugging.

eBPF and bpftrace: a safe sandbox for kernel observation

eBPF (extended Berkeley Packet Filter) programs run in a verified sandbox inside the kernel, allowing custom tracepoints, network filters, or security checks without changing kernel code or rebooting. The kernel verifier checks safety before a program is JIT-compiled and loaded, making it suitable for production use when constraints are respected.

A concrete bpftrace example: counting openat syscalls

bpftrace is a high-level language that compiles to eBPF and lets you write short programs for real-time tracing. The following one-liner counts openat system calls per process ID:

bpftrace -e 'uprobe:/usr/bin/find:entry { @openat[pid] = count(); }'

Running the example: where, permissions, and checks

• Where to run: a terminal on the target Linux host
• Required permissions: root or CAP_SYS_ADMIN capability is needed to load eBPF programs
• Meaningful placeholder: replace /usr/bin/find with the absolute path of the binary you want to trace
• Expected check: after running the command, bpftrace will emit a table of process IDs and their openat call counts until you stop it (Ctrl-C)
• Relevant risk: the kernel verifier may reject programs that exceed instruction or map limits; complex programs often require refactoring or map-based data storage

Limitations and practical verification

Before relying on eBPF tracing, verify your environment supports it:

• Check kernel version and eBPF support: uname -r and grep EBIT /proc/version (or inspect /sys/kernel/debug/tracing/ for tracing-related files)
• Load a simple one-liner to confirm the setup works: bpftrace -e 'tracepoint:sched:sched_switch { }' should complete without errors on stdout
• Inspect loaded programs and maps: ls /sys/fs/bpf/ and cat /sys/fs/bpf/trace_raw_output to confirm activity and map existence

Version assumption: this guidance applies to kernels 4.1 and newer with CONFIG_EBPF enabled, which covers most actively maintained Linux distributions. Older kernels may lack the verifier or feature set.

Actionable next steps

Start with the one-liner above on a non-critical service, inspect the per-process output, and iterate the program if you need to trace different syscalls or aggregate by user, path, or time window. eBPF tracing is most effective when you begin with a focused question, verify kernel support first, and keep programs small enough for the verifier to accept.