Linux for DevOps, Part 8: Performance & eBPF Observability

The closing part of this series. Every previous post named a subsystem — process scheduler, network stack, block layer, cgroups. This one is about method: a repeatable way to find which subsystem is actually the bottleneck, and the tool (eBPF) that now underlies most modern observability and security tooling.

The USE Method

Brendan Gregg’s framework, extending the flowchart from Part 1: for every resource (CPU, memory, disk, network), check three things.

graph LR
    R["Resource"]
    U["Utilization
% time busy"]
    S["Saturation
work queued, can't
be serviced immediately"]
    E["Errors
failed requests,
retransmits, etc."]
    R --> U
    R --> S
    R --> E

    style S fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f

Saturation is the one people skip and shouldn’t: a CPU at 60% utilization can still be badly saturated if the run queue is deep — utilization alone hides queueing delay.

vmstat 1 5
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
#  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
#  4  0      0 512340  81234 902341    0    0     2    18  245  512 12  3 84  1  0
#  ^ run queue = 4 with only ~4 vCPUs available -> CPU saturation even if %us looks moderate

Working through USE systematically beats guessing — it turns “the app feels slow” into a specific, falsifiable question about one resource at a time.

perf and Flame Graphs

perf samples the CPU at a fixed frequency and records the call stack at each sample — after enough samples, the frequency of a function appearing approximates the time spent in it, without the overhead of tracing every single call.

perf stat -a sleep 10                       # system-wide counters: IPC, cache misses, context switches
perf top                                     # live view, like top but for hot functions
perf record -F 99 -a -g -- sleep 30          # sample at 99Hz, all CPUs, with call graphs
perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg

A flame graph turns thousands of stack samples into one picture: each box is a function, width is proportional to time spent (including children), and stacking shows the call hierarchy. Wide, flat boxes near the top are where the CPU is actually spending its time — that’s where optimization effort pays off, as opposed to functions that are merely called often but individually cheap.

99Hz (not 100Hz) is a deliberate perf convention — sampling at a frequency that doesn’t line up exactly with common periodic system activity (like a 100Hz timer tick) avoids systematically under- or over-sampling those events.

eBPF: Programs Inside the Kernel

Traditionally, observing kernel internals meant either reading fixed /proc counters (Part 1) or writing a kernel module — risky, and requires a reboot or module reload. eBPF (extended Berkeley Packet Filter) lets you load small, verified programs into the kernel at runtime, attached to specific hook points, with no kernel source changes and no crash risk.

graph TB
    SRC["eBPF program
(C, compiled to bytecode)"]
    VER["Verifier
proves: no infinite loops,
no out-of-bounds memory access,
bounded execution"]
    JIT["JIT compiler
compiles to native
machine code"]
    HOOK["Attached to a hook:
syscall entry, kprobe,
tracepoint, XDP, cgroup"]
    MAP["BPF Maps
(shared memory:
kernel <-> user space)"]
    USER["User-space tool
reads results from maps"]

    SRC --> VER --> JIT --> HOOK
    HOOK --> MAP --> USER

    style VER fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f

The verifier is the safety guarantee that makes this workable in production: it statically proves the program terminates and stays within memory bounds before allowing it to load — this is why eBPF programs can’t crash the kernel the way a buggy kernel module can. Maps are how the eBPF program (running in kernel space) and your CLI tool (running in user space) share data — a map might hold a histogram of syscall latencies that a kprobe program updates on every event, which a user-space tool then reads and prints.

bpftrace: One-Liners for Kernel Events

# count syscalls system-wide by name, live, until Ctrl-C
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# every process exec, as it happens
bpftrace -e 'tracepoint:sched:sched_process_exec { printf("%s %s\n", comm, str(args->filename)); }'

# histogram of block I/O latency in microseconds
bpftrace -e 'kprobe:blk_account_io_start { @start[arg0] = nsecs; }
             kretprobe:blk_account_io_done { @us = hist((nsecs - @start[retval]) / 1000); }'

The BCC tools package ships pre-written versions of the common ones:

biolatency -m 10          # 10-second summary, latency in milliseconds
tcplife -p 1234            # only connections from PID 1234

Where eBPF Shows Up in Your Stack Already

You may never write raw eBPF, but you’re almost certainly running it: Cilium implements Kubernetes networking and NetworkPolicy enforcement with eBPF programs attached at the XDP and socket layers instead of iptables rules (Part 4) — faster because packets are processed before they traverse the full netfilter chain. Falco attaches eBPF probes to syscall tracepoints to detect suspicious runtime behavior (a shell spawned inside a container, an unexpected outbound connection) without the overhead of full ptrace-based tracing. Both are, at the implementation level, exactly the bpftrace one-liners above, compiled and packaged as a product.

A Troubleshooting Workflow

For a Kubernetes pod that’s intermittently slow:

1. USE the node first, not the pod — vmstat 1, iostat -xz 1 on the node. A noisy neighbor pod can saturate a shared resource that cgroup CPU/memory limits (Part 6) don’t fully isolate (disk I/O bandwidth, in particular, is only cgroup-limited if io.max is set).
2. Check the cgroup directly — cat .../cpu.pressure, .../memory.pressure, .../io.pressure (PSI — Pressure Stall Information) give a 0–100 score for time spent stalled on each resource, scoped exactly to that pod’s cgroup. This is often faster than reasoning from top/vmstat system-wide numbers.
3. If CPU-bound: perf record/flame graph, scoped with -p <pid> to the container’s process.
4. If I/O-bound: biolatency, iotop, cross-reference with Part 5.
5. If network-bound: tcplife, ss -s retransmit counts, cross-reference with Part 4’s conntrack table size.

cat /sys/fs/cgroup/kubepods.slice/.../cpu.pressure
# some avg10=12.40 avg60=8.15 avg300=3.02 total=48213421
# "12.4% of the last 10s, at least one task in this cgroup was stalled waiting for CPU"

PSI is the single highest-signal number to check first — it directly answers “is this cgroup actually being starved of a resource right now” without needing to correlate several other metrics yourself.

Closing the Series

Eight posts, one thread: every abstraction a DevOps or platform engineer works with daily — containers, Services, PersistentVolumes, resource limits — is a friendly name for kernel primitives covered somewhere in this series. When something breaks, the fastest path to root cause is usually to stop reasoning about the abstraction and go one layer down: check the cgroup, check the namespace, check the syscall. The tools in this series (strace, ss, iostat, bpftrace, nsenter) are how you get there.

← Back to Part 1: Ecosystem, Kernel, and What You Must Know to start the series from the beginning.