| name | linux-perf-profile |
| description | Deep Linux perf profiling — PMU counters, topdown analysis, flamegraphs, and annotated hotspot drill-down on ARM/Graviton |
Linux Perf Profiling
Deep hardware-level performance analysis using Linux perf on this ARM (Neoverse V1 / Graviton3) system. Goes beyond Criterion benchmarks to explain why code is slow using PMU counters, topdown decomposition, cache/TLB analysis, and annotated disassembly.
When to Use
- Criterion benchmarks show a regression but the cause is unclear
- You need to understand where cycles are spent inside a hot function
- Investigating cache misses, branch mispredictions, TLB pressure, or stall cycles
- Comparing two builds at the microarchitectural level
- Generating flamegraphs for visual hotspot identification
- After
/bench-compare finds a regression and you need to explain it
Prerequisites
Ensure a release build with debug info exists:
RUSTFLAGS="-C target-cpu=native -C debuginfo=2" cargo build --release
The -C debuginfo=2 flag is critical — it enables perf annotate to map samples back to Rust source lines without impacting optimization.
Profiling Modes
Use the mode that matches your investigation. Modes are ordered from broadest to most focused.
Mode 1: Topdown Overview
Get a high-level breakdown of where cycles are going: frontend stalls, backend stalls, or retiring useful work.
perf stat -e cpu_cycles,inst_retired,stall_frontend,stall_backend,stall_backend_mem,br_mis_pred_retired \
./target/release/gossip-worker 2>&1
Derived metrics to compute:
| Metric | Formula | Healthy | Investigate |
|---|
| IPC | inst_retired / cpu_cycles | > 2.0 | < 1.0 |
| Frontend bound % | stall_frontend / cpu_cycles × 100 | < 10% | > 20% |
| Backend bound % | stall_backend / cpu_cycles × 100 | < 30% | > 40% |
| Backend memory % | stall_backend_mem / cpu_cycles × 100 | < 15% | > 25% |
| Branch mispredict rate | br_mis_pred_retired / br_retired × 100 | < 1% | > 3% |
Interpretation guide:
- High frontend stalls → icache misses, large code footprint, bad branch targets. Check iTLB and L1i below.
- High backend stalls → waiting on data. Drill into cache/TLB (Mode 3).
- High backend memory stalls → specifically memory-bound. Check L2/LLC misses.
- Low IPC + low stalls → instruction mix issue (too many µops per instruction, long-latency ops).
Mode 2: Hotspot Sampling + Flamegraph
Record samples and identify which functions consume the most cycles.
2a. Record
perf record -g --call-graph dwarf,32768 -F 4999 \
./target/release/gossip-worker
Flags explained:
-g --call-graph dwarf,32768 — DWARF-based unwinding (works with Rust, unlike frame pointers). 32KB stack dump covers deep async stacks.-F 4999 — ~5000 samples/sec. Use a prime number to avoid aliasing with periodic code.
2b. Report (interactive)
perf report --no-children --sort=dso,symbol --percent-limit=1.0
--no-children shows self time only (not cumulative call-tree cost).--percent-limit=1.0 hides noise below 1%.
2c. Report (text, for Claude analysis)
perf report --no-children --sort=symbol --percent-limit=0.5 --stdio 2>&1 | head -80
Pipe the output here for analysis.
2d. Flamegraph
perf script | inferno-collapse-perf | inferno-flamegraph > flamegraph.svg
If inferno is not installed: cargo install inferno. Alternatively:
perf script > perf.script
Mode 3: Cache & TLB Drill-Down
When topdown shows backend/memory stalls, measure the cache hierarchy.
perf stat -e l1d_cache,l1d_cache_refill,l1d_cache_lmiss_rd,l2d_cache,l2d_cache_refill,l2d_cache_lmiss_rd,dtlb_walk,itlb_walk,mem_access \
./target/release/gossip-worker 2>&1
Derived metrics:
| Metric | Formula | Healthy | Investigate |
|---|
| L1d miss rate | l1d_cache_refill / l1d_cache × 100 | < 5% | > 10% |
| L1d → L2 miss rate | l1d_cache_lmiss_rd / l1d_cache_refill × 100 | < 30% | > 50% |
| L2 miss rate | l2d_cache_refill / l2d_cache × 100 | < 10% | > 20% |
| L2 → LLC/DRAM rate | l2d_cache_lmiss_rd / l2d_cache_refill × 100 | < 20% | > 40% |
| dTLB miss rate | dtlb_walk / mem_access × 100 | < 0.5% | > 2% |
| iTLB miss rate | itlb_walk / l1i_cache × 100 | < 0.1% | > 1% |
Common patterns in this codebase:
- High L1d misses → check struct layout, false sharing in concurrent paths, random-access patterns in
TimingWheel (crates/gossip-stdx/src/timing_wheel.rs) or SetAssociativeCache (crates/scanner-engine/src/lsm/set_associative_cache.rs).
- High L2 misses → working set exceeds L2 (256KB/core on Graviton3). Consider data partitioning or reducing struct sizes.
- High dTLB walks → large heap allocations scattered across pages. Consider hugepages or arena allocation.
- High iTLB walks → code bloat from monomorphization or heavy inlining. Check generic instantiation count.
Mode 4: Branch Analysis
When topdown shows frontend stalls or branch misprediction issues.
perf stat -e br_pred,br_mis_pred,br_retired,br_mis_pred_retired,inst_retired \
./target/release/gossip-worker 2>&1
Derived metrics:
| Metric | Formula | Healthy | Investigate |
|---|
| Speculative mispredict % | br_mis_pred / br_pred × 100 | < 2% | > 5% |
| Retired mispredict % | br_mis_pred_retired / br_retired × 100 | < 1% | > 3% |
| Branch density | br_retired / inst_retired × 100 | < 20% | > 30% |
To find which branches are mispredicting:
perf record -e br_mis_pred_retired -c 1000 -g --call-graph dwarf \
./target/release/gossip-worker
perf report --stdio --percent-limit=1.0 2>&1 | head -60
Mode 5: Annotated Source (Per-Line Costs)
Once you've identified a hot function from Mode 2, drill into it at the source-line level.
perf annotate --symbol=<function_name> --stdio 2>&1
For a specific event:
perf record -e l1d_cache_refill -c 10000 --call-graph dwarf \
./target/release/gossip-worker
perf annotate --symbol=<function_name> --stdio 2>&1
This shows the percentage of samples on each source line / assembly instruction. Look for:
- Lines consuming > 20% of function samples
- Unexpected memory loads (possible bounds checks)
- Spilled registers (function too complex for register allocator)
Mode 6: A/B Comparison
Compare two builds at the hardware counter level to explain a Criterion regression.
Step 1: Baseline counters
git stash push -m "changes"
RUSTFLAGS="-C target-cpu=native -C debuginfo=2" cargo build --release
perf stat -r 3 -e cpu_cycles,inst_retired,stall_frontend,stall_backend,stall_backend_mem,l1d_cache_refill,l2d_cache_refill,br_mis_pred_retired \
./target/release/gossip-worker 2>&1 | tee /tmp/perf-baseline.txt
Step 2: Changed counters
git stash pop
RUSTFLAGS="-C target-cpu=native -C debuginfo=2" cargo build --release
perf stat -r 3 -e cpu_cycles,inst_retired,stall_frontend,stall_backend,stall_backend_mem,l1d_cache_refill,l2d_cache_refill,br_mis_pred_retired \
./target/release/gossip-worker 2>&1 | tee /tmp/perf-after.txt
Step 3: Diff analysis
Compare the two files and compute deltas. Focus on:
- Did cycle count increase? If so, did IPC drop or did instruction count increase?
- If IPC dropped, which stall category increased?
- If instructions increased, was it branch density or memory ops?
Mode 7: Lock Contention & Scheduling
For diagnosing contention in async/concurrent code paths.
perf stat -e context-switches,cpu-migrations,page-faults \
-e sdt_libpthread:mutex_entry,sdt_libpthread:mutex_acquired \
./target/release/gossip-worker 2>&1
For scheduling latency:
perf sched record ./target/release/gossip-worker
perf sched latency --sort max 2>&1 | head -30
Event Groups for Copy-Paste
Pre-built event sets tuned for this system (ARM Neoverse V1, armv8_pmuv3).
Multiplexing note: This virtualized Graviton3 environment exposes ~3 simultaneous hardware counters. Do not use {} group pinning syntax — it will fail with <not supported>. Instead, pass events as a comma-separated list and let perf stat multiplex automatically. The kernel time-shares counters and scales results; the (XX.XX%) annotation next to each counter shows what fraction of runtime it was active. With workloads running ≥1 second, scaled values are reliable. For maximum accuracy on critical ratios, use -r 3 (repeat 3x) and keep groups small (3-4 events).
Set: Topdown
cpu_cycles,inst_retired,stall_frontend,stall_backend,stall_backend_mem,br_mis_pred_retired
Set: L1 Cache
l1d_cache,l1d_cache_refill,l1d_cache_lmiss_rd,l1i_cache,l1i_cache_refill,l1i_cache_lmiss
Set: L2 Cache + TLB
l2d_cache,l2d_cache_refill,l2d_cache_lmiss_rd,dtlb_walk,itlb_walk,mem_access
Set: Branches
br_pred,br_mis_pred,br_retired,br_mis_pred_retired,inst_retired,cpu_cycles
Set: Instruction Mix
inst_retired,inst_spec,op_retired,op_spec,cpu_cycles,stall
Set: Backend Memory
stall_backend,stall_backend_mem,l1d_cache_lmiss_rd,l2d_cache_lmiss_rd,mem_access,cpu_cycles
Precision set (3 events, no multiplexing)
When you need exact ratios without scaling, use ≤3 events:
cpu_cycles,inst_retired,stall_backend
Output Format
Report findings using this structure:
## Perf Profile: [target / scenario]
### Environment
- CPU: ARM Neoverse V1 (Graviton3), 16 cores
- Build: `RUSTFLAGS="-C target-cpu=native -C debuginfo=2"` release
- Target: [repo or benchmark name]
### Topdown Summary
| Metric | Value | Assessment |
|--------|-------|------------|
| IPC | X.XX | [good/investigate] |
| Frontend bound | X.X% | [ok/high] |
| Backend bound | X.X% | [ok/high] |
| Backend memory | X.X% | [ok/high] |
| Branch mispredict | X.X% | [ok/high] |
### Hotspots (top 5 by self%)
| Rank | Symbol | Self % | Module | Likely Cause |
|------|--------|--------|--------|--------------|
| 1 | func_name | XX.X% | gossip_stdx | [explanation] |
### Cache Hierarchy (if relevant)
| Level | Accesses | Misses | Miss Rate | Assessment |
|-------|----------|--------|-----------|------------|
| L1d | X.XXB | X.XXM | X.X% | [ok/high] |
| L2 | X.XXM | X.XXM | X.X% | [ok/high] |
### Root Cause Analysis
[Narrative explaining what the numbers mean for this specific code. Connect
PMU data to source-level patterns. Reference specific lines/functions.]
### Recommendations
1. **[Issue]** at `file:line` — [specific fix with rationale tied to PMU data]
- Expected impact: [which metric should improve and by roughly how much]
- Validate: `perf stat -e <relevant_events> ...`
### Validation Plan
[Commands to re-run after applying fixes to confirm improvement]
Caveats
- Multiplexing: This virtualized environment supports ~3 simultaneous counters.
perf stat multiplexes automatically — the (XX.XX%) annotation shows sampling duty cycle. With ≥1s workloads, scaled values are reliable. Never use {} group pinning.
- DWARF unwinding cost:
--call-graph dwarf adds ~5-15% overhead to the profiled process. For tight latency measurements, use perf stat instead of perf record.
- Kernel symbols: Some symbols may show as
[kernel.kallsyms]. These are kernel-side costs (syscalls, page faults). If they dominate, investigate I/O patterns or memory allocation.
- Inlining: Heavily inlined functions may not appear as separate symbols. Use
perf annotate on the caller to see inlined code.
- Async runtimes: Tokio worker threads share names. Use
-t (per-thread) recording to separate them if needed: perf record -t <tid>.
Related Skills
/bench-compare — Criterion before/after measurement (use first to detect regressions)/perf-regression — Full regression workflow with acceptance criteria/performance-analyzer — Static hotspot analysis for this project's patterns
