Use when profiling on Linux/ARM/Graviton targets, when you need PMU counter data beyond what flamegraphs show, or when /perf-topdown identifies a bottleneck class that needs source-level drill-down. Deep perf profiling with annotated hotspot analysis.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Use when profiling on Linux/ARM/Graviton targets, when you need PMU counter data beyond what flamegraphs show, or when /perf-topdown identifies a bottleneck class that needs source-level drill-down. Deep perf profiling with annotated hotspot analysis.
Linux Perf Profiling
Deep hardware-level performance analysis using Linux perf on this ARM (Neoverse V1 / Graviton3) system. Goes beyond Criterion benchmarks to explain why code is slow using PMU counters, topdown decomposition, cache/TLB analysis, and annotated disassembly.
When to Use
Criterion benchmarks show a regression but the cause is unclear
You need to understand where cycles are spent inside a hot function
Investigating cache misses, branch mispredictions, TLB pressure, or stall cycles
Comparing two builds at the microarchitectural level
Generating flamegraphs for visual hotspot identification
After /bench-compare finds a regression and you need to explain it
If inferno is not installed: cargo install inferno. Alternatively:
perf script > perf.script
# Then use https://www.speedscope.app/ — drag-and-drop perf.script
Mode 3: Cache & TLB Drill-Down
When topdown shows backend/memory stalls, measure the cache hierarchy.
perf stat -e l1d_cache,l1d_cache_refill,l1d_cache_lmiss_rd,l2d_cache,l2d_cache_refill,l2d_cache_lmiss_rd,dtlb_walk,itlb_walk,mem_access \
./target/release/gossip-worker 2>&1
Derived metrics:
Metric
Formula
Healthy
Investigate
L1d miss rate
l1d_cache_refill / l1d_cache × 100
< 5%
> 10%
L1d → L2 miss rate
l1d_cache_lmiss_rd / l1d_cache_refill × 100
< 30%
> 50%
L2 miss rate
l2d_cache_refill / l2d_cache × 100
< 10%
> 20%
L2 → LLC/DRAM rate
l2d_cache_lmiss_rd / l2d_cache_refill × 100
< 20%
> 40%
dTLB miss rate
dtlb_walk / mem_access × 100
< 0.5%
> 2%
iTLB miss rate
itlb_walk / l1i_cache × 100
< 0.1%
> 1%
Common patterns in this codebase:
High L1d misses → check struct layout, false sharing in concurrent paths, random-access patterns in TimingWheel (crates/gossip-stdx/src/timing_wheel.rs) or SetAssociativeCache (crates/scanner-engine/src/lsm/set_associative_cache.rs).
High L2 misses → working set exceeds L2 (256KB/core on Graviton3). Consider data partitioning or reducing struct sizes.
High dTLB walks → large heap allocations scattered across pages. Consider hugepages or arena allocation.
High iTLB walks → code bloat from monomorphization or heavy inlining. Check generic instantiation count.
Mode 4: Branch Analysis
When topdown shows frontend stalls or branch misprediction issues.
perf stat -e br_pred,br_mis_pred,br_retired,br_mis_pred_retired,inst_retired \
./target/release/gossip-worker 2>&1
Derived metrics:
Metric
Formula
Healthy
Investigate
Speculative mispredict %
br_mis_pred / br_pred × 100
< 2%
> 5%
Retired mispredict %
br_mis_pred_retired / br_retired × 100
< 1%
> 3%
Branch density
br_retired / inst_retired × 100
< 20%
> 30%
To find which branches are mispredicting:
perf record -e br_mis_pred_retired -c 1000 -g --call-graph dwarf \
./target/release/gossip-worker
perf report --stdio --percent-limit=1.0 2>&1 | head -60
Mode 5: Annotated Source (Per-Line Costs)
Once you've identified a hot function from Mode 2, drill into it at the source-line level.
git stash pop
RUSTFLAGS="-C target-cpu=native -C debuginfo=2" cargo build --release
perf stat -r 3 -e cpu_cycles,inst_retired,stall_frontend,stall_backend,stall_backend_mem,l1d_cache_refill,l2d_cache_refill,br_mis_pred_retired \
./target/release/gossip-worker 2>&1 | tee /tmp/perf-after.txt
Step 3: Diff analysis
Compare the two files and compute deltas. Focus on:
Did cycle count increase? If so, did IPC drop or did instruction count increase?
If IPC dropped, which stall category increased?
If instructions increased, was it branch density or memory ops?
Mode 7: Lock Contention & Scheduling
For diagnosing contention in async/concurrent code paths.
perf stat -e context-switches,cpu-migrations,page-faults \
-e sdt_libpthread:mutex_entry,sdt_libpthread:mutex_acquired \
./target/release/gossip-worker 2>&1
For scheduling latency:
perf sched record ./target/release/gossip-worker
perf sched latency --sort max 2>&1 | head -30
Event Groups for Copy-Paste
Pre-built event sets tuned for this system (ARM Neoverse V1, armv8_pmuv3).
Multiplexing note: This virtualized Graviton3 environment exposes ~3 simultaneous hardware counters. Do not use {} group pinning syntax — it will fail with <not supported>. Instead, pass events as a comma-separated list and let perf stat multiplex automatically. The kernel time-shares counters and scales results; the (XX.XX%) annotation next to each counter shows what fraction of runtime it was active. With workloads running ≥1 second, scaled values are reliable. For maximum accuracy on critical ratios, use -r 3 (repeat 3x) and keep groups small (3-4 events).
When you need exact ratios without scaling, use ≤3 events:
cpu_cycles,inst_retired,stall_backend
Output Format
Report findings using this structure:
## Perf Profile: [target / scenario]### Environment- CPU: ARM Neoverse V1 (Graviton3), 16 cores
- Build: `RUSTFLAGS="-C target-cpu=native -C debuginfo=2"` release
- Target: [repo or benchmark name]
### Topdown Summary
| Metric | Value | Assessment |
|--------|-------|------------|
| IPC | X.XX | [good/investigate] |
| Frontend bound | X.X% | [ok/high] |
| Backend bound | X.X% | [ok/high] |
| Backend memory | X.X% | [ok/high] |
| Branch mispredict | X.X% | [ok/high] |
### Hotspots (top 5 by self%)
| Rank | Symbol | Self % | Module | Likely Cause |
|------|--------|--------|--------|--------------|
| 1 | func_name | XX.X% | gossip_stdx | [explanation] |
### Cache Hierarchy (if relevant)
| Level | Accesses | Misses | Miss Rate | Assessment |
|-------|----------|--------|-----------|------------|
| L1d | X.XXB | X.XXM | X.X% | [ok/high] |
| L2 | X.XXM | X.XXM | X.X% | [ok/high] |
### Root Cause Analysis
[Narrative explaining what the numbers mean for this specific code. Connect
PMU data to source-level patterns. Reference specific lines/functions.]
### Recommendations1.**[Issue]** at `file:line` — [specific fix with rationale tied to PMU data]
- Expected impact: [which metric should improve and by roughly how much]
- Validate: `perf stat -e <relevant_events> ...`### Validation Plan
[Commands to re-run after applying fixes to confirm improvement]
Caveats
Multiplexing: This virtualized environment supports ~3 simultaneous counters. perf stat multiplexes automatically — the (XX.XX%) annotation shows sampling duty cycle. With ≥1s workloads, scaled values are reliable. Never use {} group pinning.
DWARF unwinding cost: --call-graph dwarf adds ~5-15% overhead to the profiled process. For tight latency measurements, use perf stat instead of perf record.
Kernel symbols: Some symbols may show as [kernel.kallsyms]. These are kernel-side costs (syscalls, page faults). If they dominate, investigate I/O patterns or memory allocation.
Inlining: Heavily inlined functions may not appear as separate symbols. Use perf annotate on the caller to see inlined code.
Async runtimes: Tokio worker threads share names. Use -t (per-thread) recording to separate them if needed: perf record -t <tid>.
Related Skills
/bench-compare — Criterion before/after measurement (use first to detect regressions)
/perf-regression — Full regression workflow with acceptance criteria
/perf-topdown — Cross-arch TMA + branch trace entry point; escalates to this skill for ARM-specific deep dives
/performance-analyzer — Static hotspot analysis for this project's patterns
/pgo-bolt — Branch sampling data from perf can feed BOLT for post-link binary optimization
