cpufire-dev

name

cpufire-dev

description

Build a Linux continuous CPU profiling / "black box" tool that keeps `perf record` running 7x24, shards samples by time, and replays any past time point as a flame graph. Use this skill when building Linux system daemon tools, perf-based profilers, or anything combining a long-running daemon + CLI + on-disk time-sharded storage. Trigger on keywords like "持续 profiling", "CPU 黑匣子", "perf 回放", "火焰图", "守护进程工具".

cpufire-dev — 持续 CPU Profiling 工具开发 Skill

This skill captures the methodology, design decisions and pitfalls learned while building cpufire: a 7x24 continuous CPU profiler that wraps perf record as a black box and replays past incidents as flame graphs.

When to use

• Build a tool that samples CPU continuously and can replay a historical window.
• Build any Linux daemon + CLI combo with time-sharded on-disk storage.
• Wrap a Linux subsystem tool (perf, bpftool, etc.) as a managed subprocess.

Core design pattern

daemon  ──►  Sampler(perf record)  ──►  ShardManager (shards/*.perf + index.json)
   │                                            ▲
   └────────────── Rotator (prune > retention) ┘
query  ──► ShardManager.Query(time) ──► perf script ──► merged text
flame  ──► stackcollapse-perf.pl + flamegraph.pl ──► SVG

1. Sampler wraps the external tool as a subprocess. Always:
   • expose Start()/Stop(); Stop() sends SIGINT (perf flushes on SIGINT).
   • provide CheckAvailability() that checks the binary AND the kernel knob (/proc/sys/kernel/perf_event_paranoid must be ≤ 2, ideally 1).
   • keep a //go:build !linux stub that returns ErrNotSupported so the project still compiles on macOS/Windows but fails clearly at runtime.
2. ShardManager stores fixed-duration shards named cpufire-<startUnix>-<endUnix>.perf under <storage>/shards/.
   • Persist an index.json (list of {path,start_unix,end_unix,size_bytes}).
   • Atomic write: marshal → write index.json.tmp → rename. Never truncate-then-write (a crash would leave a half index).
   • Query(from,to) returns shards overlapping [from,to].
   • QueryPoint(t) returns the containing shard plus the previous and next shard for context.
3. Rotator is a background goroutine that deletes shards with end_unix < now - retention and prunes their index entries. Inject a clock (func() time.Time) so tests are deterministic.
4. Daemon main loop: StartNewShard → Sampler.Start → wait shard-duration (or ctx.Done) → Sampler.Stop → CloseCurrentShard. Use signal.NotifyContext for graceful SIGTERM/SIGINT shutdown.
5. Query merges shards by running perf script -i <shard> over each and concatenating stdout. Produce perf-script text (consumable by FlameGraph) rather than a strict pprof binary — far simpler and fully interoperable.

Config convention

Support both YAML file and CLI flags; CLI wins. Pattern:

func Load(configPath string, overrides *Config) (*Config, error) {
    cfg := Default()
    if configPath != "" { yaml.UnmarshalFile(configPath, cfg) }
    applyOverrides(cfg, overrides)   // only non-zero fields
    return cfg, cfg.Validate()
}

Critical pitfalls

• perf_event_paranoid: unprivileged sampling needs ≤ 1. Default is often 2 or higher → perf record fails. Tell users to sudo sysctl -w kernel.perf_event_paranoid=1.
• Kernel symbols need root: non-root users get [unknown] kernel frames because /proc/kallsyms hides addresses. Run as root (or kernel.kptr_restrict=0); emit a startup advisory when not root. User-space symbols need the profiled binaries present & unstripped at perf script time (Go 1.21+ amd64 has frame pointers, so -g/fp unwinds fine).
• Sample counts vs period: perf record -F is frequency sampling; each sample carries a period. If perf script output keeps the period, stackcollapse-perf.pl sums periods and the flame graph shows billions of "samples" (still CPU-proportional widths, but misleading). Fix: run perf script -F comm,pid,tid,cpu,time,ip,sym,dso (omit the period) so counts are real sample numbers. Fall back to default output if -F is rejected.
• Shard end time granularity: if EndUnix is stored as integer seconds, a sub-second shard-duration collapses to a zero-width shard (EndUnix == StartUnix), breaking point queries. Enforce shard-duration ≥ 1s (production default 5m is safe).
• SIGINT not SIGKILL: perf record only writes the final perf.data on a clean SIGINT. SIGKILL loses the shard.
• call_graph choice: default fp (frame pointer, cheapest). For workloads compiled without frame pointers use dwarf (--call-graph dwarf); lbr for last-branch. Expose it as a config flag rather than hard-coding.
• FlameGraph scripts: find them via explicit dir then PATH (exec.LookPath("stackcollapse-perf.pl")). Missing → error with install hint.
• Cross-package test doubles: define a minimal interface in the consumer package (e.g. daemon.Sampler) so tests can inject fakes without touching the real exec path.

Testing recipe (no real perf needed)

• Unit: pure-Go packages (storage index, rotator with fake clock, config).
• Sampler: write a fake perf shell script onto PATH (--version prints a version; record does trap 'exit 0' INT; while true; do sleep 0.05; done) and assert args + that Stop sends SIGINT and exits 0.
• Integration: a fake perf (record loops, script prints stacks) + fake stackcollapse-perf.pl/flamegraph.pl (shell scripts emitting <svg>) on PATH, then run the full daemon → query → flame flow.
• E2E: scripts/e2e.sh on a real Linux host; skip the Go wrapper when perf is absent.

Reference projects

• parca-dev/parca — open-standard pprof storage, sharded + indexed, daemon + CLI. cpufire borrows the sharding/index idea but uses perf record (not eBPF) and focuses on time-point replay.
• brendangregg/FlameGraph — stackcollapse-perf.pl + flamegraph.pl.