| name | trace-analysis |
| description | The investigation harness for "why is this slow?" over a Pulp Perfetto trace (.pftrace). Runs the hypothesis→query→drill-down chain-of-evidence loop autonomously and returns a plain-English root cause + evidence + a concrete fix. TRIGGER on "why is my plugin slow to open", "find the slowest frames", "why is the UI stuttering", "why is my plugin using so much CPU", "which DSP node is expensive", "the load meter looks calm but CPU is pinned", or any `pulp trace explain "<question>"` / `/trace "<question>"` / `pulp_trace_explain` invocation. Ships Pulp-specific hints for dsp, frame, js, gpu, and cross-platform symptoms. |
| requires | [".agents/skills/trace-analysis/references/hints_dsp.md",".agents/skills/trace-analysis/references/hints_frame.md",".agents/skills/trace-analysis/references/hints_js.md",".agents/skills/trace-analysis/references/hints_gpu.md",".agents/skills/trace-analysis/references/hints_crossplatform.md"] |
trace-analysis — the "why is this slow?" investigation harness
Someone who has never opened a profiler types one line — "why is my plugin
slow to open?" — and gets a real, plain-English answer with a suggested fix.
This skill is the protocol that makes that happen: you load it, capture or
accept a .pftrace, and run a disciplined investigation over it, returning a
narrated root cause and a chain of evidence — never raw SQL. You are reading
this because a tracing question fired (L1 explain) or an expert is driving an
iterative investigation (L2).
This skill is provider-agnostic by construction: it is a plain SKILL.md read
identically by Claude Code and Codex from .agents/skills/. The whole
"why is this slow?" experience works under any agent because it is a shipped
skill + CLI/MCP surface, not a vendor API.
Its companion is trace-sql — the SQL discipline and the Pulp trace-stdlib of
named query primitives. This harness decides what to ask; trace-sql is
how to ask it. Load both for an L2 session.
Attribution. The investigation methodology below — the chain-of-evidence
scratchpad, the hypothesis→query→drill-down loop, the wall-time-vs-CPU-time
rule, follow-the-blocker, exhaustive global verification, and p95/p99
discipline — is adapted from Google's android/skills
perfetto-trace-analysis (Apache-2.0). The methodology is reused; the
Android domain content (SurfaceFlinger, binder, RenderThread, ftrace, cpu
governor) is not — the Pulp domain hints in references/ are authored fresh
against Pulp's own seams. See NOTICE.md.
The tiers (where this skill sits)
| Tier | Who | Entry | This skill's role |
|---|
| L0 | novice, no agent | pulp trace slowest-frames, --preset dsp-hotspots/xruns | not needed — canned preset → plain table |
| L1 | novice, one-shot | pulp trace explain "<q>" · pulp_trace_explain · /trace "<q>" | run this protocol autonomously, return narrated root cause + evidence + fix |
| L2 | expert, iterative | pulp trace query "<sql>" + this skill + trace-sql loaded | drive the full loop by hand on hard/multi-bottleneck cases |
L0 needs no agent. L1 is the headline: you run the whole protocol and hand back
prose. L2 is the same protocol, interactive, for cases the presets cannot crack.
Capture (if you don't already have a .pftrace)
If a capture or query fails for an unclear reason, run the readiness check first
— it tells you, in one shot, whether the inspector is reachable, whether the
host was built with -DPULP_TRACING=ON, and whether a trace_processor is
available for offline SQL:
pulp trace doctor
ready_to_capture:false usually means the inspector is unreachable (start it
with PULP_TRACE_SERVER=1) or tracing was compiled out; ready_to_query:false
means no trace_processor (on $PULP_TRACE_PROCESSOR, the pinned Pulp-fetched
build, or $PATH) or no captured trace yet. For zero-install, run
pulp trace fetch once — it downloads the pinned trace_processor_shell
(Perfetto v57.2), SHA-256-verified, into $PULP_HOME. (pulp tool install trace-processor fetches the same pinned artifact via the tool registry.)
pulp trace start --categories render,gpu,text,js,layout
pulp trace stop
Or accept a --trace FILE.pftrace the user hands you. Choose categories from
the question: startup → render,gpu,text,js,layout; DSP cost → dsp,dsp.node
(offline render); UI hitch → render,layout,canvas,text,js,gpu plus a
concurrent pulp motion record so the motion trace_id joins in.
Before trusting the capture, confirm it is not silently empty/truncated
(ring overflow → empty trace): SELECT DISTINCT category FROM slice. No rows,
or missing the category you asked for, means re-capture with a larger
--ring-mb or a shorter window — not "nothing was slow."
When you already have a flushed .pftrace and no live session (the common case
for a trace a user handed you), run SQL against the file directly — no inspector
needed:
pulp trace query "SELECT DISTINCT category FROM slice" --trace /tmp/pulp-<ts>.pftrace
This shells out to trace_processor_shell ($PULP_TRACE_PROCESSOR → pinned
Pulp-fetched build → $PATH; pulp trace doctor reports which, pulp trace fetch installs the pinned one) and returns its native table. --format
json/csv and --preset apply only to the live inspector path; offline is raw
SQL against the file.
To eyeball a trace in the Perfetto timeline instead of querying it, hand it to
the UI (browsers block file://, so this serves it over loopback and opens the
UI at it): pulp trace open /tmp/pulp-<ts>.pftrace (--no-browser prints the
URL to paste; --json for agents).
The investigation protocol
1. Keep a chain-of-evidence scratchpad
Write down, as you go: the question → your current hypothesis → the query you
ran → what it showed → the next hypothesis. Every claim in your final answer
must trace back to a specific query result. "Looks slow" is not a finding; "a
text span of 540 ms precedes the first frame, from query X" is.
2. Hypothesis → query → drill-down loop
Form one hypothesis at a time, query it via the trace-sql stdlib (start with
the named views), read the result, refine or discard. Drill from coarse to
fine: category totals (pulp_layout_vs_paint) → the fat span → the thread it
ran on → its children → its args. Do not write one giant query; walk down.
3. Wall time vs CPU time — the rule that prevents wrong answers
slice.dur is wall-clock duration. A long slice may be blocked (waiting
on a lock, I/O, the GPU, another thread), not computing. Before blaming a
long span, check whether it was actually running: join to thread_state /
inspect the thread's scheduling for that window. A 600 ms span that was
Runnable/blocked 90% of the time is a waiting problem (fix the blocker),
not a compute problem (optimize the code). Getting this backwards sends the
fix in the wrong direction.
4. Follow the blocker across threads
When step 3 says a span was blocked, follow the blocker: which thread /
resource held it? The audio block waited on a mutex the UI thread took; the
present waited on a GPU pass; the layout waited on a JS callback. Trace the
wait to its holder on the other thread and make that the next hypothesis.
The bottleneck is frequently not on the thread that looks slow.
5. Exhaustive global verification — do not stop at the first bottleneck
Finding one fat span is not the end. Verify globally: is this the dominant
cost, or one of several? Sum the category totals and check the found span's
share. Look for a second offender of comparable size. A fix that removes a
300 ms span from a 2.4 s open still leaves 2.1 s — say so. Report the full
budget, ranked, not just the first thing you noticed.
6. p95/p99 discipline — the mean lies
For anything recurring (per-block DSP, per-frame render), the mean hides the
spike. A node at 40% mean can eat 60% of the worst block. Look at max_ms
next to mean_ms (the stdlib views expose both), and when the gap is
ambiguous compute the tail (p95/p99) over the raw slices — see trace-sql.
"The load meter said 40%; the trace said WHY" is exactly this: the average was
calm, one node's tail was not.
The mean also lies in the other direction: a per-node average can be
dominated by a one-time cold-start spike on the first block (the first
process() call warms caches / touches fresh pages), making a genuinely cheap
node look like the worst offender. Always separate the first-block outlier
from steady-state cost — subtract the per-node MAX(dur) (the steady-state
query in trace-sql's "Common query shapes"), or just read the flamegraph and
ignore block 0. examples/trace-plugin-chain is a runnable demonstration:
gain's whole-run average is ~158× its steady-state cost, and the real per-block
hot node is the biquad filter, not the gain the average fingers.
7. Consult the domain hints
Match the symptom to a hints file and read it before drawing conclusions — each
grounds the analysis in Pulp's real seams and names the specific traps:
| Symptom | Hints file |
|---|
| xruns, per-node DSP cost, deadline miss, "meter calm but one node dominates", jitter/denormals | references/hints_dsp.md |
dropped frames vs vsync budget, layout-vs-paint, TextShaper::prepare re-runs, dirty-rect churn, GPU-submit stalls | references/hints_frame.md |
| QuickJS bridge dispatch cost, a JS callback invalidating layout | references/hints_js.md |
| Dawn submit/present stalls, Graphite record cost, per-pass GPU time | references/hints_gpu.md |
| standalone vs plugin-in-DAW vs iOS/iPadOS AUv3 vs Android/Oboe vs Simulator; sample-position args, thread naming, atrace interleave | references/hints_crossplatform.md |
8. Answer in plain English (L1) — never surface SQL
Return: root cause (one or two sentences), chain of evidence (numbered,
each tied to what a span/query showed), and a concrete fix. Give magnitudes
("~620 ms Dawn/Graphite init, one-time" ), say whether the cost repeats, and
estimate the win. See the worked narrative below.
Escalating to the user — only at a genuine priority fork
Default: investigate autonomously. Follow the blocker, gather evidence, and
verify exhaustively. Resolve every factual gap from the trace itself — a
category total, a thread state, a span's args are things you query, not things
you ask. Never stop to ask the user something a query can answer.
Reach for AskUserQuestion only when the direction turns on the user's
priorities, not on data you can gather. Genuine forks: startup splits
near-evenly across font-shaping and shader-compile and only the user knows
which path matters for their use case; or the fix has a fast-approximate branch
and a slower-thorough branch and the tradeoff is theirs to pick. These are
preference forks, not missing facts.
Form it well: put the recommended option first and label it recommended,
with a terse pro/con per option. Never use it as a progress checkpoint ("should
I keep going?") — that is exactly the pause to avoid — and never to re-confirm a
decision already made or to ask what a query would answer.
Worked example — "why is my plugin slow to open?" (the flagship)
This is the safest, most relatable case: bounded, one-shot, main-thread,
deterministic — no real-time hazard, works regardless of the DSP story.
pulp trace start --categories render,gpu,text,js,layout
pulp trace stop
pulp trace explain "why is my plugin slow to open?"
A good answer reads like this:
Root cause: first editor-open spends ~2.4 s, and ~1.9 s of it is one-time
GPU/text setup on the UI thread before the first frame — none of it cached
between opens.
Breakdown: Dawn device + Graphite context init ~620 ms, Skia font-atlas
build for the UI typeface ~540 ms, QuickJS eval of the bundled UI script
~410 ms, first Yoga layout + TextShaper::prepare of every label ~330 ms.
Chain of evidence: (1) a gpu/render span pair brackets the ~620 ms
device init on the main thread. (2) text spans show the font-atlas build
(~540 ms) preceding the per-label TextShaper::prepare. (3) the js span for
script eval is ~410 ms, single-shot. (4) a second open repeats the same spans
identically — nothing is reused.
Fix: warm the Dawn/Graphite context and font atlas once per process (not
per editor open) and cache the compiled JS module. Re-opens should drop from
~2.4 s to well under 500 ms.
The other canonical case is the offline DSP reveal — "CPU pinned but the meter
looks calm — which node?" — run against a deterministic offline_process()
render (examples/trace-demo) so the answer reproduces exactly. See
hints_dsp.md and docs/guides/tracing.md use case 3.
Agent contract
- State the question as a measurable target before querying.
- Verify the capture is non-empty and has the categories you need before
analyzing. An empty trace is a capture bug, not a "fast" program.
- Distinguish wall time from CPU time before blaming a long span (step 3).
- Follow blockers across threads; the slow thread is often not the guilty one.
- Verify globally — report the ranked budget, not the first bottleneck.
- Every claim cites a query result. No evidence, no finding.
- Investigate autonomously;
AskUserQuestion only at a genuine priority fork,
never for something a query answers or a progress checkpoint.
- L1 returns prose (root cause + evidence + fix); never dump SQL at a novice.
Files this skill covers
.agents/skills/trace-analysis/references/hints_dsp.md.agents/skills/trace-analysis/references/hints_frame.md.agents/skills/trace-analysis/references/hints_js.md.agents/skills/trace-analysis/references/hints_gpu.md.agents/skills/trace-analysis/references/hints_crossplatform.md.agents/skills/trace-sql/SKILL.md — the SQL substrate + trace-stdlibcore/runtime/include/pulp/runtime/trace.hpp — macro surface + category taxonomydocs/guides/tracing.md — the guide, tiers (L0/L1/L2), worked use cases, gotchas
