tracing-thoroughly

name

tracing-thoroughly

description

A disciplined workflow for tracing hard / stubborn bugs in the 5thPlanet Saturn emulator without tunnel-visioning — classify the failure, make it deterministic, write the hypothesis with explicit falsification, win by register-completeness + an EARLY oracle diff, instrument the transform's OUTPUT, ALWAYS fan out to multiple parallel agents with competing hypotheses, and run a self-check tripwire when probes stop narrowing. Use it for LLE fidelity gaps (render / audio / timing / boot), intermittent stalls, "random" crashes, and any bug where the first few probes haven't eliminated a cause.

★ ALWAYS FAN OUT — spawn multiple parallel agents by default. This skill mandates multi-agent tracing: whenever you do real investigation (step 4 completeness sweeps, step 6 divergence hunts, and always at step 7's stuck check), spawn several parallel agents with competing hypotheses via the Agent tool (e.g. one audits every register, one runs the oracle diff, one traces the divergence). Do not wait to be asked and do not ride a single thread — parallel competing agents are the anti-tunnel-vision default here. (Using the Agent tool this way needs no opt-in; ultracode is only needed for the heavier deterministic Workflow orchestration, which you may additionally reach for on very large sweeps.)

★ Also surface the reasoning-effort lever (it is NOT automatic). Up front, remind the user to adjust the reasoning effort to the top tier for a hard bug (/effort — the direct command; or /config → reasoning effort) — debugging is reasoning-bound, and a low setting under-powers the whole trace. Say this before deep tracing so the work isn't run under-resourced — don't let a hard bug be tackled at low effort.

Trace hard bugs in this order. The most important step is the stuck check (step 7) — invoke it the moment you've spent a few probes without eliminating a hypothesis, or the moment you reach for a "last resort." This skill is the entry point; the reference material is doc/debugging-playbook.md (workflow, instruments, case studies), the CLAUDE.md "Project conventions" debugging bullets, and ADR-0017 (the behavioral-oracle policy). Every probe must be observer-only and golden-safe (env-gated read or #[serde(skip)] field; if it moves a golden it changed the core and is wrong). Prefer extending sdbg over ad-hoc trace tests.

1. Classify before explaining. Decide which subsystem the symptom belongs to — frontend pacing, audio queueing, render, input, or core emulation — and which it is only visible in. The symptom subsystem is rarely the buggy one: a full CD buffer is a downstream victim; blank graphics don't mean the VDP is wrong; repeated audio doesn't mean the frontend is wrong. For a stall, sample frame progress, CD FAD, buffer occupancy, audio-queue depth, and SH-2 PCs first (SAT_MOVIE_PROBE, SAT_PERFLOG).

2. Make it deterministic FIRST — record → replay → snapshot. The core is deterministic given (RTC seed + pad stream); an interactively-"random" failure is between-session variation, not core non-determinism. Capture a failing session as a jupiter input movie (SAT_INPUT_REC), replay it headless (sdbg replay) for a bit-identical repro, then sdbg save a snapshot just before the failure. Don't trace until you have a fixed foothold. (For a timing-sensitive bisection use the snapshot, not the recording — a recording desyncs across code versions.)

3. Write the hypothesis down — with falsification and a rival. Before probing, state: (a) the current hypothesis, (b) the single observation that would falsify it, (c) at least one competing hypothesis. This is the anti-tunnel-vision anchor — if you can't name what would disprove your hypothesis, you'll confirm it forever.

4. Win by COMPLETENESS, and diff the oracle EARLY. A fidelity gap is found by ruling things out exhaustively, not by one clever narrow probe.

   • Dump the FULL register set of the suspect chip/layer before declaring it faithful. One unread control register sends a whole layer wrong — VDP2 SCRCTL (line scroll) was the BIOS-Memory-Manager miss. Enumerate all of them: for a VDP2 layer that means char-mode, scroll, zoom / coord-increment, line scroll (SCRCTL), vertical cell scroll, window, plane/map, and the bitmap-vs-tile / base regs.
   • Run the Mednafen-oracle diff EARLY (mednaref, ADR-0017) — not as a last resort. The project's whole method is oracle-diffing; deferring it wastes the trace. Diff our state vs the oracle at the divergence, then read the matching mednaref handler (sh7095.inc, the VDP2/SCU sources, …).

5. Instrument correctly. For a coordinate/address transform, print the OUTPUT next to the input (screen (x,y) next to source (sx,sy); logical next to physical) and never gate the probe on the value you are validating — a probe that prints only the input of a transform will happily "confirm" a broken mapping (the line-scroll probe printed sy, filtered on sy==168, and "proved" sy=y when it was really 2*y). Extend sdbg with the capability rather than rolling one-off trace tests; the bus write-/read-watch (SAT_WWATCH / SAT_RWATCH, tagged by AccessKind) is the "silently dropped / duplicated transfer" instrument; SAT_MOVIE_PROBE classifies pacing vs core.

6. Trace TO the divergence, then PIVOT to audit. The instant a trace shows wrong data coming out of correct-looking code, STOP chasing the symptom downstream and audit the hardware feature at that point. The game is never wrong — a commercial title is real SH-2 code that runs on hardware + Mednafen, so any misbehavior is an emulation fidelity gap, not a game-logic puzzle. Reuse a title's signature failure mode — once it reveals one class, check that class FIRST on its next blocker.

7. ★ THE STUCK CHECK — run it, don't skip it. After ~3–4 probes on one hypothesis that haven't eliminated anything, or the moment you reach for a "last resort" (ask the user, consult the oracle, give up), STOP and write:

   • the current hypothesis and exactly what you have ruled out, with evidence;
   • whether the data is narrowing the cause or just accumulating (re-running variants of the same probe = tunnel vision);
   • one fresh competing hypothesis you have not tried;
   • then FAN OUT (mandatory, not a choice) — spawn parallel agents with competing hypotheses (one audits every register, one runs the oracle diff, one traces) rather than riding a single thread. This is the default response to being stuck; if you catch yourself about to run one more solo probe instead, that is the tell to fan out now.

   Tells you are looping: gathering more data without options shrinking; explaining away a result that doesn't fit; reaching for the oracle "as a last resort" (that is the signal you should have done step 4 first).

8. Fix the ROOT, anchored in a hardware invariant — and add a regression. Target the timing-/state-independent root that covers the whole class, not "make this snapshot pass." Cite the invariant (interrupts are not accepted in branch delay slots; SCU IST is write-0-clear; cache-through aliases fold to physical addresses; CCR access width matters; LSCY supplies the source-Y base; …). Add a focused unit / golden regression, then rerun the original title repro — the unit test validates the invariant, the repro validates the user-visible failure is actually gone. Confirm bios_boot + the per-title render goldens are unchanged unless the fix intends to move them.

9. Record the lesson. Add a scrubbed case study to doc/debugging-playbook.md (symptom → trace → root → lesson, with the commit ref), note any new fertile-gap class, and keep the negative evidence (ruled-out hypotheses) with the case so the next session doesn't re-chase them.