trace

name	trace
description	Use when encountering bugs, test failures, runtime errors, broken builds, or "this doesn't work" reports. Systematic root-cause analysis before any patch — never blind-patches symptoms. Standalone, ends with a final-integration review of the fix. Trigger with /hyperflow:trace, "debug this", "find the root cause", "why is this failing", "this test is broken".
allowed-tools	Read, Bash(git:*), Bash(npm:*), Bash(pnpm:*), Glob, Grep, Agent, AskUserQuestion
argument-hint	<bug description or failing test name>
version	3.1.3
license	MIT
compatibility	Portable — Claude Code, Codex, OpenCode, Antigravity, Cursor, Grok (semantic ops via runtime-contract)
tags	["debugging","root-cause","systematic","multi-agent"]

Trace

Root cause, not symptom. Never patch over a bug without understanding why it happened.

All agents inherit the current session model — there is no model-tier routing and no model configuration. Reviewers and Debuggers bold-labeled; Workers plain.

Semantic host ops: spawn, wait, follow_up, structured_question, shell, edit, web_research, usage_metrics (runtime-contract.md). Specialist briefs: ../../scripts/render-specialist-brief.py when the host lacks native agent discovery.

Role separation (hard): workers never review their own output. Reviewers never coordinate. Missing spawn → labelled inline worker phase, then a separate labelled inline reviewer phase — never merge.

Per-Step Agent Map (DOCTRINE rule 12)

Every substantive step dispatches at least one agent role via spawn (or a labelled inline phase). Atomic steps (per DOCTRINE 12.2.8) are a single Worker → Reviewer pair with no independent angles to fan out.

Step	Status	Workers	Reviewers	Notes
1 — Reproduce	Atomic (12.2.8)	Searcher	Reviewer	Runs if repro missing; single Worker→Reviewer pair
2 — Gather evidence	Atomic (12.2.8)	searcher specialist × 3	Reviewer	Parallel Searchers when spawn concurrency allows; else sequential labelled phases → single Reviewer
3 — Hypothesize	Atomic (12.2.8)	debugger specialist	Reviewer	Single Debugger (5 Whys + ranked hypotheses); may fan out Step 4 across hypotheses (rule 18)
4 — Verify	2 sub-phases	Implementer × N	Debugger · Reviewer	4a: Implementers per hypothesis → Reviewer; 4b: Debugger re-evaluation → Reviewer
5 — Fix at root	Atomic (12.2.8)	Implementer × N	Reviewer	N Implementers (one per file) → Reviewer
6 — Regression test	Atomic (12.2.8)	Writer	Reviewer	Single Writer → single Reviewer
7 — Memory + final	Atomic (12.2.8)	Writer	Reviewer	Single Writer → integration Reviewer

Step 1 — Reproduce

Atomic — single Searcher → Reviewer pair (DOCTRINE 12.2.8). No parallel angles: artifact retrieval is a single-scope search when the symptom is unknown.

If the user supplied a stack trace, test name, or log snippet — skip the Worker dispatch entirely (Step 1 is then trivially fulfilled by existing input; proceed to Step 2).

Otherwise spawn (or labelled inline) Searcher — locating bug reproduction in recent changes/tests.

Collect: failing test name or command, error message, stack trace, log lines, recent commits touching the affected surface.

Then spawn / labelled inline **Reviewer** — confirming reproduction is valid with the collected artifacts — separate from the Searcher.

Reviewer confirms:

• The failure is consistent and deterministic (or flags intermittent).
• The error matches the stated symptom.
• The repro is not a test-environment artifact (missing seed data, wrong env vars, clock skew).

If environmental (CI-only, intermittent, time-dependent) — flag explicitly before proceeding to Step 2.

Step 2 — Gather Evidence

Atomic — one Worker-group (3 Searchers) → single Reviewer pair (DOCTRINE 12.2.8). The three Searchers are parallel angles inside one sub-phase, not independent sub-phases. They are dispatched as the searcher specialist — path-anchored evidence, read-only, no fan-out.

When spawn concurrency is available, dispatch simultaneously:

• Searcher — reading error stack traces and logs
• Searcher — mapping the code paths involved
• Searcher — finding related tests (passing and failing)

When concurrent children are unavailable, run the three as sequenced labelled inline searcher phases with honest serial wording — still cover all three angles before review.

Each Searcher writes its findings as a structured list: file paths, line numbers, key values, timestamps.

Then **Reviewer** — verifying evidence coverage over all three Searcher outputs (always a separate phase).

Reviewer confirms the three Searchers actually triangulate the failure surface. If gaps remain (e.g., no log found, code path incomplete), the Reviewer names specific missing angles — re-run the Searcher(s) for those gaps only, then re-run the Reviewer. Repeat until coverage is confirmed.

Failure recovery: Searcher tool errors and NEEDS_REVISION verdicts follow DOCTRINE rule 14 (skills/hyperflow/failure-recovery.md). For trace specifically, a Searcher that aborts mid-evidence-gathering leaves the debugger with incomplete coverage — flag the gap explicitly in the Step 3 Reviewer output and carry it forward as a known uncertainty in the root-cause synthesis. Do not silently proceed as if evidence is complete.

Step 3 — Hypothesize

Atomic — single Debugger → Reviewer pair (DOCTRINE 12.2.8). 5 Whys and hypothesis ranking are a single sequential reasoning task; one Debugger call produces both in one pass.

spawn (or labelled inline) **Debugger** — 5 Whys + hypothesis ranking: <bug-summary> — the debugger specialist agent, carrying its charter (root-before-symptom, written 5-Whys chain, web-research-first on a gated flow for known-issue/changelog lookups via web_research when present). When ≥ 2 hypotheses are genuinely independent, the debugger may fan out Step 4 verification across them (depth 1, ≤ 3 sub-workers — DOCTRINE rule 18); a single-hypothesis bug never fans out.

Single call produces:

Part A — 5 Whys causal chain (depth-first):

• Why does this fail? → because X → why X? → because Y → continue to root.
• Goal: reach a structural cause (data contract violation, state mutation, missing guard, timing assumption), not a surface symptom.
• Output: one causal chain ending at the deepest reachable root.

Part B — Hypothesis fan-out (using Part A's causal chain):

• Emit 1–3 ranked hypotheses. Each must include:
  • What — suspected root cause
  • Evidence — what from Step 2 supports it
  • Counter-evidence — what would falsify it
  • Test — minimal change to verify (used by Step 4)

Then **Reviewer** — validating causal chain and hypothesis set over the Debugger's output — always separate.

Reviewer confirms the causal chain reaches a structural root (not a symptom) and that each hypothesis is independently testable.

Step 4 — Verify

Two sub-phases (genuine sequential dependency: 4b depends on 4a results; 4b Debugger does substantive re-evaluation work, not a pure review pass).

Step 4a — Minimal change verification

Workers: Implementer × N, where N = number of hypotheses to test. One Implementer per hypothesis.

When spawn concurrency allows, dispatch hypothesis tests in parallel; otherwise run as sequenced labelled implementer phases. All ranked hypotheses must still be tested before synthesis (or the single-hypothesis path justified).

• Implementer — verifying hypothesis 1: <hypothesis-1-test> — make the minimal change to confirm/falsify
• Implementer — verifying hypothesis 2: <hypothesis-2-test> (if applicable)

Each Implementer makes only the change described in the hypothesis's Test field from Step 3. No additional cleanup, no reformatting. Run the failing test/command after each change and capture the result.

If only one hypothesis exists — single Implementer is justified (no parallel angle; single-Worker sub-phase per DOCTRINE 12.2.3 single-Worker exception).

Reviewer: **Reviewer** — checking verification results are deterministic over the Implementer outputs. Confirms each test run is deterministic and the result cleanly maps to a confirm/falsify verdict.

Failure recovery (4a): Implementer tool errors and NEEDS_REVISION verdicts follow DOCTRINE rule 14 (skills/hyperflow/failure-recovery.md). An Implementer that aborts or cannot confirm/falsify its hypothesis marks that hypothesis INCONCLUSIVE — the chain does not abort. Other hypotheses proceed normally; the 4b Debugger receives the full set including any INCONCLUSIVE entries.

Step 4b — Re-evaluation + loop gate

Worker: **Debugger** — re-evaluating hypotheses against verification results. Substantive reasoning — the Debugger compares hypothesis predictions against actual test outcomes and decides the next branch. Not a pass/fail check: the Debugger may emit CONFIRMED, FALSIFIED ALL, or PARTIALLY CONFIRMED with new directions.

Reviewer: **Reviewer** — confirming re-evaluation verdict is sound over the Debugger's verdict — separate phase.

Failure recovery (4b): Debugger tool errors and NEEDS_REVISION verdicts follow DOCTRINE rule 14 (skills/hyperflow/failure-recovery.md). A failed 4b Debugger dispatch does not abort the chain; retry once, then escalate. If all attempts fail, mark the entire verify step as INCONCLUSIVE and surface to the user — do not advance to Step 5 without a root-cause verdict.

Debugger verdicts:

• CONFIRMED <hypothesis-N> → proceed to Step 5 with that hypothesis as the confirmed root cause.
• FALSIFIED ALL → loop back to Step 2 with a broader evidence scope.
• PARTIALLY CONFIRMED → redispatch Step 4a for the leading candidate with a tighter test.

Revert all minimal changes from 4a before entering Step 5 (the real fix goes in Step 5, not here).

Step 5 — Fix at Root

Atomic — one Worker-group (N Implementers) → single Reviewer pair (DOCTRINE 12.2.8). N Implementers are parallel angles inside one Worker-group when concurrency allows; the Reviewer gates the group output.

Dispatch one Implementer per affected file (or one Implementer total if single-file):

• Implementer — fixing root cause in <file-1>: <change-description>
• Implementer — fixing root cause in <file-2>: <change-description> (if applicable)

Each Implementer receives: the bug, the verified root cause from Step 4b, the minimal change. No extra refactoring, no opportunistic cleanup — root cause only.

Constraints (non-negotiable):

• No error swallowing
• No defensive try/catch around the symptom
• No flags or feature gates to hide the bug

Then **Reviewer** — checking fix is at root over all Implementer outputs.

Reviewer verifies:

• The fix addresses the confirmed root cause from Step 4b, not the symptom.
• No constraint violations (error-swallow, try/catch workaround, feature gate).
• Files changed are internally consistent (no partial fix across N files).

On rejection — loop Step 5 with the Reviewer's specific objection attached. Do NOT commit until Step 5 passes.

Step 6 — Regression Test

Atomic — single Writer → single Reviewer pair (DOCTRINE 12.2.8). Test authorship has no parallel angle: two Writers would produce duplicate or conflicting tests.

spawn / inline Writer — adding regression test for <bug>.

The test must:

• Exercise the exact code path that was broken.
• Assert the behavior that was missing (not just assert the fix is present).
• Be named to describe the bug scenario, not the implementation.

Then **Reviewer** — confirming regression test fails-without and passes-with the fix.

Reviewer process:

1. Mentally (or via shell) revert the Step 5 fix.
2. Confirm the new test fails in the broken state.
3. Re-apply the fix.
4. Confirm the test passes in the fixed state.

If the test passes both with and without the fix — reject; Writer rewrites. The test must demonstrably distinguish the buggy and fixed states.

If existing suite had coverage gaps that allowed this bug → note for Step 7.

Step 7 — Memory + Final Review

Atomic — single Writer → single integration Reviewer pair (DOCTRINE 12.2.8). Single-artifact write with no parallel angle; Reviewer covers the full cumulative diff.

spawn / inline Writer — appending pitfall to .hyperflow/memory/pitfalls.md per memory-system.md.

Entry must include:

• The bug pattern (generalized, not project-specific)
• Why existing tests missed it
• Prevention strategy
• Tags: pitfall plus domain tags (e.g., auth, async, state)

Then **Reviewer** — final validation of fix + test + memory entry.

This is the integration review for the entire trace flow. Reviewer assesses the cumulative diff:

• Fix lands at root (not symptom).
• Regression test distinguishes broken vs fixed.
• Memory entry generalizes the pattern correctly.
• No constraint violations introduced anywhere in the chain.

This is the sole integration Reviewer for the trace chain. Pass required before hand-off to deploy.

Anti-Patterns (refuse these)

Symptom patch	Why it's wrong
"Just catch the exception"	Find why it threw
"Add a null check"	Find why it was null
"Increase the timeout"	Find why it's slow
"Retry on failure"	Understand the failure mode first

Output Format

── Debug Result ─────────────────────
Bug: <one-line>
Reproducible: yes / no / intermittent
Root cause: <one-line>
Fix: <one-line summary>
Files changed: <list>
Regression test: <path>
─────────────────────────────────────

End with usage summary when host usage_metrics or the usage ledger provides data (output-style.md). When metrics are unavailable: print unavailable or mark estimated=true only for documented estimators — never fabricate tokens, durations, or agent counts. Inline/sequenced work must not claim parallel subagents.

Hand-off

Debug is off the auto-chain — it's standalone. After Step 7 reviewer passes, stop and suggest /hyperflow:deploy to run pre-push gates and commit the fix + regression test together. Do not auto-invoke ship — push requires explicit user opt-in. Any later plan/dispatch continuation uses skill_continuation per chain-router.md.

Doctrine

Full rules in DOCTRINE.md. See also worker-prompt.md and reviewer-prompt.md. Runtime ops: runtime-contract.md.

Failure recovery (rule 14). Worker errors and NEEDS_REVISION verdicts follow the canonical policy in skills/hyperflow/failure-recovery.md. For trace, a failed hypothesis test (Step 4) marks the hypothesis INCONCLUSIVE rather than aborting the chain — other hypotheses can still resolve the bug. A Searcher abort in Step 2 leaves incomplete evidence; flag the gap in the root-cause synthesis rather than proceeding as if coverage is full.

Overview

/hyperflow:trace is the systematic-debugging skill. It refuses to symptom-patch — every fix starts with reproduction, evidence gathering, hypothesis ranking via a Debugger, and verification before any code changes. Three Searchers triangulate the failure surface (parallel when the host allows; sequenced labelled phases otherwise); a Debugger applies 5-Whys + hypothesis ranking in one pass; independent Reviewers confirm the fix lands at the root and a regression test fails-without / passes-with. Off the auto-chain — standalone.

Prerequisites

• A reproducible bug (or enough symptom info to reproduce). If unclear, Step 1 dispatches a Searcher to locate the failure.
• Git repository — for diffing recent changes and committing the fix + regression test together.
• Test runner detected in .hyperflow/testing.md (vitest/jest/playwright/pytest/etc.) — required for Step 6 regression test.
• .hyperflow/memory/pitfalls.md writable — Step 7 appends the learned pattern.

Portability

Capability	Behavior
spawn present	Independent searchers / implementers / debugger / reviewers as children
spawn absent	Labelled inline worker phases; separate labelled inline reviewer after each worker group
Concurrent children	Parallel hypothesis tests when inventory allows
No concurrency	Sequential labelled hypothesis tests — still test all ranked hypotheses before synthesis
web_research absent	Debugger records offline skip for known-issue lookup; never invent CVEs/changelogs
usage_metrics absent	Print unavailable; never fabricate

Instructions

The 7 numbered steps live in Step 1 — Reproduce through Step 7 — Memory + Final Review above. Steps 1, 2, 3, 5, 6, 7 are atomic (DOCTRINE 12.2.8). Step 4 has 2 sub-phases (genuine sequential dependency). Summary:

1. Reproduce — Atomic. Searcher locates repro artifacts (if needed); Reviewer validates reproducibility. Flag intermittent before proceeding.
2. Gather evidence — Atomic. 3 Searchers (logs, code paths, related tests); Reviewer verifies coverage; re-runs specific Searchers if gaps remain.
3. Hypothesize — Atomic. Debugger runs 5-Whys causal chain AND fans out 1–3 ranked hypotheses in one call; Reviewer validates the causal chain and hypothesis set.
4. Verify — 2 sub-phases. 4a: Implementers make minimal change per hypothesis → Reviewer confirms determinism. 4b: Debugger re-evaluates against results and emits verdict → Reviewer confirms verdict is sound. Loops 4a or proceeds.
5. Fix at root — Atomic. Implementers fix per affected file; Reviewer confirms fix is not a symptom-patch; loops if rejected.
6. Regression test — Atomic. Writer adds a test that must fail on broken code; Reviewer confirms fail-without / pass-with; rejects if test is trivially passing.
7. Memory + final review — Atomic. Writer appends pitfall pattern to .hyperflow/memory/pitfalls.md; integration Reviewer covers the full cumulative diff.

Output

See Output Format above for the structured block (Bug, Reproducible, Root cause, Fix, Files changed, Regression test). Ends with an honest usage summary (agent count + tokens when observed; unavailable otherwise).

Error Handling

Failure	Behavior
Cannot reproduce	Step 1 prints Cannot reproduce — needs more info; ask user via structured_question (or Hyperflow Question chat block) for additional repro context. Do NOT proceed to Step 2 with unreliable repro.
Intermittent / flaky	Flag explicitly in Step 1 output; ask whether user wants to proceed treating as flake vs investigate root cause.
All hypotheses falsified	Loop back to Step 2 with broader evidence collection scope. After 2 full cycles, surface to user: Cannot localize root cause — need additional traces.
Reviewer says fix is a symptom-patch	Reject and loop back to Step 5 with the Reviewer's feedback. Do NOT commit a symptom-patch.
Regression test passes both with and without fix	Reject; Writer rewrites the test. The test must demonstrably distinguish the buggy and fixed states.
Test runner missing	Skip Step 6 with explicit warning: No test runner detected — fix committed without regression test. Suggest user add one.

Examples

Standard trace — failing test

/hyperflow:trace one of my auth tests is failing — find the root cause and fix it

Searcher — locating bug reproduction in recent changes/tests
**Reviewer** — confirming reproduction is valid
Searcher — reading error stack traces and logs
Searcher — mapping the code paths involved
Searcher — finding related tests (passing and failing)
**Reviewer** — verifying evidence coverage
**Debugger** — 5 Whys + hypothesis ranking: auth.test.ts:42 "refresh token rejected"

Hypothesis 1 (likely): refresh token TTL changed in PR #189 but test fixture wasn't updated
Hypothesis 2 (possible): clock skew between test env and JWT issuer

Implementer — verifying hypothesis 1: refresh token TTL
Implementer — verifying hypothesis 2: clock skew check
**Reviewer** — checking verification results are deterministic
**Debugger** — re-evaluating hypotheses against verification results
**Reviewer** — confirming re-evaluation verdict is sound
[hypothesis 1 confirmed]

Implementer — fixing root cause: align test fixture TTL with new TOKEN_REFRESH_TTL constant
**Reviewer** — checking fix is at root
Writer — adding regression test for TTL drift
**Reviewer** — confirming regression test fails-without and passes-with the fix
Writer — appending pitfall to .hyperflow/memory/pitfalls.md
**Reviewer** — final validation of fix + test + memory entry

── Debug Result ─────────────────────
Bug: auth.test.ts:42 "refresh token rejected"
Reproducible: yes
Root cause: test fixture TTL hardcoded to old value; not synced with TOKEN_REFRESH_TTL constant
Fix: import TOKEN_REFRESH_TTL into test fixture; remove magic number
Files changed: src/auth/test-fixtures.ts, test/auth/refresh.test.ts
Regression test: test/auth/refresh.test.ts::"TTL constant drift catches stale fixtures"
─────────────────────────────────────
Agents: 12 total · tokens unavailable

Refuses symptom-patch request

/hyperflow:trace just catch the exception in src/payments/processor.ts

Refusing — trace never patches symptoms. The exception is a signal. Let me find why it throws.

Searcher — reading error stack traces and logs
... (proceeds with full root-cause flow)

Intermittent bug

/hyperflow:trace tests pass locally but fail in CI ~30% of the time

Flagged — intermittent. Possible causes: ordering dependency, race condition,
environmental difference, flaky external. Proceeding with extra evidence gathering.

Searcher — reading CI logs vs local logs
Searcher — looking for shared state between test files
...

Inline mode (no spawn)

/hyperflow:trace flaky null in cache layer

inline searcher — locating bug reproduction
inline reviewer — confirming reproduction is valid
inline searcher — reading error stack traces and logs
inline searcher — mapping the code paths involved
inline searcher — finding related tests
inline reviewer — verifying evidence coverage
inline debugger — 5 Whys + hypothesis ranking
inline reviewer — validating causal chain and hypothesis set
inline implementer — verifying hypothesis 1
inline implementer — verifying hypothesis 2
inline reviewer — checking verification results
inline debugger — re-evaluating against results
inline reviewer — confirming re-evaluation verdict
... (same 5-Whys + independent review semantics; sequential wording only)

Resources

• DOCTRINE.md — orchestration rules (especially #12 per-step agents).
• runtime-contract.md — spawn / inline fallback, metrics honesty.
• worker-prompt.md — worker prompt template.
• reviewer-prompt.md — reviewer prompt template.
• memory-system.md — pitfall entry format.
• output-style.md — agent label format + usage summary spec.