debugging

name	debugging
description	Structured root-cause investigation for bugs and failures.

Debugging

Constraint

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION

If Phase 1 is incomplete, no fix proposals allowed. Period.

When to Activate

Any technical malfunction: test failures, runtime errors, unexpected behavior, performance regressions, build breaks, integration mismatches.

Activate especially when:

• Time pressure makes guessing tempting
• An "obvious quick fix" presents itself
• Previous attempts have not resolved the issue
• The failure mechanism is not fully understood

Investigation Protocol

Complete each phase sequentially. Do not skip ahead.

Phase 1: Root Cause Investigation

1. Read error output thoroughly

• Full stack traces, not just the top frame
• Note line numbers, file paths, error codes
• Warnings preceding the error often contain the answer

2. Reproduce consistently

• Determine exact trigger steps
• Confirm it repeats reliably
• If non-reproducible: gather more data — do not guess

3. Check recent changes

• git diff, git log --oneline -10
• New dependencies, config modifications, environment drift

4. Instrument multi-component boundaries

When the system spans multiple layers (CI pipeline, API chain, build tooling):

For EACH component boundary:
  - Log inputs entering the component
  - Log outputs leaving the component
  - Verify config/env propagation across the boundary
  - Run once, collect evidence, identify the failing seam

5. Trace data flow backward

See references/root-cause-tracing.md for the full technique.

Short version: start at the crash site, ask "where did this bad value come from?", trace upstream until you reach the originating mutation. Fix there.

Phase 2: Pattern Comparison

1. Locate analogous working code in the same codebase
2. If implementing a known pattern, read the reference implementation completely — no skimming
3. Diff working vs. broken — list every difference, however trivial
4. Map dependencies: config, environment, shared state, ordering assumptions

Phase 3: Hypothesis Testing

1. State one hypothesis explicitly: "X causes the failure because Y"
2. Make the smallest possible change to test it — one variable only
3. Evaluate:
   • Confirmed → proceed to Phase 4
   • Refuted → form a new hypothesis from the new evidence
   • Do not layer fixes on top of each other

Phase 4: Fix and Verify

1. Write a failing test that captures the bug (use arsyn:tdd if available)
2. Apply a single fix targeting the root cause — no bundled refactoring
3. Run the full relevant test suite — not just the new test
4. If the fix fails:
   • Count total attempts
   • Under 3: return to Phase 1 with new data
   • 3 or more: stop fixing. The problem is likely architectural.

On 3+ failed attempts: Indicators of structural mismatch:

• Each fix surfaces new coupling in a different location
• Fixes demand large-scale refactoring to implement
• Each patch spawns new symptoms elsewhere

Stop and reconsider fundamentals: Is the current approach viable, or is inertia the only reason to continue? Escalate for discussion before further changes.

File-System-as-State Pattern

For investigations spanning many files or long debugging sessions, offload findings to disk instead of relying on context memory:

# Initialize a debug log at the start of investigation
echo "# Debug Log — $(date -Iminutes)" > debug-log.md

# Append findings as you go
echo "## Phase 1 — Error Analysis" >> debug-log.md
echo "- Stack trace points to auth.ts:142" >> debug-log.md
echo "- Token is null at middleware boundary" >> debug-log.md

# Append hypotheses and results
echo "## Hypothesis 1: Token not forwarded by proxy" >> debug-log.md
echo "- Result: REFUTED — proxy logs show token present" >> debug-log.md

# Later, grep across your own findings instead of scrolling context
grep "REFUTED\|CONFIRMED" debug-log.md

Benefits:

• Survives context window limits in long sessions
• Grep-searchable — faster than re-reading conversation history
• Shareable artifact for post-incident review
• Prevents re-testing already-refuted hypotheses

Use this when: the investigation touches 5+ files, spans multiple tool calls, or you catch yourself re-reading earlier findings.

Stop Signals

If any of these thoughts arise, return to Phase 1 immediately:

• "Quick fix now, investigate later"
• "Let me try changing X and see"
• "Multiple changes at once, then run tests"
• "Skip the test, I'll verify manually"
• "It's probably X" (without evidence)
• "I don't fully understand but this might work"
• Proposing solutions before tracing data flow
• "One more attempt" after 2+ failures

Common Rationalizations

Rationalization	Counter
"Too simple for a process"	Simple bugs have root causes. The protocol is fast for simple cases.
"Emergency — no time"	Structured investigation is faster than guess-and-check cycling.
"Let me try this first"	The first attempted fix sets the pattern. Start correctly.
"I'll add the test after"	Untested fixes regress. Test-first proves resolution.
"Multiple fixes save time"	Prevents isolating what worked. Introduces new defects.
"I see the problem"	Seeing symptoms is not understanding causation.
"One more try" (after 2+ failures)	3+ failures signal structural issues, not missing patches.

Quick Reference

Phase	Core Activity	Exit Criterion
1. Root Cause	Read errors, reproduce, trace data	Understand what fails and why
2. Pattern	Find working analogues, diff against broken	Differences catalogued
3. Hypothesis	State theory, test minimally	Confirmed or replaced
4. Fix	Failing test, single fix, verify	Bug resolved, suite green

Supporting References

• references/root-cause-tracing.md — backward tracing through call stacks
• references/defense-in-depth.md — multi-layer validation after root cause is found
• references/condition-based-waiting.md — replace arbitrary sleeps with condition polling

Gotchas

• Fix-first impulse — Modifying code before understanding the root cause. The constraint is absolute: investigate first.
• Assuming the obvious — "It's probably X" leads to symptom patches. Follow evidence chains, not intuition.
• Symptom suppression — Adding null guards around crashes instead of determining why the value is null.
• Skipping reproduction — Attempting fixes without a reliable trigger. No reproduction means no verification.
• Confirmation bias — Seeking only evidence that supports the first theory. Actively attempt to disprove each hypothesis.
• Scope drift — "While I'm here, let me also..." Stay on the reported defect. File separate issues for other findings.
• Context amnesia — In long sessions, forgetting earlier findings and re-testing refuted hypotheses. Use the file-system-as-state pattern to persist investigation state.
• Layered patches — Applying fix #2 on top of fix #1 without reverting. Each hypothesis test should start from a clean baseline.