debugging

# Debugging ## The Iron Law Never propose a fix without first identifying the root cause. "Quick fix now, investigate later" is forbidden — it creates harder bugs. ## Process **1. Reproduce** — make the bug consistent. If intermittent, run N times under stress or simulate poor conditions (slow network, low memory) until it triggers reliably. **2. Investigate** — trace backward through the call chain from the symptom. Add diagnostic logging at each component boundary. Compare working vs broken state using a differential table (environment, version, data, timing — what changed?). **3. Hypothesize and test** — one change at a time. If a hypothesis is wrong, fully revert before testing the next. Use `git bisect` to find regressions efficiently. **4. Fix and verify** — create a failing test FIRST, then fix. Run the test. Confirm the original reproduction case passes. No completion claims without fresh verification evidence. ## Three-Fix Threshold After 3 failed fix attempts, STOP. The problem is likely architectural, not a surface bug. Step back and question assumptions about how the system works. Read the actual code path end-to-end instead of spot-checking. ## Escalation: Competing Hypotheses When the cause is unclear across multiple components, use Analysis of Competing Hypotheses: - Generate hypotheses across failure modes: logic error, data issue, state problem, integration failure, resource exhaustion, environment - Investigate each with evidence: Direct (strong), Correlational (medium), Testimonial (weak) - Cite evidence with `file:line` references - Rank by confidence. If multiple hypotheses are equally supported, suspect compound causes. ## Intermittent Issues - Track with correlation IDs across distributed components - Race conditions: look for shared mutable state, check-then-act patterns, missing locks - Resource exhaustion: monitor memory growth, connection pool depletion, file descriptor leaks - Timing-dependent: replace arbitrary `sleep()` with condition-based polling — wait for the actual state, not a duration ## Defense-in-Depth Validation After fixing, validate at every layer — not just where the bug appeared: - **Entry**: does invalid input get caught? - **Business logic**: does the fix handle edge cases? - **Environment**: does it work across configurations? - **Instrumentation**: add logging to detect recurrence ## Bug Triage When multiple bugs exist, prioritize by: - **Severity** (data loss > crash > wrong output > cosmetic) separately from **Priority** (blocking release > customer-facing > internal) - Reproducibility: always > sometimes > once. "Sometimes" bugs need instrumentation before fixing. - Quick wins: if a fix is < 5 minutes and unblocks others, do it first ## Common Patterns - **Null/undefined access** — trace where the value was expected to be set, check all code paths - **Off-by-one** — check `<` vs `<=`, array length vs last index, loop boundaries - **Async ordering** — missing `await`, unhandled promise rejection, callback firing before setup completes - **Type coercion** — `==` vs `===`, string-to-number conversion, truthy/falsy edge cases - **Timezone** — always store UTC, convert at display. Check DST transitions. ## Anti-Patterns - Shotgun debugging (random changes without hypothesis) — revert and think instead - Multiple simultaneous changes — isolate each change or you can't learn what worked - Fixing the symptom not the cause — the same bug will resurface differently - Ignoring intermittent failures ("works on my machine") — instrument and reproduce under load instead