debugging-methodology

# Debugging Methodology Systematic approach to finding and fixing bugs. ## When to Use This Skill | Use this skill when... | Use something else instead when... | |------------------------|------------------------------------| | Diagnosing a live bug, memory leak, race, or perf regression | Bug is hidden by a swallowed error → `code-hidden-failures --track errors` | | Reasoning about reproduction, isolation, and root cause | Bug is hidden by silent success-on-empty → `code-hidden-failures --track degradation` | | Choosing strace/eBPF/perf for system-level investigation | Reviewing surrounding code quality once root cause is known → `code-review` | | Documenting hypotheses and binary-searching the failure | Refactoring the buggy module after the fix → `code-refactor` | ## Core Principles 1. **Occam's Razor** - Start with the simplest explanation 2. **Binary Search** - Isolate the problem area systematically 3. **Preserve Evidence** - Understand state before making changes 4. **Document Hypotheses** - Track what was tried and didn't work ## Debugging Workflow ``` 1. Understand → What is expected vs actual behavior? 2. Reproduce → Can you trigger the bug reliably? 3. Locate → Where in the code does it happen? 4. Diagnose → Why does it happen? (root cause) 5. Fix → Minimal change to resolve 6. Verify → Confirm fix works, no regressions ``` ## Common Bug Patterns | Symptom | Likely Cause | Check First | |---------|--------------|-------------| | TypeError/null | Missing null check | Input validation | | Off-by-one | Loop bounds, array index | Boundary conditions | | Race condition | Async timing | Await/promise handling | | Import error | Path/module resolution | File paths, exports | | Type mismatch | Wrong type passed | Function signatures | | Flaky test | Timing, shared state | Test isolation | ## Diagnose at the Failure Point — Sentinel Values and the Resource Reading Promoted from `~/.claude/rules/diagnose-at-the-failure-point.md`, whose stub keeps the two gate lines. When a tool or library reports a failure, two surface features routinely misdirect the diagnosis: the **named entity** in the error, and the **framing** you inherited going in. Both are cheap to check against reality at the exact point of failure. ### 1. A named entity in an error may be a sentinel/default, not a real object An error that names something by a **low / zero / default identifier** — `Memory page 0 doesn't exist`, `id 0`, an empty-string name, `index -1`, `0.0.0.0`, a null UUID — is often reporting an **uninitialized or sentinel default**, not a real entity that misbehaved. **Before theorizing on the entity, verify its identifier came from a real success** (a completed allocation / lookup / registration), not a fallback path. > Canonical break (2026-07, cubecl CUDA): `couldn't find resource for that > handle: Memory page 0 doesn't exist` was read — by a handoff issue, the > upstream tracker, and me for many iterations — as a memory-pool *reclaim > race* corrupting "page 0". It was `MemoryLocation::uninit() = {pool:0,page:0}`: > a **failed allocation** left its handle unbound at the zero default, and every > downstream lookup of an unbound handle reported "page 0." **Zero** pool > reclaims had fired. The tell was mechanical: the count of distinct "missing" > handles equalled the count of allocation failures **exactly**. The tell is usually a **count or ratio that matches something upstream** (missing-entity count == failure count) — look for it before accepting the entity as real. ### 2. For any exhaustion symptom, read the actual resource at the failure point `OOM` / `out of memory` / `can't allocate` / `ENOSPC` / `too many open files` / `pool exhausted` are **symptoms with three different fixes** — genuine exhaustion, fragmentation, or a manager/allocator bug — and one measurement at the failure point discriminates them: | Symptom | Read at the failure point | Genuine exhaustion iff | |---|---|---| | CUDA/GPU OOM | `cuMemGetInfo` (free, total) | free ≪ requested | | `ENOSPC` | `df` on the target fs | free ≈ 0 | | `EMFILE` / too many files | fd count vs `ulimit -n` | count ≈ limit | | allocator/pool "full" | pool `in_use` vs `reserved` vs device total | in_use ≈ device total | Instrument the *failing call site* to print this; don't infer it from a sampler taken seconds earlier (peak may be between samples). In the same case, `cuMemGetInfo` at the failing `malloc` showed **0.58 GB free of 25.2 GB** — genuinely full, not fragmented, not a pool bug. That one reading ended the allocator hunt: the working set simply exceeded the card. Every competing theory (reclaim race, cursor guard, sync-before-reclaim, a resolution "lever") had already died to a measurement, not an argument. **Evidence at the failure point > the framing you were handed > a plausible mechanism.** A confidently-written issue, a maintainer's stated root cause, and an error's own wording are all *inputs to verify*, not conclusions. When a theory and a measurement disagree, the measurement wins — re-diagnose, don't patch the theory. It bites when an error names an entity by a suspicious round/zero/default value, when the fix for a "resource exhausted" failure depends on *why*, and when an inherited handoff or upstream tracker already asserts a root cause. Skipping the one cheap reading trades a five-second `cuMemGetInfo`/`df`/count for a multi-iteration hunt down a mechanism that never occurred. Siblings: `agent-patterns-plugin:probe-input-integrity` (an id *you* invent), `git-plugin:git-issue-scoping` and `git-plugin:git-upstream-fix-check` (don't trust the inherited framing), `agent-patterns-plugin:tool-result-traps` (a well-formed output that silently doesn't match reality). ## Debugging Questions When stuck, ask: 1. What changed recently that could cause this? 2. Does it happen in all environments or just one? 3. Is the bug in my code or a dependency? 4. What assumptions am I making that might be wrong? 5. Can I write a minimal reproduction? ## Effective Debugging Practices - **Targeted changes**: Form a hypothesis, change one thing at a time - **Use proper debuggers**: Step through code with breakpoints when possible - **Find root causes**: Trace issues to their origin, fix the source - **Reproduce first**: Create a minimal reproduction before attempting a fix - **Verify the fix**: Confirm the fix resolves the issue and passes tests For system-level tools (memory analysis, profiling, strace/eBPF tracing, network debugging) and language-specific debugger commands, see [REFERENCE.md](REFERENCE.md).