debugging

name	debugging
description	Methodical debugging with evidence and hypothesis testing. Use when troubleshooting fails or root cause is unclear.

Methodical debugging using scientific method: gather evidence, form hypotheses, test systematically, verify fixes. Treats code you wrote with MORE skepticism than unfamiliar code — cognitive bias about "how it should work" is the enemy of debugging.

<core_principle> VERIFY, DON'T ASSUME. Every hypothesis must be tested. Every fix must be validated. No solutions without evidence.

Code you designed or wrote is guilty until proven innocent. Your intent doesn't matter — only the code's actual behavior.

This rule applies symmetrically to defensive code. When the user asks "is this even needed?" or "doesn't the library already do that?", run the underlying library/tool with the bad-input case BEFORE explaining why your code defends against it. Often the library already raises the exact exception you're catching — making your validation dead code. Concrete example: redis-py's RedisCluster(load_balancing_strategy="typo") accepts the bad string at construction and only raises on first read. A wrapper-level enum lookup that "fails loudly at startup" sounds defensible — until empirical test shows the wrapper isn't actually fast-failing earlier than the library would in practice. Run it before defending it. </core_principle>

<context_scan> Run at invocation to detect project type:

[ -f "package.json" ] && cat package.json | grep '"typescript"' > /dev/null && echo "DETECTED: TypeScript" || ([ -f "package.json" ] && echo "DETECTED: Node.js")
[ -f "Cargo.toml" ] && echo "DETECTED: Rust"
([ -f "pyproject.toml" ] || [ -f "setup.py" ] || [ -f "requirements.txt" ] || find . -maxdepth 2 -name "*.py" | head -1 | grep -q .) && echo "DETECTED: Python"
[ -f "go.mod" ] && echo "DETECTED: Go"
[ -f "pom.xml" ] || [ -f "build.gradle" ] && echo "DETECTED: Java"
(find . -maxdepth 2 -name "*.cpp" -o -name "*.cc" -o -name "CMakeLists.txt" | head -1 | grep -q .) && echo "DETECTED: C++"

If domain expertise skills exist (~/.claude/skills/expertise/), offer to load them before investigation. </context_scan>

<evidence_gathering> Before proposing any solution:

A. Document Current State

• What is the EXACT error message?
• What are the EXACT reproduction steps?
• What is ACTUAL vs EXPECTED output?
• When did this start?

B. Map the System

• Trace execution path from entry point to failure
• Identify all components involved
• Read relevant source files completely (don't skim)
• Note all dependencies, imports, configurations

C. Gather External Knowledge (when needed)

• Search for exact error messages
• Check library/framework docs for intended behavior
• Look for known issues, breaking changes, version quirks </evidence_gathering>

<root_cause_analysis> A. Form Hypotheses List possible causes with evidence:

1. [Hypothesis] — because [specific evidence]
2. [Hypothesis] — because [specific evidence]

B. Test Each Hypothesis For each:

• What would prove this true?
• What would prove this false?
• Design minimal test
• Document results

C. Eliminate or Confirm Don't proceed until: which hypothesis is supported by evidence? What evidence contradicts others? </root_cause_analysis>

<solution_development> Only after confirming root cause:

1. Design — What is the MINIMAL change that addresses the root cause?
2. Implement — Make the change with verification logging if needed
3. Test — Does the original issue still occur? Run reproduction steps. Check for regressions. </solution_development>

<critical_rules>

1. NO DRIVE-BY FIXES: If you can't explain WHY a change works, don't make it
2. ONE VARIABLE: Change one thing at a time, verify, then proceed
3. COMPLETE READS: Don't skim code. Read entire relevant files.
4. CHASE DEPENDENCIES: If libraries/configs/external systems are involved, investigate those too
5. QUESTION PREVIOUS WORK: Maybe the earlier "fix" was wrong. Re-examine with fresh eyes. When a hypothesis is refuted mid-investigation, do NOT immediately propose the next fix. Re-run evidence-gathering with the refuting observation now in scope. A dead hypothesis often carried a hidden assumption that also poisons the "obvious next thing to try" — the second fix will fail for the same reason unless you re-baseline first.
6. DON'T RAISE TO ROUTE: If a code path is unsafe in some contexts (wrong thread, missing dep, wrong env), split the call sites so the unsafe path is unreachable — don't raise and lean on a framework retry / failure-list to recover. Raise-as-control-flow is a band-aid: it leaks resources, hides intent, and couples your fix to framework internals. The correct fix routes at the call site (separate helpers, separate phases, separate executors).
7. MIRROR CI's ENV BEFORE GREEN: After any dep bump that may drop a transitive (sensi-* version change, redis-py major, etc.), pip uninstall -y <orphaned-transitive> (or pip install -r requirements.txt --force-reinstall in a fresh venv) BEFORE running pytest. A green local run in a polluted venv that still has the old transitive doesn't prove anything about CI's clean Docker build. The pre-push hook runs against your venv, not the wheel set requirements.txt produces.
8. CLEAN STATE BEFORE VERIFYING: Tear down stale containers/volumes/networks/locks before re-running any verification. Don't act-then-think. Common contamination sources: prior docker compose up left a partially-formed Redis Cluster nodes.conf; previous test run's temp files still on disk; observability query window overlapping with the failed run; local branch behind remote. Before ANY re-run, ask "what state would invalidate this result, and have I cleared it?" If unsure, investigate before running. Trust pushback when a result feels off — contamination is often the cause. When a verification "fails" but the harness output looks correct (table printed, artifact written, but the system-under-test returned 500s), distinguish a HARNESS bug from a SYSTEM bug — re-running won't fix the latter.
9. VERIFY THE LIBRARY ACCEPTS THE VALUE BEFORE REASONING ABOUT IT: When the user (or you) proposes a config knob change — 1/100milliseconds on a rate limiter, timeout=0.5s on a synchronous client, port=0 for ephemeral assignment — open a REPL and pass the value to the underlying library FIRST. Sub-second granularity, fractional seconds, custom granularity strings, value-of-zero special cases, units the parser doesn't recognize — every library has a value range, and "it sounds reasonable" is not evidence. The cost of one python -c "from <lib> import parse; print(parse('<value>'))" is two seconds; the cost of a long reasoned response that turns out to assume a value the parser rejects is the rest of the turn. Same rule applies symmetrically: if the user's intuition is "this should be configurable", verify it actually is before agreeing or disagreeing.
10. INCIDENT STABILIZERS ARE THE SMALLEST REVERSIBLE CHANGE: When prod is on fire, ship the env-var kill switch / feature flag / config flip in PR #1 — minimal lines, no behavior change in the off state, deployable today. File the structural fix (single-flight, lock manager, schema rewrite) as the NEXT ticket. Don't bundle them; you'll either ship the stabilizer slower waiting for the structural review, or ship the structural change without the review rigor it needs. The two changes optimize for different things: the stabilizer optimizes for time to mitigation, the structural fix optimizes for durability. Conflating them costs you both. Incident kill switches should default to the SAFE state (e.g., for a "limiter disabled" knob during a thundering-herd incident, enabled=False is the default — re-enable per env when the underlying race is fixed, don't preserve the broken-in-prod behavior as the default).
11. TWO STRIKES AGAINST A BLACK-BOX = SWITCH MODES: When a fix targets an opaque system (docker daemon, container orchestrator, cloud API, network fabric, third-party service) and two attempts fail without producing a validated hypothesis, STOP retrying. Return to <evidence_gathering> — pull logs, docker inspect, network traces, or reduce to a minimal reproducer. Trial-and-error against a black-box burns turns and produces false-signal fixes; a change that "makes it green" without an explained mechanism is a coincidence, not a fix. If two retries didn't reveal the cause, the third won't either. Same rule symmetrically for CI: green in CI + red locally against the same commit points to environment state (daemon, cached images, orphaned volumes, DNS), not code — investigate the delta, don't keep re-running.
12. VERIFY THE OBSERVABILITY BACKEND'S ACTUAL FIELD NAMES BEFORE FILTERING: Log-search backends (Groundcover, Datadog, CloudWatch Logs, ELK, Splunk, Loki) each expose different canonical fields for tenant scoping — and different tenants on the same backend often diverge. Before filtering by env=prod, service=X, namespace=Y, src_cluster=Z, run one values-probe query first (| field_values <candidate> on gcQL; fields @log | stats count() by <field> on CloudWatch Insights; _field_names / | label_values on Loki; fields.list on Datadog) and confirm the field exists AND has non-empty values in your window. Filtering on the wrong field silently returns zero results — indistinguishable from "the bug isn't there," which routes you toward wrong hypotheses. Two-second probe beats a two-minute chase. Same shape as rule 9 (verify library accepts a value) but at the schema layer, not the value layer. </critical_rules>

<output_format>

## Issue: [Problem Description]

### Evidence
[Exact errors, behaviors, outputs observed]

### Investigation
[What you checked, found, and ruled out]

### Root Cause
[The actual underlying problem + evidence]

### Solution
[What you changed and WHY it addresses the root cause]

### Verification
[How you confirmed it works and doesn't break anything else]

</output_format>

<language_examples>

Python — Adding Debug Points

# Temporary debug prints (remove after fixing)
import json
print(f"DEBUG input: {json.dumps(data, indent=2, default=str)}")
print(f"DEBUG result type: {type(result).__name__}, value: {result!r}")

# Using pdb for interactive debugging
import pdb; pdb.set_trace()  # drops into debugger at this line

# pytest debugging — run with -s flag to see prints
# pytest -s -x tests/test_problematic.py::test_specific_case

Python — Minimal Reproduction

# Isolate the bug — strip everything non-essential
def test_reproduce_bug():
    """Minimal reproduction of the issue."""
    # Arrange: minimum state needed
    data = {"key": "value"}

    # Act: the problematic operation
    result = process(data)

    # Assert: what should happen vs what does happen
    assert result == expected_value, f"Got {result!r}, expected {expected_value!r}"

Node.js — Debug Points

// console.debug with structured data
console.debug('DEBUG processOrder:', { orderId, items: items.length, total });

// Using Node.js debugger
debugger; // pause here when running with --inspect

// Run specific test with verbose output
// npx jest --testNamePattern "specific test" --verbose --no-coverage

Node.js — Async Debugging

// Common async bug: unhandled rejection
async function debugAsyncIssue() {
  try {
    const result = await problematicOperation();
    console.debug('result:', result);
  } catch (err) {
    // Log the FULL error including stack
    console.error('FULL ERROR:', err);
    console.error('Stack:', err.stack);
    throw err; // re-throw after logging
  }
}

</language_examples>

<success_criteria> Before starting:

• Context scan run to detect project type

During investigation:

• Root cause identified with evidence (not just "it works now")
• Fix verified against original reproduction steps
• Adjacent functionality checked for regressions
• Can explain the solution to another developer
• One variable changed at a time (no simultaneous changes)
• Would pass code review scrutiny </success_criteria>

<reference_index> All in references/:

• debugging-mindset.md — Cognitive biases, first-principles thinking
• hypothesis-testing.md — Forming and testing falsifiable hypotheses
• investigation-techniques.md — Binary search, minimal reproduction, rubber duck
• verification-patterns.md — What "verified" actually means </reference_index>