debugging-methodology

name	debugging-methodology
description	Use when the user says "I have a bug", "this isn't working", "help me debug", "I can't reproduce this", "this works locally but not in production", or needs to trace a failure to its root cause. Provides systematic debugging methodology for software engineers.

Debugging Methodology

Debugging is hypothesis generation and testing under time pressure. The worst strategy is random mutation — changing things and hoping. The best strategy is systematic: reproduce, isolate, hypothesize, verify. Every debugging session should end with a root cause, not just a fix.

When to Activate

• A bug is reported and the cause is unknown
• A test is failing unexpectedly
• A production failure needs diagnosis
• "It works on my machine" situations
• Performance unexpectedly degrades

The Debugging Loop

1. REPRODUCE → get a reliable, minimal reproduction
2. OBSERVE  → gather data without forming conclusions yet
3. HYPOTHESIZE → form one specific hypothesis
4. TEST      → design an experiment that falsifies or confirms
5. CONCLUDE  → root cause identified OR loop back to step 3
6. FIX       → fix the cause, not the symptom
7. VERIFY    → confirm the fix, add a regression test

Never skip to step 3. The most common debugging mistake is forming a hypothesis before observing enough data.

Step 1: Reproduce

A bug you cannot reproduce is a bug you cannot fix. Invest in reproduction before diagnosis.

Reproduction checklist:

• Can you reproduce it locally? If not, what's different about the environment?
• Is it deterministic or intermittent?
• What is the minimal input that triggers it?
• Does it reproduce in a clean environment (no local state, fresh database)?

Intermittent bugs: require either a way to increase the reproduction rate or extensive logging to catch the failure in the act. Never debug an intermittent bug by staring at the code — instrument and wait.

Environment parity: if it fails in production but not locally, the environment is part of the bug. Systematically enumerate differences: OS, dependencies, config, data shape, load, time.

Step 2: Observe

Before forming a hypothesis, gather all available data:

• What is the exact error message or incorrect output?
• What is the stack trace? Read the full stack, not just the top frame.
• What do the logs say immediately before the failure?
• What was the system state? (memory, CPU, connections, queue depth)
• When did this start? Was there a recent deployment, config change, or traffic spike?

Read the error message. Engineers routinely skip this step and spend hours investigating the wrong thing. The message often names the problem.

Step 3: Hypothesize

A good hypothesis is:

• Specific: "the connection pool is exhausted" not "it's a database problem"
• Falsifiable: there is an experiment that would prove it wrong
• Grounded: based on observed data, not intuition

Generate multiple hypotheses ranked by likelihood. Work the most likely one first, but be willing to discard it when the data doesn't fit.

Step 4: Test

Design the minimal experiment that would falsify the hypothesis:

• Add a log statement at the suspected point
• Isolate the code in a unit test with the failing input
• Comment out the suspected code path
• Reproduce with a known-good vs known-bad configuration side by side

Binary search debugging: if you don't know where in the code the failure occurs, use bisection. Confirm the failure exists at point A and not at point B, then test the midpoint. Repeat. This works for both code bisection (git bisect) and logic bisection (add assertions at 50%, then 25%, etc).

git bisect: for regressions, automate the binary search across commits. git bisect start, git bisect bad HEAD, git bisect good <last-known-good-commit>. Let git find the introducing commit.

Common Bug Patterns

Off-by-One

Boundary conditions: index 0 vs 1, < vs <=, empty collection handling. Always test with: empty input, single-element input, boundary values.

Race Condition

Symptoms: intermittent failures, failures under load, failures that disappear when you add logging. Diagnosis: look for shared mutable state accessed without synchronization. Add thread IDs to logs to reconstruct interleaving.

Null/Undefined Dereference

Read the full stack trace — the null was usually introduced several frames above where the dereference occurs. Find where the null-producing value originates, not where it explodes.

Incorrect Assumptions About External Systems

The external API does not behave as documented. The database returns rows in a different order. The time zone is not UTC. The float comparison is not exact. Verify assumptions with logging and assertions at every boundary.

State Mutation

A value changes unexpectedly because it is shared by reference and mutated elsewhere. Symptoms: a value is correct when logged in function A but wrong by the time it reaches function B. Add immutability, clone objects at boundaries, or add assertions on the expected state.

Memory Leak

Symptoms: memory grows monotonically over time, eventually causing OOM or slowdown. Diagnosis: take heap snapshots at intervals and diff them. Look for collections that grow without bound, listeners that are registered but not removed, or cache entries that are never evicted.

Production Debugging

What You Have vs What You'd Want

In production, you cannot attach a debugger. You have:

• Logs (quality depends on what was instrumented before the incident)
• Metrics (aggregate, not per-request)
• Distributed traces (if you built the infrastructure)
• Error reports (Sentry, Bugsnag, etc.)

Safe Production Debugging Techniques

• Add verbose logging to a single instance — route a fraction of traffic and log everything
• Feature flag to disable suspected code paths — confirms whether the path is the problem
• Replay production requests — capture and replay the failing request against a staging environment
• Tail logs in real time — kubectl logs -f, CloudWatch Logs Insights, Datadog Live Tail

Never run UPDATE or DELETE without a WHERE clause. Never modify production data without a backup and a rollback plan.

Root Cause vs Symptom

A fix that addresses the symptom without the root cause will recur. Root cause analysis asks "why" five times:

Bug: payment service times out
Why? Database queries are slow
Why? The payments table has 500M rows with no index on status
Why? The index was dropped during a migration 6 weeks ago
Why? The migration script did not include index recreation
Why? There was no post-migration validation checklist
Root cause: Missing migration validation process

Fix: add the index. Prevent recurrence: add migration validation to the deployment checklist.

After the Fix

1. Write a regression test that fails without the fix and passes with it
2. Document the root cause in the commit message or PR description
3. Consider whether the same bug exists elsewhere
4. If the bug reached production, write a postmortem (see incident-response)

Gotchas

1. Fixing the symptom and calling it done: a crash that disappears when you add a null check is not fixed — the null is still being produced somewhere. Find the source.

2. Debugging in production without observability: if you cannot answer "what happened?" from logs and metrics, your next priority is instrumentation, not features.

3. Assuming the bug is in your code: the bug could be in a library, the framework, the OS, the infrastructure, or your assumptions about an external service. Eliminate your code as the cause before blaming others — but don't assume it's you either.

4. Debugging alone for too long: if you've been stuck for more than 30-60 minutes, explain the problem to someone else. Rubber duck debugging works because articulation forces structured thinking.

5. Changing multiple things at once: you will not know which change fixed the bug. Change one thing, observe the result, then change the next thing.

6. Not writing the regression test: the fix without the test allows the same bug to be re-introduced. The test is not optional.

Integration

• incident-response — production failures require the debugging loop plus stakeholder communication
• testing-strategy — regression tests prevent re-introduction of fixed bugs
• performance-engineering — performance bugs require profiling in addition to the debugging loop
• code-review — reviewing code with debugging eyes catches bugs before they ship

References

• Root Cause Tracing Guide — structured templates for 5-why analysis and postmortem preparation

Skill Metadata

Created: 2026-04-10 Version: 1.0.0