Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Systematic debugging methodology covering root cause analysis, log-based debugging, production debugging, memory leak diagnosis, deadlock detection, performance anomaly investigation, and reproducible bug reproduction. Distilled from real experience at Google, DeepMind, ByteDance, and Huawei.
USE WHEN: investigating production incidents, debugging hard-to-reproduce bugs, analyzing crash dumps, finding race conditions, diagnosing memory leaks, debugging performance regressions, or any situation requiring systematic root cause analysis. Triggers on "debugging", "root cause", "bug", "crash", "segfault", "deadlock", "race condition", "memory leak".
Debugging Mastery
Source: "Debugging" (David Agans) + Google/DeepMind/ByteDance production
debugging experience + years of midnight production incidents
Core Philosophy: "Debugging is the art of systematically testing hypotheses
until the root cause is found. It is NOT randomly changing things hoping."
The Nine Indispensable Rules
From David Agans' "Debugging" — the debugging bible:
1. Understand the system
2. Make it fail
3. Quit thinking and look
4. Divide and conquer
5. Change one thing at a time
6. Keep an audit trail
7. Check the plug
8. Get a fresh view
9. If you didn't fix it, it ain't fixed
1. The Debugging Workflow
1.1 Systematic RCA Flow
┌──────────────────────────────────────────────────────────────┐
│ Step 1: REPRODUCE │
│ Can you make it happen? │
│ If not: this is the first problem to solve │
│ │
│ Step 2: ISOLATE │
│ Binary search through the system to find the component │
│ Remove variables: isolate the MINIMAL reproducing case │
│ │
│ Step 3: MEASURE │
│ Log everything. Add MORE logging if needed. │
│ "The resolution of your debugging is limited by the │
│ resolution of your instrumentation" │
│ │
│ Step 4: HYPOTHESIZE │
│ Form a specific, testable hypothesis │
│ Bad: "Maybe there's a memory issue" │
│ Good: "The cache eviction runs on the wrong goroutine" │
│ │
│ Step 5: TEST THE HYPOTHESIS │
│ If hypothesis is wrong → return to Step 3 │
│ If hypothesis is right → FIX it │
│ │
│ Step 6: VERIFY THE FIX │
│ Run the reproduction case again (it should pass) │
│ Run the full test suite │
│ Add the reproduction case as a REGRESSION TEST │
└──────────────────────────────────────────────────────────────┘
1.2 The Binary Search Debugging Technique
Step through the system components:
┌──────────────────────────────────────────────────────────────┐
│ Input → [A] → [B] → [C] → [D] → [E] → Output │
│ │
│ Check output at C: │
│ ✅ C passes → bug is in D or E │
│ ❌ C fails → bug is in A, B, or C │
│ │
│ Repeat: check at B (or D, depending on result) │
│ Continue until you've isolated the single component │
│ │
│ This is O(log n) — exponentially faster than linear scan │
└──────────────────────────────────────────────────────────────┘
┌──────────┬────────────────────────────────────────────────────┐
│ TRACE │ Every function entry/exit (noisy, turn off by │
│ │ default) │
├──────────┼────────────────────────────────────────────────────┤
│ DEBUG │ Detailed state dumps, intermediate values │
│ │ Enable when investigating specific issue │
├──────────┼────────────────────────────────────────────────────┤
│ INFO │ Normal operations, key business events │
│ │ "Order created", "Payment confirmed" │
├──────────┼────────────────────────────────────────────────────┤
│ WARN │ Anomalous but non-critical │
│ │ "Retry attempt 2/3", "Cache miss" │
├──────────┼────────────────────────────────────────────────────┤
│ ERROR │ Something is broken — needs investigation │
│ │ "Database connection failed", "Timeout exceeded" │
└──────────┴────────────────────────────────────────────────────┘
2.3 Finding the Needle in the Haystack
# Find all ERROR lines in last hour
journalctl -u your-service --since "1 hour ago" | grep ERROR
# Follow logs in real-time for a specific requesttail -f /var/log/app.log | grep "order-123"# Extract structured logs for a trace IDcat app.log | grep "trace_id=abc123" | jq .
# Show how long something took (timing from structured logs)cat app.log | grep "order.processed" | awk '{print $2}' | sort | uniq -c
# Count error types
grep ERROR app.log | grep -oP '"errorCode":"[^"]*"' | sort | uniq -c | sort -rn
# Find slow operations (duration > 1000ms)cat app.log | jq 'select(.duration > 1000)' | jq -r '.message + ": " + (.duration|tostring)'
3. Production Debugging
3.1 The Production Debugging Checklist
□ Is it happening NOW?
→ Check monitoring dashboard (latency, error rate, saturation)
→ Check alerting for related incidents
→ Check recent deployments/configuration changes
□ Is it a known issue?
→ Search internal KB / runbooks / postmortems
→ Search Slack history for similar symptoms
→ Check GitHub issues / bug tracker
□ Can we observe the problem?
→ Check logs (see section 2)
→ Check metrics (CPU, memory, disk, network, GC)
→ Check distributed traces (Jaeger/Tempo)
□ Can we reproduce it in staging?
→ Same deployment version
→ Same data or data pattern
→ Same traffic pattern
→ If not reproducible → add more instrumentation
3.2 Safe Production Debugging
# IMPORTANT: Never SSH into production unless absolutely necessary# Try these first:# 1. Check monitoring (Grafana, Datadog, etc.)# 2. Check logs (Loki, Splunk, ELK)# 3. Check traces (Jaeger, Tempo)# 4. Check health endpoints
curl -s http://service:8080/health | jq .
curl -s http://service:8080/metrics | head -50
curl -s http://service:8080/debug/vars # Go expvar# 5. Only if above fails → limited safe access
kubectl exec -it pod/app -- /bin/sh -c "curl localhost:8080/debug/pprof/heap"# NEVER modify production data# NEVER kill/restart services without understanding the impact# ALWAYS have a rollback plan before making any change
3.3 Postmortem-Driven Debugging
When you encounter a bug, ask:
❓ "What type of bug is this?"
- Logic bug (wrong condition, missing case)
- Concurrency bug (race, deadlock, stale data)
- Data bug (corruption, encoding, validation)
- Configuration bug (wrong env, wrong feature flag)
- Dependency bug (upstream change, API drift)
- Resource bug (memory, file handles, connections)
❓ "Does this bug belong to a known category?"
If yes → apply known pattern fix
If no → write a new postmortem entry about this pattern
4. Debugging by Bug Type
4.1 Concurrency / Race Conditions
// Symptom: intermittent crashes, wrong values, data corruption// Root cause detection:// Go race detectorgo test -race ./...
go run -race ./...
// The output pinpoints the exact lines where race occurs:// WARNING: DATA RACE// Read at 0x00c00028c010 by goroutine 7:// main.updateCounter()// main.go:42 +0x39// Previous write at 0x00c00028c010 by goroutine 5:// main.incrementCounter()// main.go:38 +0x58
Symptoms: RAM grows over time, GC overhead increases, OOM crashes
Debug flow:
1. Take a heap snapshot (time 0)
2. Run operation n times
3. Take another heap snapshot
4. Compare: what grows?
- Growing maps (map without cleanup) ← common
- Growing slices (append without limit) ← common
- Event listeners without removal
- Closed-over variables in callbacks
- Cached objects with no eviction
4.3 Deadlocks
// Go: detect deadlocks in testsgo test -v -timeout=5s ./...
// Go race detector catches lock ordering issuesgo test -race ./...
// Debug deadlock: get all goroutine stacksimport"net/http/pprof"funcmain() {
// ...
}
// Then get stacks:
curl http://localhost:6060/debug/pprof/goroutine?debug=2// Look for:// goroutine 1 [chan receive]:// main.waitForResult()// main.go:25 +0x45// goroutine 2 [chan receive]:// main.waitForResult()// main.go:25 +0x45// → Two goroutines waiting on each other = deadlock
4.4 Heisenbugs (Bugs that Disappear When You Look)
Symptoms:
- Adding a log line "fixes" the bug
- Debugger breakpoints "fix" the bug
- Bug only happens in production, never staging
Causes:
- Timing-dependent bugs (race, channel, timeout)
- Buffer flush / log delay timing
- Heisenberg uncertainty principle of debugging:
"The act of observing changes the behavior"
Debug strategies:
1. Use structured logging (less I/O impact than console.log)
2. Use tcpdump / strace (observe without modifying)
3. Add counters instead of log lines:
metrics.counter('bug_scenario.hit').inc()
4. Capture state, don't log it:
Take periodic snapshots → analyze offline
4.5 Non-Deterministic Bugs
Symptoms:
- "Sometimes it works, sometimes it doesn't"
- "I can't reproduce it"
- "It only happens on Tuesdays"
The 5 most common causes of non-determinism:
1. Uninitialized memory → read before write
2. Map iteration order (random in many languages)
3. Goroutine/thread scheduling order
4. Network timing / retry interactions
5. Hash collision / random seed
# Process status
top -H -p <PID> # Show per-thread CPU
strace -p <PID> -e trace=network # Trace syscalls
lsof -p <PID> # Open file descriptorsls /proc/<PID>/fd/ | wc -l # Count open files# Network
ss -tulpn # List listening ports
tcpdump -i eth0 port 8080 -w capture.pcap # Packet capture# Disk
iostat -x 1 # Disk I/O statsdf -h # Disk spacedu -sh /path # Directory size# System
dmesg | tail -20 # Kernel messages (OOM kills!)
free -m # Memoryulimit -a # Resource limits
6. The Debugging Mindset
6.1 What Great Debuggers Do Differently
As a junior dev, I'd panic and change random things.
As a senior engineer, I use the scientific method.
┌──────────────────────────────────────────────────────────────┐
│ Junior Engineer: │
│ "Oh no! The database is failing! Let me restart it!" │
│ → Restarts the DB → "It works now!" │
│ → Same bug happens tomorrow │
│ │
│ Senior Engineer: │
│ "The database is failing. Let me check the logs." │
│ → Finds "disk space 100% full" │
│ → Cleans up old data, sets up disk alert │
│ → Bug never comes back │
└──────────────────────────────────────────────────────────────┘
6.2 The Two-Day Rule
If you can't fix a bug after 2 hours of focused debugging:
1. Step away from the keyboard (5-10 minutes)
→ Fresh perspective is the #1 debugging tool
2. Explain the bug to someone else (rubber duck debugging)
→ Saying it out loud forces clarity
3. Write down what you KNOW vs what you ASSUME
→ Most stuck bugs come from a wrong assumption
4. Ask yourself:
"What would have to be TRUE for this bug to reproduce?"
"What evidence DISPROVES my current hypothesis?"
5. If still stuck after 2 more hours:
→ Escalate or pair with someone who hasn't seen the issue
6.3 The Rubber Duck Debugging
# Pseudocode execution:defrubber_duck_debug(code_bug):
"""
Explain the code line by line to a rubber duck.
The duck doesn't know anything, so you need to
be precise enough that a complete beginner would
understand.
90% of the time, you find the bug mid-explanation.
"""whilenot bug_found:
for line in code_bug:
explain_out_loud(line)
7. Debugging Anti-Patterns
❌ "Let me just try restarting it"
→ You learned nothing. Same bug will return.
❌ "Let me change this randomly and see if it helps"
→ If it "fixes" the bug, you still don't know WHY.
→ The "fix" might have introduced a different, worse bug.
❌ "I checked the code, it looks fine"
→ The code is NOT fine — the bug proves it.
→ The bug is in the gap between "what you think the code does"
and "what the code actually does."
❌ "It must be a compiler/interpreter bug"
→ It's NEVER the compiler. (Google's postmortem: zero compiler bugs)
→ The compiler is better tested than your code.
❌ "This worked yesterday, nothing changed!"
→ Something ALWAYS changed. Find it.
→ Deployment, config, data, traffic pattern, time of day.
