| name | debugging |
| description | Systematic debugging methodology, log analysis, and error diagnosis for software development workflows. |
Debugging
Systematic debugging methodology for identifying, diagnosing, and resolving software issues efficiently.
Tier 1: Guided Debugging
What This Skill Does
Helps you debug software issues systematically using proven debugging methodologies and error analysis techniques.
When to Use This Skill
- Error Analysis: Understanding error messages and stack traces
- Log Investigation: Analyzing application logs to identify issues
- Reproduction: Creating minimal reproducible examples
- Hypothesis Testing: Testing hypotheses about root causes
- Isolation: Isolating the problem scope (code, data, environment)
Guided Debugging Workflow
Step 1: Define the Problem
Questions to Answer:
- What is the exact symptom or error?
- When does it occur (timing, frequency)?
- Under what conditions (environment, data, user action)?
- What is expected vs. actual behavior?
Template:
Problem Definition:
- Symptom: [what went wrong]
- Occurs when: [timing/conditions]
- Expected: [what should happen]
- Actual: [what actually happens]
- Impact: [how severe is this?]
Step 2: Gather Information
Sources to Check:
- Error messages and stack traces
- Application logs (debug, info, error levels)
- System logs (OS, network, database)
- User reports (screenshots, steps to reproduce)
- Recent changes (git log, deployments, config updates)
Information Collection Checklist:
✓ Error message text
✓ Stack trace (if available)
✓ Relevant log entries
✓ Reproduction steps
✓ Environment details (OS, versions, config)
✓ Recent changes
Step 3: Reproduce the Issue
Reproduction Strategy:
- Start from a clean state (fresh environment)
- Follow the exact steps to trigger the issue
- Note any variables that affect reproduction
- Try variations to understand the scope
Minimal Reproduction:
Minimal Example:
- Simplify inputs
- Reduce dependencies
- Remove unrelated code
- Isolate the failing component
Step 4: Formulate Hypotheses
Hypothesis Generation:
- What could cause this symptom?
- What changed recently?
- What environmental factors could be involved?
- What code paths are involved?
Hypothesis Template:
Hypothesis: [I believe X is causing Y]
Because: [evidence/reasoning]
Test: [how to verify]
Expected: [if hypothesis is true, what happens?]
Step 5: Test Hypotheses
Testing Approach:
- Design a test to prove or disprove hypothesis
- Execute test and observe results
- Document findings
- Formulate new hypotheses if needed
Test Documentation:
Test: [what you tested]
Result: [pass/fail with evidence]
Conclusion: [what this tells us]
Next: [next step based on result]
Step 6: Isolate the Root Cause
Isolation Techniques:
- Binary search (divide code in half)
- Comment out code blocks
- Add debug logging at key points
- Use breakpoints (if using debugger)
- Test in different environments
Isolation Questions:
- Is it a data issue?
- Is it a logic error?
- Is it a configuration issue?
- Is it a dependency issue?
- Is it a race condition?
Step 7: Implement Fix
Fix Strategy:
- Make minimal changes
- Test the fix thoroughly
- Consider edge cases
- Document the fix
- Add regression tests
Fix Checklist:
✓ Addresses root cause (not just symptoms)
✓ Minimal code changes
✓ Tested with reproduction case
✓ Tested edge cases
✓ Doesn't break existing functionality
✓ Documented with comments
Step 8: Verify Resolution
Verification Steps:
- Reproduce the original issue - should not occur
- Run existing tests - all should pass
- Add regression tests
- Monitor for recurrence
- Document for future reference
Common Debugging Patterns
Pattern 1: Divide and Conquer
- Comment out half the code
- If issue disappears, problem is in commented code
- If issue persists, problem is in remaining code
- Repeat until isolated
Pattern 2: Rubber Duck Debugging
- Explain the problem line by line to someone (or a rubber duck)
- Often leads to realizing the solution
- Forces careful code analysis
Pattern 3: Log Injection
- Add debug logs at key points
- Trace execution flow
- Identify where behavior diverges
- Remove or upgrade logs after fixing
Pattern 4: Environment Testing
- Test in development vs. production
- Test with different data sets
- Test with different configurations
- Identify environment-specific issues
Integration with Other Skills
Works With:
- code-review: Debugging insights improve code reviews
- code-refactoring: Debugging reveals refactoring opportunities
- quick-research: Research error messages and patterns
- knowledge-graph: Link bugs to related issues and fixes
Tier 2: Confident Debugging
Advanced Debugging Techniques
1. Systematic Root Cause Analysis
5 Whys Technique:
Problem: [issue]
Why 1: [first reason]
Why 2: [second reason]
Why 3: [third reason]
Why 4: [fourth reason]
Why 5: [root cause]
Fishbone Diagram:
- Identify main categories of causes (People, Process, Technology, Environment)
- Brainstorm potential causes within each category
- Test and eliminate causes systematically
Example Categories:
- People: Code changes, deployment, configuration
- Process: Testing, review, deployment
- Technology: Dependencies, frameworks, libraries
- Environment: OS, network, database, external services
2. Advanced Log Analysis
Log Query Patterns:
# Find all errors in last hour
grep -i "error" application.log | tail -100
# Find errors by user
grep "user_123" application.log | grep -i "error"
# Correlate errors across services
grep -i "error" app1.log | awk '{print $3}' | xargs -I{} grep {} app2.log
# Time-based analysis
awk '/2026-03-27 14:00/,/2026-03-27 15:00/' application.log
Log Aggregation Strategies:
- Centralized logging (ELK stack, Splunk, Graylog)
- Structured logging (JSON, key-value pairs)
- Log correlation by request ID
- Real-time log monitoring
3. Performance Debugging
Profiling Approach:
1. Identify performance bottleneck
2. Measure current performance (baseline)
3. Profile the code (CPU, memory, I/O)
4. Identify hot spots
5. Optimize targeted areas
6. Measure improvement
7. Repeat if needed
Common Performance Issues:
- N+1 query problems (databases)
- Inefficient algorithms (O(n²) instead of O(n log n))
- Memory leaks (unreleased resources)
- Excessive logging (I/O bottleneck)
- Blocking operations (synchronous I/O)
Profiling Tools:
- Python: cProfile, Py-Spy, memory_profiler
- JavaScript: Chrome DevTools, Node.js profiler
- Java: JProfiler, YourKit, VisualVM
- Go: pprof, go tool trace
4. Concurrency Debugging
Common Concurrency Issues:
- Race conditions (timing-dependent bugs)
- Deadlocks (circular waits)
- Livelocks (resource contention)
- Memory consistency (visibility across threads)
Debugging Strategies:
1. Identify shared resources
2. Analyze access patterns
3. Check for proper synchronization
4. Look for non-atomic operations
5. Verify lock ordering (deadlock prevention)
6. Test with different thread counts
7. Use thread sanitizers (when available)
Thread Sanitizer Examples:
gcc -fsanitize=thread -g program.c -o program
./program
go test -race ./...
5. Memory Debugging
Memory Leak Detection:
1. Take memory snapshot (baseline)
2. Perform operations that might leak
3. Take another memory snapshot
4. Compare snapshots - look for growing objects
5. Trace references to find why objects aren't collected
Common Memory Issues:
- Unclosed resources (file handles, connections)
- Circular references
- Caches without limits
- Event listeners not removed
- Global collections growing unbounded
Memory Profiling Tools:
- Python: tracemalloc, objgraph, memray
- JavaScript: Chrome DevTools Memory Profiler
- Java: VisualVM, YourKit, JProfiler
- Node.js: heapdump, v8-profiler
6. Distributed System Debugging
Challenges:
- Partial failures (some services fail)
- Network partitions (services can't communicate)
- Timing issues (clock skew, latency)
- Complexity (many moving parts)
Distributed Tracing:
1. Add trace IDs to requests
2. Propagate trace IDs across services
3. Log trace IDs in all services
4. Use tracing tools (Jaeger, Zipkin, Datadog APM)
5. Analyze request flows and latency
Debugging Checklist:
✓ Trace the request through all services
✓ Check for timeouts and retries
✓ Verify data consistency
✓ Check network connectivity
✓ Review recent deployments
✓ Check configuration changes
✓ Look for resource exhaustion
Debugging Best Practices
1. Log Effectively
Log Levels:
- DEBUG: Detailed information for debugging
- INFO: General information about program execution
- WARNING: Unexpected but recoverable issues
- ERROR: Errors that don't prevent program execution
- CRITICAL: Serious errors that prevent program execution
Structured Logging:
{
"timestamp": "2026-03-27T14:30:00Z",
"level": "ERROR",
"service": "payment-service",
"request_id": "abc-123",
"user_id": "user-456",
"message": "Payment failed",
"error_code": "PAYMENT_DECLINED",
"amount": 99.99,
"currency": "USD"
}
2. Write Testable Code
Testing Debugging Hypotheses:
def test_bug_reproduction():
data = {"value": 42}
result = buggy_function(data)
assert result == expected_value
3. Use Debuggers Effectively
Common Debugger Features:
- Breakpoints (pause execution at specific line)
- Watchpoints (pause when variable changes)
- Step through code (line-by-line execution)
- Inspect variables (view current values)
- Evaluate expressions (run code in context)
- Call stack navigation (see how you got here)
4. Document Bugs
Bug Report Template:
Title: [Concise description]
Description:
[Detailed explanation of the issue]
Steps to Reproduce:
1. [Step 1]
2. [Step 2]
3. [Step 3]
Expected Behavior:
[What should happen]
Actual Behavior:
[What actually happens]
Environment:
- OS: [operating system]
- Version: [software version]
- Configuration: [relevant settings]
Attachments:
- Screenshots
- Logs
- Code snippets
Advanced Integration Patterns
Multi-Skill Workflows:
1. Use quick-research to understand error messages
2. Use knowledge-graph to find similar past issues
3. Use code-review to find related code patterns
4. Use debugging skill to isolate and fix issue
5. Use code-refactoring to prevent recurrence
6. Use workflow-orchestrator to automate debugging tasks
Tier 3: Anticipatory Debugging
Proactive Debugging Strategies
1. Predictive Error Detection
Pattern Recognition:
- Learn from past bugs and their symptoms
- Identify code patterns that lead to bugs
- Predict likely failure points
- Suggest preemptive fixes
Implementation:
1. Track bug history (types, causes, locations)
2. Identify high-risk code areas
3. Code smells analysis (complexity, coupling, coverage)
4. Suggest refactoring or testing
Risk Indicators:
- High code complexity (cyclomatic complexity)
- Low test coverage
- Frequent recent changes
- Known error-prone patterns
- Dependencies with known issues
2. Automated Debugging
Self-Diagnosing Systems:
1. System monitors its own health
2. Detects anomalies (metrics, logs, errors)
3. Automatically collects debugging information
4. Attempts self-recovery (restart, rollback)
5. Escalates to human if needed
Health Checks:
def health_check():
checks = {
"database": check_database(),
"cache": check_cache(),
"api": check_external_api(),
"disk_space": check_disk_space(),
}
return all(checks.values()), checks
3. Debugging Infrastructure
Debugging Tools Integration:
- Error tracking (Sentry, Rollbar, Bugsnag)
- Performance monitoring (Datadog, New Relic, AppDynamics)
- Log aggregation (ELK, Splunk, Graylog)
- Distributed tracing (Jaeger, Zipkin, Datadog APM)
Automated Alerting:
1. Define alert conditions (error rate, latency, anomalies)
2. Set up monitoring dashboards
3. Configure alert notifications (email, Slack, PagerDuty)
4. Create runbooks for common issues
5. Automate initial diagnosis steps
4. Debugging Knowledge Base
Building a Debugging Knowledge Graph:
1. Document all bugs and their fixes
2. Link bugs to symptoms, causes, and solutions
3. Tag by component, severity, frequency
4. Enable semantic search for bug patterns
5. Learn from debugging history
Knowledge Base Schema:
{
"bug_id": "BUG-123",
"symptoms": ["error message", "stack trace"],
"root_cause": "null pointer dereference",
"fix": "add null check",
"prevention": "add defensive coding",
"related_bugs": ["BUG-100", "BUG-200"],
"component": "payment-service",
"frequency": 5,
"severity": "high"
}
Expert Debugging Techniques
1. Binary Debugging
When Needed:
- Core dumps and segmentation faults
- Memory corruption
- Assembly-level debugging
- Performance optimization
Tools:
- GDB (GNU Debugger)
- LLDB (LLVM Debugger)
- WinDbg (Windows)
- Binary analysis tools (IDA Pro, Ghidra)
2. Kernel Debugging
Challenges:
- No user-space debugging
- System crashes
- Hardware interactions
- Concurrency issues
Tools:
kdump and crash (Linux)
dtrace and perf (Linux)
eBPF (Linux)
systemtap (Linux)
- Windows Performance Toolkit
3. Network Debugging
Common Issues:
- Connection timeouts
- Packet loss
- MTU problems
- DNS resolution
- SSL/TLS errors
Tools:
tcpdump and wireshark (packet capture)
netstat and ss (network connections)
traceroute and ping (routing)
dig and nslookup (DNS)
openssl s_client (SSL/TLS)
4. Database Debugging
Common Issues:
- Slow queries
- Lock contention
- Deadlocks
- Data corruption
- Replication lag
Tools:
- Query analysis (
EXPLAIN, EXPLAIN ANALYZE)
- Slow query logs
- Profiling tools (
pg_stat_statements, Performance Schema)
- Deadlock logs
- Replication monitoring
Debugging Culture
1. Blameless Post-Mortems
Post-Mortem Format:
1. Executive Summary
- What happened
- Impact
- Timeline
2. Root Cause Analysis
- How did it happen
- Contributing factors
- Systemic issues
3. Resolution
- What we did to fix it
- Timeline of fixes
4. Lessons Learned
- What went well
- What didn't
- What to improve
5. Action Items
- Prevent recurrence
- Improve detection
- Better documentation
Blameless Principles:
- Focus on process, not people
- Assume good intent
- Understand systemic factors
- Learn from mistakes
- Improve, don't punish
2. Debugging as Learning
Continuous Improvement:
- Track debugging patterns
- Learn from past bugs
- Improve code to prevent recurrence
- Share knowledge with team
- Build debugging playbooks
Metrics:
- Mean time to detect (MTTD)
- Mean time to resolve (MTTR)
- Bug recurrence rate
- Code quality trends
Best Practices Summary
Do:
- Debug systematically, not randomly
- Use logs and metrics effectively
- Document findings and fixes
- Learn from bugs and prevent recurrence
- Share debugging knowledge
- Use tools and automation
- Write reproducible tests
- Monitor system health
Don't:
- Debug without understanding the problem
- Ignore logs and error messages
- Make assumptions without testing
- Fix symptoms without addressing root cause
- Work in isolation without documentation
- Rely on print statements alone
- Skip testing fixes
- Ignore monitoring and alerting
Examples and Use Cases
Example 1: Debugging a Database Timeout
Problem:
Database connection timeout after 30 seconds
Tier 1 Approach:
- Check error logs
- Test database connectivity
- Check query performance
- Identify slow query
- Add index to optimize query
Tier 2 Approach:
- Enable query logging
- Analyze slow query log
- Use EXPLAIN to understand query plan
- Add appropriate indexes
- Optimize query structure
- Add monitoring for slow queries
Tier 3 Approach:
- Set up database monitoring
- Configure alerts for slow queries
- Add query performance dashboard
- Implement automated query analysis
- Build knowledge base of query patterns
- Create playbooks for common database issues
Example 2: Debugging a Memory Leak
Problem:
Application runs out of memory after 24 hours
Tier 1 Approach:
- Monitor memory usage
- Take memory snapshots
- Identify growing objects
- Find code that creates objects
- Fix by removing reference or limiting cache
Tier 2 Approach:
- Use memory profiler
- Analyze object allocation patterns
- Identify cache growing unbounded
- Implement cache eviction policy
- Add memory usage monitoring
- Add automated memory tests
Tier 3 Approach:
- Set up continuous memory profiling
- Configure memory usage alerts
- Implement automatic cache sizing
- Build memory usage dashboard
- Create automated leak detection
- Document memory patterns in knowledge base
Example 3: Debugging a Race Condition
Problem:
Intermittent data corruption under load
Tier 1 Approach:
- Reproduce with high load
- Add logging to track access
- Identify concurrent access to shared resource
- Add proper locking
- Test with thread sanitizer
Tier 2 Approach:
- Use thread sanitizer to detect race conditions
- Analyze all shared resource access
- Implement proper synchronization
- Test under various loads
- Add stress testing to CI/CD
- Document concurrency patterns
Tier 3 Approach:
- Set up continuous integration with race detection
- Build concurrency testing framework
- Implement automated stress testing
- Create playbooks for concurrency issues
- Train team on best practices
- Build knowledge base of concurrency patterns
Common Mistakes
Mistake 1: Debugging Without Understanding the Problem
Problem: Jumping into code without understanding the issue
Fix: Start with problem definition and gather information first
Mistake 2: Fixing Symptoms Instead of Root Cause
Problem: Adding error handling instead of fixing underlying bug
Fix: Use root cause analysis (5 Whys, fishbone diagram)
Mistake 3: Not Reproducing the Issue
Problem: Making fixes without confirming they work
Fix: Always create a reproducible test case
Mistake 4: Making Too Many Changes at Once
Problem: Changing multiple things, don't know which fixed it
Fix: Make minimal changes, test each change
Mistake 5: Not Testing Edge Cases
Problem: Fix works for main case but fails edge cases
Fix: Test with various inputs and conditions
Mistake 6: Not Documenting Findings
Problem: Forgetting what was learned, repeating same mistakes
Fix: Document bugs, fixes, and lessons learned
Mistake 7: Relying Only on Print Statements
Problem: Print statements are inefficient and hard to manage
Fix: Use proper logging and debugging tools
Mistake 8: Not Monitoring After Fixes
Problem: Issue recurs later, no way to detect
Fix: Add monitoring and alerting for recurrence
Troubleshooting
Q: I can't reproduce the issue. What should I do?
A: Collect as much information as possible from the original occurrence (logs, screenshots, environment details). Try to understand the conditions that led to the issue. Create tests based on your understanding, even if they don't fail.
Q: The bug appears randomly and intermittently. How do I debug it?
A: Intermittent bugs are often timing-related (race conditions) or data-dependent. Focus on concurrency, timing, and data variations. Use stress testing to increase frequency of occurrence. Add detailed logging to capture state when the bug occurs.
Q: The code is complex and hard to understand. Where do I start?
A: Start with understanding the inputs and outputs. Trace the execution path. Simplify by removing unrelated code. Use a debugger to step through code. Ask for help from someone familiar with the code.
Q: How do I know when I've found the root cause?
A: The root cause is the deepest level cause that, if addressed, prevents the issue from recurring. Use the "5 Whys" technique or fishbone diagram to ensure you've reached the root cause, not just a symptom.
Q: Should I add defensive coding or fix the actual bug?
A: Always fix the actual bug first. Defensive coding is appropriate as a safety net, but not as a substitute for fixing the root cause.
Related Skills
- code-review: Debugging insights improve code reviews
- code-refactoring: Debugging reveals refactoring opportunities
- quick-research: Research error messages and patterns
- knowledge-graph: Link bugs to related issues and fixes
- workflow-orchestrator: Automate debugging workflows