memory-leak-investigation

name	memory-leak-investigation
enabled	true
description	Use when performing memory leak investigation — provides a systematic methodology for detecting, diagnosing, and resolving memory leaks in production applications. Covers heap analysis, allocation profiling, leak pattern identification, garbage collection tuning, and validation procedures across common runtime environments.
required_connections	[{"prefix":"apm","label":"APM / Profiling Platform"},{"prefix":"monitoring","label":"Monitoring Platform"}]
config_fields	[{"key":"runtime","label":"Application Runtime","required":true,"placeholder":"e.g., JVM (Java 17), Node.js 20, Python 3.12, Go 1.22, .NET 8"},{"key":"symptom","label":"Primary Symptom","required":true,"placeholder":"e.g., OOM kills, growing RSS, increasing GC pauses"},{"key":"time_to_oom","label":"Approximate Time to OOM","required":false,"placeholder":"e.g., 6 hours, 3 days"}]
features	["PERFORMANCE","DEBUGGING","MEMORY"]

Memory Leak Investigation

Phase 1: Symptom Confirmation

1. Verify memory leak exists
   • Memory usage grows continuously over time (not plateauing)
   • Memory not reclaimed after GC cycles
   • OOM kills or restarts observed
   • GC pause times increasing over time
   • Application performance degrades over uptime
2. Establish the leak rate (MB/hour or MB/day)
3. Determine if leak correlates with specific operations
4. Check when the leak started (deployment, config change, traffic change)

Memory Growth Profile

Metric	At Start	After 1hr	After 6hr	After 24hr	Trend
Heap used (MB)					+MB/hr
RSS (MB)
GC frequency
GC pause (ms)
Live objects

Phase 2: Heap Snapshot Analysis

1. Capture heap snapshots at intervals
   • Snapshot 1: Shortly after application start
   • Snapshot 2: After several hours of operation
   • Snapshot 3: When memory is significantly elevated
2. Compare snapshots to identify growing object types
3. Find objects that should be garbage collected but are retained
4. Identify the retention path from GC root to leaked objects

Top Growing Object Types

Object Type	Count (T1)	Count (T2)	Size (T1)	Size (T2)	Growth Rate	Suspect
			MB	MB	MB/hr	[ ]

Phase 3: Allocation Profiling

1. Enable allocation tracking in profiler
2. Identify hot allocation sites (where leaked objects are created)
3. Trace allocation call stacks for suspected object types
4. Correlate allocations with specific request types or code paths
5. Check for known leak patterns

Common Leak Patterns Checklist

Pattern	Description	Detected
Event listener accumulation	Listeners added but never removed	[ ]
Cache without eviction	Unbounded cache or map growth	[ ]
Connection/stream not closed	DB connections, file handles, HTTP connections	[ ]
Circular references	Objects preventing GC (weak references needed)	[ ]
Static/global collections	Collections growing without bounds	[ ]
Timer/interval not cleared	Scheduled tasks holding references	[ ]
Thread-local accumulation	Thread-local storage not cleaned	[ ]
Closure capturing	Closures retaining large scope references	[ ]
Native memory leak	JNI, native buffers, off-heap allocations	[ ]

Phase 4: Root Cause Isolation

1. Reproduce the leak in a controlled environment
2. Narrow down to specific code module or dependency
3. Write a minimal reproduction if possible
4. Verify the root cause with targeted profiling
5. Document the leak mechanism

Root Cause Analysis

Finding	Code Location	Leak Mechanism	Impact	Fix Approach
	file:line		MB/hr

Phase 5: Fix Implementation

1. Implement the fix
   • Add proper resource cleanup (close, dispose, remove listener)
   • Add eviction policy to caches (LRU, TTL, size limit)
   • Fix object lifecycle management
   • Use weak references where appropriate
   • Add connection pool limits and timeout
2. Add defensive measures
   • Add memory usage logging
   • Implement resource tracking in development mode
   • Add unit tests for resource cleanup

Phase 6: Validation

1. Deploy fix and monitor memory profile
2. Verify memory stabilizes at expected level
3. Run extended soak test (48+ hours)
4. Compare memory profile against pre-leak baseline
5. Set up ongoing memory monitoring and alerts

Validation Results

Metric	Before Fix	After Fix	Target	Status
Heap growth rate	MB/hr	MB/hr	~0 MB/hr	Pass/Fail
Stable heap size	N/A (growing)	MB	< MB
GC pause P95	ms	ms	< ms
OOM incidents/week		0	0

Counter-Rationalizations

Shortcut	Counter	Why
"We can skip some steps for this case"	Adapt the workflow steps, don't skip them	Skipped steps are where incidents and oversights originate
"The user seems to already know what to do"	Complete all workflow phases with the user	The workflow catches blind spots that experience alone misses
"This is a minor case, full process is overkill"	Scale the process down, don't turn it off	Minor cases become major when unstructured; the process scales, not disappears
"I'll fill in the details later"	Complete each section before moving on	Deferred details are forgotten; real-time capture is more accurate
"The template output isn't necessary"	Always produce the structured output format	Structured output enables comparison, audit trails, and handoff to other teams

Output Format

• Leak Confirmation: Growth rate and correlation analysis
• Heap Analysis: Object type growth and retention paths
• Root Cause Report: Code location and leak mechanism
• Fix Description: Changes made with rationale
• Validation Report: Post-fix memory stability proof

Action Items

• Confirm memory leak with growth rate measurement
• Capture and compare heap snapshots
• Identify leaking object types and retention paths
• Isolate root cause to specific code
• Implement and test fix
• Validate with extended soak test
• Set up memory growth alerting to detect future leaks