| name | diagnose-memory-leak |
| description | Use when a process shows continuously growing memory usage that doesn't stabilize, when OOM kills are occurring in production, or when heap dumps are needed to identify object accumulation |
| source | Brendan Gregg "Systems Performance" (2020) memory chapter; Java/JVM GC analysis (Oracle docs); Gregg "BPF Performance Tools" (2019) |
| tags | ["performance","debugging","memory","reliability"] |
| verified | true |
Diagnose Memory Leak
Systematically identify the source of unbounded memory growth in a process by profiling allocations, analyzing heap contents, and tracing the code path responsible.
Why This Is Best Practice
Adopted by: Netflix (Java heap analysis for streaming services), Google (memory profiling as standard performance practice), all production engineering teams that run long-lived processes
Impact: Memory leaks are the #3 cause of production incidents (after hardware and deployment failures) in long-running services; undiagnosed leaks cause 2-4 AM OOM kills that take 20+ minutes to recover; structured diagnosis reduces MTTI from hours to 30 minutes
Why best: Memory leaks have many causes (event listener accumulation, cache without eviction, retained closures, circular references, native memory leaks); only systematic profiling pinpoints the specific cause
Sources: Gregg "Systems Performance" 2nd ed. (2020) Ch. 7; Gregg "BPF Performance Tools" Pearson (2019); Oracle JVM GC documentation; Node.js memory profiling guide
Steps
-
Confirm the leak exists — Monitor process memory over time (at least 24-48 hours): ps -o pid,rss,vsz,comm -p <pid> every minute, or use Prometheus process_resident_memory_bytes. A genuine leak shows monotonically increasing RSS that doesn't return to baseline after load decreases. Natural heap growth that stabilizes is not a leak.
-
Determine the memory region — Distinguish heap vs. non-heap memory growth. JVM: heap (-Xmx bounded) vs. off-heap (Metaspace, direct buffers, native libraries). Node.js: V8 heap vs. native addons. A growing heap is a managed code leak; growing non-heap is a native memory or off-heap allocation leak. Diagnosis tools differ by region.
-
Enable detailed GC logging (JVM) — Add JVM flags: -Xlog:gc*:file=/tmp/gc.log:time,uptime:filecount=10,filesize=10m. Analyze with GCEasy.io or IBM GC analyzer. A rising old-gen baseline after full GC indicates objects surviving collection that should be collected — classic leak signature. GC logs reveal memory pressure before OOM.
-
Capture heap snapshots at intervals — JVM: jmap -dump:format=b,file=/tmp/heap1.hprof <pid> at two points 30 minutes apart. Node.js: v8.writeHeapSnapshot() or --inspect + Chrome DevTools heap snapshot. Compare the two snapshots to identify which object types grew. Objects that accumulate between snapshots without being released are leak candidates.
-
Analyze heap with a profiler — JVM: open heap dump in Eclipse MAT (Memory Analyzer Tool). Use "Leak Suspects" report — MAT identifies objects with high retained heap. Look for: collections (HashMap, ArrayList) growing without bound, listener registries, thread-local variables. Node.js: Chrome DevTools "Comparison" between two heap snapshots shows which object types increased.
-
Identify the retaining reference chain — In Eclipse MAT: right-click the leaking object → "Path to GC Roots" → "Exclude soft/weak/phantom references". This shows the reference chain keeping the object alive. The root of the chain (a static field, a thread-local, a framework registry) is the leak source. In Node.js DevTools: "Retainers" panel shows the same chain.
-
Profile allocations in real time — JVM: use async-profiler (./profiler.sh -e alloc -d 60 -f alloc.svg <pid>) to generate an allocation flame graph. Shows which call stacks allocate the most objects — high-frequency allocation of objects that should be short-lived but are retained points to the leak site. Gregg's BPF tools: memleak from bcc-tools traces native memory allocations.
Rules
- Heap dumps from production processes containing user data must be handled as sensitive data (PII may be in memory); apply the same access controls as production databases.
- Never tune heap size (-Xmx) as the first response to a memory leak; you are delaying the OOM, not fixing the leak.
- A fix that reduces growth rate but doesn't eliminate it is not a fix; verify heap is flat (zero net growth) after warmup.
- Always reproduce the leak in a controlled test before claiming a fix; "it seems better" after a production deploy is confirmation bias, not validation.
Common Mistakes
- Increasing heap size as the fix — this delays the OOM crash by hours or days; the leak recurs and the system still crashes; find and fix the root cause.
- Analyzing heap during high load — heap snapshots during load show normal temporary allocations mixed with leaked objects; capture snapshots after load decreases to isolate what should have been collected.
- Ignoring non-heap memory — a Java process where heap is stable but RSS is growing has an off-heap leak (DirectByteBuffer, JNI, Metaspace); heap analysis tools miss this.
- Single heap snapshot — a single snapshot shows the heap state but cannot distinguish what grew; always compare two snapshots taken at different points in time to isolate growth.
When NOT to Use
- Memory growth that stabilizes after warmup — this is natural heap sizing, not a leak; JVMs fill heap to improve GC efficiency
- Memory growth proportional to data volume — storing user data in memory that grows with users is architectural, not a leak; evaluate caching strategy instead
- OOM during initialization before the service is serving traffic — this is a configuration issue (insufficient heap for the workload), not a leak
