diagnose-memory-leak

name	diagnose-memory-leak
description	Use when a process shows continuously growing memory usage that doesn't stabilize, when OOM kills are occurring in production, or when heap dumps are needed to identify object accumulation
source	Brendan Gregg "Systems Performance" (2020) memory chapter; Java/JVM GC analysis (Oracle docs); Gregg "BPF Performance Tools" (2019)
tags	["performance","debugging","memory","reliability"]
verified	true

Diagnose Memory Leak

Systematically identify the source of unbounded memory growth in a process by profiling allocations, analyzing heap contents, and tracing the code path responsible.

Why This Is Best Practice

Adopted by: Netflix (Java heap analysis for streaming services), Google (memory profiling as standard performance practice), all production engineering teams that run long-lived processes Impact: Memory leaks are the #3 cause of production incidents (after hardware and deployment failures) in long-running services; undiagnosed leaks cause 2-4 AM OOM kills that take 20+ minutes to recover; structured diagnosis reduces MTTI from hours to 30 minutes Why best: Memory leaks have many causes (event listener accumulation, cache without eviction, retained closures, circular references, native memory leaks); only systematic profiling pinpoints the specific cause

Sources: Gregg "Systems Performance" 2nd ed. (2020) Ch. 7; Gregg "BPF Performance Tools" Pearson (2019); Oracle JVM GC documentation; Node.js memory profiling guide

Steps

1. Confirm the leak exists — Monitor process memory over time (at least 24-48 hours): ps -o pid,rss,vsz,comm -p <pid> every minute, or use Prometheus process_resident_memory_bytes. A genuine leak shows monotonically increasing RSS that doesn't return to baseline after load decreases. Natural heap growth that stabilizes is not a leak.

2. Determine the memory region — Distinguish heap vs. non-heap memory growth. JVM: heap (-Xmx bounded) vs. off-heap (Metaspace, direct buffers, native libraries). Node.js: V8 heap vs. native addons. A growing heap is a managed code leak; growing non-heap is a native memory or off-heap allocation leak. Diagnosis tools differ by region.

3. Enable detailed GC logging (JVM) — Add JVM flags: -Xlog:gc*:file=/tmp/gc.log:time,uptime:filecount=10,filesize=10m. Analyze with GCEasy.io or IBM GC analyzer. A rising old-gen baseline after full GC indicates objects surviving collection that should be collected — classic leak signature. GC logs reveal memory pressure before OOM.

4. Capture heap snapshots at intervals — JVM: jmap -dump:format=b,file=/tmp/heap1.hprof <pid> at two points 30 minutes apart. Node.js: v8.writeHeapSnapshot() or --inspect + Chrome DevTools heap snapshot. Compare the two snapshots to identify which object types grew. Objects that accumulate between snapshots without being released are leak candidates.

5. Analyze heap with a profiler — JVM: open heap dump in Eclipse MAT (Memory Analyzer Tool). Use "Leak Suspects" report — MAT identifies objects with high retained heap. Look for: collections (HashMap, ArrayList) growing without bound, listener registries, thread-local variables. Node.js: Chrome DevTools "Comparison" between two heap snapshots shows which object types increased.

6. Identify the retaining reference chain — In Eclipse MAT: right-click the leaking object → "Path to GC Roots" → "Exclude soft/weak/phantom references". This shows the reference chain keeping the object alive. The root of the chain (a static field, a thread-local, a framework registry) is the leak source. In Node.js DevTools: "Retainers" panel shows the same chain.

7. Profile allocations in real time — JVM: use async-profiler (./profiler.sh -e alloc -d 60 -f alloc.svg <pid>) to generate an allocation flame graph. Shows which call stacks allocate the most objects — high-frequency allocation of objects that should be short-lived but are retained points to the leak site. Gregg's BPF tools: memleak from bcc-tools traces native memory allocations.

Rules

• Heap dumps from production processes containing user data must be handled as sensitive data (PII may be in memory); apply the same access controls as production databases.
• Never tune heap size (-Xmx) as the first response to a memory leak; you are delaying the OOM, not fixing the leak.
• A fix that reduces growth rate but doesn't eliminate it is not a fix; verify heap is flat (zero net growth) after warmup.
• Always reproduce the leak in a controlled test before claiming a fix; "it seems better" after a production deploy is confirmation bias, not validation.

Common Mistakes

• Increasing heap size as the fix — this delays the OOM crash by hours or days; the leak recurs and the system still crashes; find and fix the root cause.
• Analyzing heap during high load — heap snapshots during load show normal temporary allocations mixed with leaked objects; capture snapshots after load decreases to isolate what should have been collected.
• Ignoring non-heap memory — a Java process where heap is stable but RSS is growing has an off-heap leak (DirectByteBuffer, JNI, Metaspace); heap analysis tools miss this.
• Single heap snapshot — a single snapshot shows the heap state but cannot distinguish what grew; always compare two snapshots taken at different points in time to isolate growth.

When NOT to Use

• Memory growth that stabilizes after warmup — this is natural heap sizing, not a leak; JVMs fill heap to improve GC efficiency
• Memory growth proportional to data volume — storing user data in memory that grows with users is architectural, not a leak; evaluate caching strategy instead
• OOM during initialization before the service is serving traffic — this is a configuration issue (insufficient heap for the workload), not a leak