Use when a Node.js process is growing memory unbounded in production, hits OOM, restarts on schedule, or shows monotonic RSS in Grafana. Triggers: heap snapshot diff (Comparison mode in Chrome DevTools), v8.writeHeapSnapshot, --heapsnapshot-signal=SIGUSR2, clinic doctor / clinic heap, Allocation sampling vs Allocation instrumentation timeline, --max-old-space-size, retainers / dominators, EventEmitter listener leak, closures over big objects, Buffer.allocUnsafe, intern caches that never evict. NOT for CPU profiling (use clinic doctor for the smoke signal then a CPU-specific skill), Go pprof, browser-side memory leaks, or worker_threads-only debugging.
Instrucciones de origen · Vista previa de solo lectura
name
node-memory-leak-hunting
description
Use when a Node.js process is growing memory unbounded in production, hits OOM, restarts on schedule, or shows monotonic RSS in Grafana. Triggers: heap snapshot diff (Comparison mode in Chrome DevTools), v8.writeHeapSnapshot, --heapsnapshot-signal=SIGUSR2, clinic doctor / clinic heap, Allocation sampling vs Allocation instrumentation timeline, --max-old-space-size, retainers / dominators, EventEmitter listener leak, closures over big objects, Buffer.allocUnsafe, intern caches that never evict. NOT for CPU profiling (use clinic doctor for the smoke signal then a CPU-specific skill), Go pprof, browser-side memory leaks, or worker_threads-only debugging.
license
Apache-2.0
allowed-tools
Read,Grep,Glob,Edit,Write,Bash
metadata
{"category":"Backend & Infrastructure","tags":["nodejs","memory","heap","performance","debugging","v8"],"pairs-with":[{"skill":"kubernetes-debugging-runbook","reason":"OOMKilled pods (exit 137) are triaged there first; this skill takes the handoff for the heap-snapshot-diff root cause inside the Node process"},{"skill":"observability-apm-expert","reason":"The heap_used/heap_total/rss/external metrics, panels, and alerts this skill's Quality Gates require are that skill's monitoring surface"},{"skill":"daemon-development","reason":"Long-lived daemons are where unbounded caches and per-request listeners accumulate; that skill owns the process-lifecycle side of keeping them healthy"}],"provenance":{"kind":"first-party","owners":["port-daddy"]},"io-contract":{"kind":"deliverable","consumes":["[Truncated]","[Truncated]"],"produces":["[Truncated]","[Truncated]"]}}
Node.js Memory Leak Hunting
A leak in Node.js is "RSS goes up, never comes down." V8 garbage-collects what it can prove is unreachable; a leak means something is still reachable that you didn't expect. The single most powerful tool, recommended by the Node.js diagnostics docs and the practitioner field, is the two-snapshot diff in Chrome DevTools' Memory tab. (Node.js — Using Heap Snapshot)
The compressed playbook:
1. Confirm the leak (Grafana shows monotonic RSS over hours).
2. Snapshot before load.
3. Drive load that exercises the suspect path.
4. Snapshot after.
5. Open both in DevTools → Memory tab → Comparison mode.
6. Sort by "Delta" desc. The retained, growing constructor at the top is your leak.
Production process RSS grows monotonically; restarts on schedule are masking it.
OOM kills (signal SIGKILL, exit 137) on Kubernetes pods.
p99 latency climbs as the process ages.
Memory CI gate failing on a synthetic load run.
Core capabilities
Confirm before chasing
Heap pressure can look like a leak without being one. Before doing snapshot work, rule out the usual:
Smell
Likely cause
Real fix
RSS climbs to a ceiling, then plateaus
V8 reaching --max-old-space-size. Not a leak.
Raise the limit; revisit if growth resumes
RSS climbs after each request, never falls
Genuine leak — proceed.
Heap snapshot diff
RSS climbs for an hour, then drops sharply
Deferred GC of a large old generation. Not a leak.
Watch over a longer window
RSS jumps on a recurring schedule (cron, batch job)
A periodic job allocating a lot transiently
Look at the job, not the long-lived state
RSS flat, but heap_used climbing
True leak; OS hasn't returned pages yet
Heap snapshot diff
Wire process.memoryUsage() to your metrics exporter (heap_used, heap_total, rss, external, arrayBuffers). See grafana-dashboard-builder for the panel and alert.
Trigger snapshots safely
In Node 12+, signal-based snapshots avoid the need for an admin endpoint:
node --heapsnapshot-signal=SIGUSR2 index.js
# Then from another shell:kill -USR2 <pid>
# Snapshot written to ./Heap-<pid>-<timestamp>.heapsnapshot
Snapshots are big. Multi-GB heaps produce multi-GB files. Practitioner pattern: write to a mounted S3 (or equivalent) volume rather than local disk. (Halodoc — Find & Fix Node.js Memory Leaks) Snapshotting also pauses the event loop while V8 walks the heap; on a 4 GB heap expect a few seconds of pause. Consider taking the snapshot from a replica or canary, not the busy primary.
The two-snapshot diff
The technique the Node.js docs spell out, paraphrased: (node-heap-snapshot)
1. Let the process bootstrap completely.
2. Snapshot 1.
3. Drive the suspect path (curl loop, k6 script, real traffic).
4. Snapshot 2.
5. In Chrome DevTools → Memory tab, load both .heapsnapshot files.
6. Select snapshot 2; in the dropdown, choose "Comparison".
7. Sort by "Delta" descending. The constructor with a large positive #New
AND a large positive #Delta — and zero #Deleted — is your leak candidate.
The Node.js docs frame it: "Open the older snapshot first, then load the newer one. Switch the view from Summary to Comparison. Look for large positive deltas." (node-heap-snapshot)
A useful three-snapshot variant for slow leaks: snapshot, run load, snapshot, run more load, snapshot. Compare 1↔2 and 2↔3. The constructor that grows in both diffs is the leak. (One-shot diffs sometimes catch transient allocations that look like leaks but aren't retained after a few more GCs.)
Retainers and dominators
Once you've found the leaking constructor, click into instances and read Retainers (bottom panel). Retainers are the chain of references keeping the object alive. Read up the chain:
MyHandler @ 0x1234
└─ in [callbacks] of EventEmitter @ 0x5678
└─ in [errorHandlers] of Server @ 0x9abc
└─ in (closure) of express()
└─ in (Global) — the root
You're looking for the unexpected reference — the moment the chain becomes "oh, that's still holding it." Common shapes are in the next section.
The Dominators view answers a different question: "if this object went away, how much would I free?" Useful when one Map or array is hoarding a long tail of small objects.
Sampling heap profiler
When the production process can't tolerate the pause of a full snapshot, the Allocation Sampling profiler is safe to leave on: (Node.js — Using Heap Profiler)
node --inspect index.js
In Chrome DevTools → Memory → Allocation sampling → Start. Statistically samples allocations by stack trace; low overhead, production-safe. (node-heap-profiler) Doesn't tell you what's retained (snapshots do that) but tells you where the allocation pressure is coming from.
For a programmatic, file-output equivalent, the node --heap-prof flag (V8 sampling heap profiler) writes a .heapprofile you can open in DevTools too:
(--heap-prof-interval is the average bytes between samples; default 512 KB.)
Common Node.js leak shapes
These are the constructor names you'll typically see on top of the diff:
EventEmitter listener leak
// Every request adds one. Never removed.functionhandler(req, res) {
emitter.on('event', () => res.send('ok')); // accumulates per-request closures
}
Symptom in heap diff: explosive growth of [ListenerNode] / Function retained by an EventEmitter's _events map. Often paired with the runtime warning "MaxListenersExceededWarning".
Fix:emitter.once, or removeListener after handling. Better: don't subscribe per-request to a long-lived emitter.
Symptom in heap diff: the leaking constructor IS your Map / Object, with Distance increasing by the count of unique keys.
Fix: an LRU with a hard cap (lru-cache), WeakMap if keys are objects you want GC'd when callers drop them.
Closure over a big object
functionload() {
const big = readFileSync('big.json'); // 200 MBconst small = JSON.parse(big).meta; // tinyreturn() => small; // closure retains `big` AND `small`!
}
Symptom:(closure) in retainer chain pointing at a constructor that is far larger than what the function "needs."
Fix: narrow the closure: function load() { const big = readFileSync('big.json'); const small = JSON.parse(big).meta; return makeFn(small); } where makeFn is at module scope.
Buffer / ArrayBuffer growth
process.memoryUsage().external is climbing while heap_used is flat. V8 manages the heap; native buffers are tracked separately. Common cause: cached Buffers or ArrayBuffers held by user code.
Fix: find them via Buffer retainer, free explicitly when done. Check Buffer.allocUnsafe callers — fast, but easy to leak.
Global state added per-request
// req.app.locals.metrics is global to the app.
app.use((req, res, next) => {
app.locals.metrics.lastByPath ??= {};
app.locals.metrics.lastByPath[req.path] = Date.now(); // grows per unique pathnext();
});
Fix: bound the size, or stop using app.locals as a database.
Promise chain holding context
A long-running Promise that captures large request state and never resolves is a slow leak that's hard to spot. Set timeouts on every external call.
Crank the limit, but not before fixing
node --max-old-space-size=4096 index.js # 4 GB old space (V8's tenured generation)
Useful when you legitimately need more headroom (large in-memory cache, big JSON parsing). Counter-productive when used to mask a leak — you just push the OOM out by a few hours. Always confirm with two-snapshot diff that growth is bounded before raising the ceiling.
Tooling shortcuts
clinic doctor index.js — runs your app under load, gives a "is it CPU? memory? event loop? I/O?" verdict before you commit to a deeper tool. (dev.to — Debugging Memory Leaks)
clinic heap — focused heap profile.
node --inspect-brk=0.0.0.0:9229 index.js + Chrome DevTools — full interactive debugger.
--report-on-fatalerror — V8 writes a JSON diagnostic report on OOM crash.
Anti-patterns
Restarting on a cron to "fix" the leak
Symptom: PM2/Docker scheduled restart at 03:00 daily; nobody knows why.
Diagnosis: A leak was masked years ago; nobody ever debugged it.
Fix: Take a snapshot diff during the day. Find the actual leak. Restarts hide alerting and cause cold caches.
Reading heap_used without heap_total and rss
Symptom: Dashboard shows flat heap_used, alerts never fire, but pods OOM.
Diagnosis:heap_used is what V8 has allocated within the heap. rss includes external buffers and code. heap_total is the cap V8 grew to.
Fix: Track all four (heap_used, heap_total, rss, external) plus arrayBuffers (Node 13+).
Snapshot on the busy primary
Symptom: Snapshot triggers; primary pauses for 8s; uphouse 503s.
Diagnosis:writeHeapSnapshot() is synchronous and walks the entire heap.
Fix: Snapshot from a canary, or drain traffic from one pod, snapshot it, return it to rotation.
Comparing snapshots without driving load between them
Symptom: Diff shows mostly noise (tiny deltas in (string), (closure)).
Diagnosis: Without exercising the suspect path between snapshots, the diff is just GC variance.
Fix: Drive the suspect path with k6/curl/real traffic for several minutes between snapshots.
Trusting the heap dump's source positions blindly
Symptom: Tracing the leak to a line in node_modules/some-lib/dist/index.cjs:1.
Diagnosis: Bundled / minified code; the position is meaningless.
Fix:--enable-source-maps to get the original source positions; or read the un-bundled package source.
--max-old-space-size set on every Node process automatically
Symptom: Every container has 8 GB allocated, most use 200 MB.
Diagnosis: Default applied broadly to silence one rare OOM.
Fix: Set per-service based on measured working set + headroom (typically 1.5–2× p99). Different services need different limits.
Quality gates
Test: synthetic load test (k6, autocannon) drives the suspect path for ≥ 30 minutes; RSS curve flat at the end (or grows < 1% per hour).
Test: memory-leak regression test asserts process.memoryUsage().heapUsed doesn't grow > N MB after running M iterations of the load.
OOM-on-crash report enabled: node --report-on-fatalerror index.js; reports archived from CI / production.
Heap snapshots can be triggered via --heapsnapshot-signal=SIGUSR2 AND via authenticated admin endpoint (v8.writeHeapSnapshot).
Source maps enabled (--enable-source-maps) so retainer chains point at real source.
No scheduled restart hiding a leak (cron/PM2/Kubernetes liveness probe). If one exists, leak is investigated and either confirmed-fixed or documented.
--max-old-space-size set deliberately per service, not as a global default.
lru-cache (or equivalent) is used for any in-process cache; raw Map/Object caches reviewed for unbounded growth.
Long-running event subscribers (emitter.on) audited; per-request subscription patterns replaced with once or removed.
Buffer / ArrayBuffer hot paths reviewed; external memory tracked.
OTel span around requests with node.heap_used_after_ms recorded for drift detection (see opentelemetry-instrumentation).
Deterministic Audit
Before starting a hunt (or reviewing another agent's), write it as a JSON plan matching
schemas/leak-hunt-plan.schema.json and run the deterministic auditor:
auditNodeMemoryLeakHunting(plan) (in scripts/node_memory_leak_hunting_audit.mjs)
encodes this skill's playbook as machine-checkable rules over structured fields only:
a scheduled restart planned as the "fix", raising --max-old-space-size against monotonic
growth, a plateau misread as a confirmed leak, a two-snapshot diff with no load driven
between the snapshots, snapshotting the busy primary, incomplete
process.memoryUsage() metric coverage, and missing source maps. It returns
{ pass, score, findings, recommendations } and exits 1 on failure.
examples/sample-input.json is a correctly-shaped monotonic-growth hunt (canary snapshot,
load-driven diff, all five metrics) that audits pass: true. Version history lives in CHANGELOG.md.
NOT for
CPU profiling — different tool. → clinic doctor for the smoke signal; then a CPU-specific tool / skill.
Go memory profiling — different runtime. → go-pprof-profiling.
Browser memory leaks — heap snapshots in Chrome work but the constructors and retainers are different. No dedicated skill yet.
Python memory leaks — different runtime. No dedicated skill yet.
worker_threads-specific debugging — overlapping but distinct.
