go-perf-production-incidents

name	go-perf-production-incidents
description	Guides diagnosing a LIVE Go performance incident under on-call pressure — OOMKill / RSS climb, GC death-spiral / CPU pegged in GC, p99 latency cliff, goroutine leak / pile-up, CPU saturation — with the capture-evidence-before-restart discipline, safe prod pprof / flight-recorder / heap-profile runbooks, and GOMEMLIMIT triage. Auto-invokes on "my service is OOMing", "production latency spiked", "pod keeps getting OOMKilled", "diagnose this incident", "goroutines climbing in prod", "CPU pegged in GC". Routes GC knobs to go-perf-gc-tuning, tail latency to go-perf-tail-latency, the tracer/flight recorder to go-perf-execution-tracer.
allowed-tools	Read, Glob, Grep
compatibility	Claude Code, Codex CLI, Gemini CLI

Go Performance Production Incidents

"Your first response in a major outage may be to start troubleshooting and try to find a root cause as quickly as possible. Ignore that instinct!" — Google SRE, Effective Troubleshooting

This is the human / on-call layer of Go performance: a service is on fire right now, you are tired, it is 3am, and the wrong move makes the diagnosis impossible. The other go-perf-* skills own the theory (which knob, which profiler, why one run lies). This one owns the lived incident — the five failures you actually get paged for and the order to handle them in. The single rule that survives the panic: capture evidence before you restart. A restart "fixes" the symptom and destroys the heap profile, the goroutine dump, and the trace that tell you why — so the incident recurs next week with no new data.

All Go below builds clean under gofmt, go vet, and go test on Go 1.25/1.26.

Triage index — match the symptom, jump to the runbook:

What you're seeing	First evidence to capture	Section
Pod OOMKilled (exit 137), RSS climbing	heap profile (inuse_space) + goroutine profile	§3
CPU 100%, throughput dead, gctrace GC% high & rising	GODEBUG=gctrace=1 line	§4
p99/p999 cliff, mean fine	flight-recorder snapshot at the spike	§5
Goroutine count climbing without bound	goroutine profile (debug=1)	§6
CPU pegged, gctrace GC% low	30s prod CPU profile	§7

---

1. The calm order — stabilize, capture, diagnose, fix, gate

Under pressure the instinct is to jump straight to root cause. SRE doctrine is the opposite: "Stopping the bleeding should be your first priority; you aren't helping your users if the system dies while you're root-causing" (Effective Troubleshooting). The fixed order:

1. Stabilize — roll back the bad deploy, scale out, shed load, or drain the sick pod. Restore service first: "Stop the bleeding, restore service, and preserve the evidence for root-causing" (Managing Incidents).
2. Capture evidence — before the restart. Pull the heap profile, the goroutine profile, the trace snapshot off the sick instance while it is still sick. This is the step everyone skips and the one that decides whether you ever find the bug. If you must kill the pod, grab profiles from it first, or keep one sick replica cordoned for forensics.
3. Diagnose from the captured artifacts, calmly, after service is restored — not in the middle of the page.
4. Fix the root cause (an allocation, a leak, a missing GOMEMLIMIT, an unbounded goroutine spawn).
5. Add a regression gate so it can't silently return — an alloc/goroutine budget or a benchmark in CI (→ go-perf-methodology §9).

Resist the classic traps while diagnosing: "latching on to causes of past problems, reasoning that since it happened once, it must be happening again," and remember "correlation is not causation" (Effective Troubleshooting). Capture first, theorize later.

---

2. Safe production profiling — the discipline that keeps profiling from becoming the incident

net/http/pprof registers handlers under /debug/pprof/; you "typically only import [it] for the side effect of registering its HTTP handlers" (net/http/pprof). Three rules make it safe to use on a live, paying system:

• Never on the public port. Mount pprof on a private admin listener bound to localhost or the internal network, never the same mux that serves user traffic — /debug/pprof/ is an unauthenticated heap/goroutine/CPU dump of your process and a denial-of-service handle. (Experiential judgment; the safe-exposure mechanics live in go-perf-pprof-profiling.)

  // Separate admin mux on a private port — NOT http.DefaultServeMux on :443.
  adminMux := http.NewServeMux()
  adminMux.HandleFunc("/debug/pprof/", pprof.Index)
  adminMux.HandleFunc("/debug/pprof/profile", pprof.Profile)
  go func() { log.Fatal(http.ListenAndServe("127.0.0.1:6060", adminMux)) }()
  

• One profile at a time. A CPU profile perturbs the very latency you're measuring, and "precise memory profiling skews CPU profiles" (Diagnostics). Take the CPU profile, then the heap profile — never both at once, or both lie.

• Know the cost. A CPU profile (/debug/pprof/profile?seconds=30) costs ~30s of sampling overhead; a heap or goroutine profile is a cheap point-in-time snapshot you can take immediately. On a process with hundreds of thousands of goroutines, the goroutine dump itself briefly stops the world — expect a small hiccup, and prefer debug=1 over the giant debug=2 stack dump.

Fetch directly with the tool you already have (net/http/pprof):

go tool pprof http://127.0.0.1:6060/debug/pprof/heap            # inuse_space (live) by default
go tool pprof http://127.0.0.1:6060/debug/pprof/goroutine       # leak hunt
go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=30   # CPU
curl -o sick.heap http://127.0.0.1:6060/debug/pprof/heap        # save the artifact off the dying pod

---

3. OOMKill / memory blow-up — the pod keeps getting killed

What it looks like: the container is OOMKilled (exit 137), RSS climbing on the dashboard. The dashboard shape is the first diagnostic: a sawtooth RSS (climbs, GC reclaims, climbs) that crests the limit is a transient-spike or limit-too-low problem; a monotonic climb that never comes back down is a leak — your code is holding references the GC can't free.

Runbook:

1. Capture the heap profile from the sick pod — inuse_space is the default and shows live objects: it "tracks ... the allocation sites for all live objects" and pprof defaults to "-inuse_space (live objects, scaled by size)" (runtime/pprof). The top frames are who is holding the memory.
2. Check for a goroutine leak in the same breath (§6) — leaked goroutines hold their entire stack and captured variables live, and present as a memory leak. A monotonic climb is a goroutine leak until proven otherwise.
3. Confirm GOMEMLIMIT is set. With no soft limit, the runtime GCs on GOGC growth ratio alone and will happily let the heap sail past the container limit into an OOMKill instead of GCing harder as it approaches the ceiling. Setting GOMEMLIMIT to ~90% of the container limit makes the GC fight to stay under it — "leave an additional 5-10% of headroom to account for memory sources the Go runtime is unaware of" (GC Guide). Knob details → go-perf-gc-tuning.

The trap to refuse: do not "just bump the memory limit" on a monotonic climb. A leak grows without bound; a bigger limit only buys a longer fuse before the same OOMKill, and you've thrown away the heap profile that names the leak. Bump the limit only to confirm the shape is a sawtooth (real working set) and not a leak.

---

4. GC death-spiral — CPU pegged in GC

What it looks like: CPU at 100% but throughput collapsed, p99 exploded, and GODEBUG=gctrace=1 shows GCs firing back-to-back with the GC-time percentage high and rising:

gc 412 @30.1s 58%: 0.10+820+0.05 ms clock, ... 980->998->995 MB, 1000 MB goal, 8 P
gc 413 @30.9s 59%: 0.09+810+0.06 ms clock, ... 996->999->996 MB, 1000 MB goal, 8 P

The 58%/59% is the fraction of total time spent in GC since start (→ go-perf-gc-tuning §6 reads this line), and a live heap (996 MB) pinned against the goal (1000 MB) is the signature. This is thrashing: "the program fails to make reasonable progress due to constant GC cycles ... It's particularly dangerous because it effectively stalls the program" (GC Guide). The usual cause is a GOMEMLIMIT set too tight (or GOGC too low) for the live working set — the GC is being asked to maintain "an impossible memory limit."

The decision under pressure: relax the limit. A too-tight GOMEMLIMIT trades an OOM for an indefinite stall, and "in many cases, an indefinite stall is worse than an out-of-memory condition, which tends to result in a much faster failure" (GC Guide). Raise the limit (or GOGC) to stop the spiral now; the durable fix is cutting the allocation rate so the live heap sits well below the ceiling. Which knob, by how much → go-perf-gc-tuning.

---

5. p99 latency cliff — capture WITH the flight recorder, at the spike

What it looks like: mean latency fine, p99/p999 suddenly cliff-edges. A CPU profile is the wrong tool — it is "blind to off-CPU time," and a request slow because it's blocked on a lock, a channel, GC mark-assist, or the scheduler contributes nothing to it (Diagnostics). The right tool is the Go 1.25 flight recorder, which exists precisely because by the time you notice a rare spike "it's already too late to call Start" (Flight Recorder).

It keeps the last few seconds of execution trace in a ring buffer in memory and lets you snapshot the window after the spike has already happened — "the power of hindsight ... capture just what went wrong, after it's already happened" (Flight Recorder):

fr := trace.NewFlightRecorder(trace.FlightRecorderConfig{
    MinAge:   200 * time.Millisecond, // ~2x the event window you care about
    MaxBytes: 1 << 20,                // bound the memory cost
})
fr.Start()
// ... in the request path, snapshot only when this request was pathologically slow:
if fr.Enabled() && time.Since(start) > 100*time.Millisecond {
    go captureSnapshot(fr) // fr.WriteTo(f) — only the first call has effect
}

Then open the snapshot in go tool trace and read scheduler-latency and GC mark-assist at the spike (tracer mechanics → go-perf-execution-tracer; tail-latency method, coordinated omission, the alloc-rate lever → go-perf-tail-latency). The dominant tail lever is usually lowering the allocation rate, not a GC knob — Go STW pauses are sub-millisecond and rarely dominate the tail.

---

6. Goroutine leak / pile-up — the count climbs without bound

What it looks like: goroutine count rising monotonically on the dashboard; memory and scheduler pressure follow. The diagnostic is the goroutine profile — "stack traces of all current goroutines" (runtime/pprof) — and the tell is unmistakable: thousands of goroutines stuck on the same line, blocked on a channel send/receive or a mutex that will never be released.

# Top stacks, aggregated — the leak is the one with a count in the thousands and climbing.
go tool pprof -top http://127.0.0.1:6060/debug/pprof/goroutine
# Or the readable full dump for the exact blocked line (debug=1, not the huge debug=2):
curl 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1' -o goroutines.txt

A leak count that grows across two snapshots 30s apart confirms it. This is the same symptom that masquerades as an OOM (§3): each leaked goroutine pins its stack and captured variables. The fix is correctness — a missing context cancellation, an unbounded spawn, a send on a channel with no receiver. Goroutine-lifecycle and leak-prevention discipline → go-concurrency-goroutines; block/mutex profiling to find what they're stuck on → go-perf-block-mutex-profiles.

---

7. CPU saturation / runaway loop

What it looks like: CPU pegged but — unlike §4 — gctrace shows a low GC percentage, so it's mutator work, not the collector. This is the one case the CPU profile is exactly right for: it "determines where a program spends its time while actively consuming CPU cycles" (Diagnostics). Take a 30s production CPU profile off the admin port (§2) and read the hot frames:

go tool pprof -top http://127.0.0.1:6060/debug/pprof/profile?seconds=30

A single function dominating flat time is a hot path or a runaway/spin loop (a busy-wait, a regex backtracking, an O(n²) on grown input). Profile-reading depth — flat vs cumulative, list, sampling caveats → go-perf-pprof-profiling and go-perf-methodology §2. Before optimizing the hot function, confirm it's the bottleneck and not a symptom of saturation elsewhere (run-queue depth, lock contention) via the USE method → go-perf-methodology §3.

---

8. Don't

• Don't restart before capturing evidence. The restart resolves the symptom and erases the heap profile, goroutine dump, and trace — guaranteeing a recurrence with zero new data. Capture from the sick instance first (Managing Incidents).
• Don't "just bump the memory limit" on an OOM without checking the dashboard shape — a monotonic climb is a leak, and a bigger limit only delays the same OOMKill while you discard the heap profile that names it (§3).
• Don't run a service with no GOMEMLIMIT in a fixed-memory container — without it the runtime OOMs instead of GCing harder as it nears the ceiling; set it ~5–10% under the container limit (GC Guide).
• Don't set a GOMEMLIMIT you can't honor. A too-tight limit thrashes, and "an indefinite stall is worse than an out-of-memory condition" (GC Guide) (§4).
• Don't reach for a CPU profile to explain a latency cliff. It's blind to off-CPU waits (locks, channels, mark-assist); use the flight recorder + tracer (Diagnostics) (§5).
• Don't run multiple profiles at once on a sick box — "precise memory profiling skews CPU profiles" (Diagnostics); take them one at a time (§2).
• Don't expose /debug/pprof/ on the public port. It's an unauthenticated process dump and a DoS handle — private admin listener only (§2).
• Don't jump to a root-cause theory at 3am. Stabilize, capture, then diagnose calmly — "Ignore that instinct!" (Effective Troubleshooting) (§1).
• Don't close the incident without a regression gate — an alloc/goroutine budget or CI benchmark, or it returns silently (§1, → go-perf-methodology §9).

---

9. Routing to Related Skills

This plugin (go-perf-* depth):

• go-perf-gc-tuning — GOGC/GOMEMLIMIT knobs, the thrashing math, reading gctrace, allocation rate as the durable fix (§3, §4).
• go-perf-tail-latency — p99/p999 method, coordinated omission, the alloc-rate tail lever (§5).
• go-perf-execution-tracer — runtime/trace, go tool trace, flight-recorder mechanics (§5).
• go-perf-pprof-profiling — CPU/heap pprof subcommands, inuse vs alloc, safe production-profiling exposure depth (§2, §7).
• go-perf-block-mutex-profiles — block/mutex/goroutine profiles and the enablement gotcha (§6).
• go-perf-godebug-and-metrics — runtime/metrics catalog and GODEBUG for dashboards/alerts that page you in the first place.
• go-perf-methodology — the measure-first discipline, the tool-selection matrix, USE method, CI regression gating (§1, §7).

Parent / sibling marketplace (go-* correctness):

• go-concurrency-goroutines — goroutine lifecycle, cancellation, and leak prevention — the fix behind §6.
• go-context — cancellation/propagation, the usual missing piece in an unbounded-goroutine leak.

---

10. Reference Files

Incident anti-patterns in LLM-generated Go on-call work, each with wrong/right contrast and citations:

${CLAUDE_SKILL_DIR}/references/common-mistakes.md

Source provenance for every claim in this skill:

${CLAUDE_SKILL_DIR}/references/sources.yaml