go-perf-godebug-and-metrics

name	go-perf-godebug-and-metrics
description	Guides Go runtime observability — the GODEBUG trace knobs (gctrace, schedtrace/scheddetail, inittrace, allocfreetrace, madvdontneed) and reading a gctrace line, the low-overhead runtime/metrics package and its histogram metrics, preferred over runtime.ReadMemStats which stops the world, and exposing metrics via expvar/Prometheus. Fires on "GODEBUG", "gctrace", "read runtime metrics", "monitor GC in production", "ReadMemStats vs runtime/metrics". Routes GC to go-perf-gc-tuning, scheduler to go-perf-goroutines-scheduler.
allowed-tools	Read, Glob, Grep
compatibility	Claude Code, Codex CLI, Gemini CLI

Go GODEBUG & runtime/metrics

This skill owns runtime observability for performance: the GODEBUG knobs that make the runtime narrate what it is doing, and the runtime/metrics package that lets a process read its own runtime stats cheaply. It is a reference skill — it tells you what to look at; the action you take routes to a sibling (go-perf-gc-tuning for GC knobs, go-perf-goroutines-scheduler for scheduler depth, go-perf-tail-latency for the histograms, go-perf-execution-tracer for the tracer, go-perf-os-tooling for OS profilers).

All Go below builds clean under gofmt, go vet, and go test on Go 1.25 / 1.26.

---

1. GODEBUG — Two Jobs in One Variable

GODEBUG is a comma-separated list of key=value settings that "controls the execution of certain parts of a Go program" (godebug). It carries two unrelated kinds of setting, and conflating them is a common error:

• Compatibility toggles — Go's backwards-compatibility escape hatch. When fixing a bug would break code that "depends on buggy (including insecure) behavior," GODEBUG lets old code keep the old behavior (godebug). These have defaults derived from the go line in go.mod, a godebug directive, or a //go:debug line — e.g. panicnil=1, http2client=0, and the perf-relevant containermaxprocs ({Name: "containermaxprocs", Package: "runtime", Changed: 25, Old: "0"} — toggles the Go 1.25 container-aware GOMAXPROCS) (godebugs/table.go). Each compatibility setting also exports a counter /godebug/non-default-behavior/<name>:events so you can see in production whether old behavior is firing (godebug; metrics/description.go).
• Runtime trace knobs (§2) — diagnostic output to stderr. These are not compatibility settings; they are debugging instrumentation set only in the environment (GODEBUG=gctrace=1 ./server). "Unrecognized settings in the GODEBUG environment variable are ignored" (godebug) — so a typo silently does nothing.

---

2. The Trace Knobs (set in the environment, read stderr)

Each knob below is documented in runtime's package doc (runtime: Environment Variables, mirrored in src/runtime/extern.go). They emit lines to standard error; the formats are "subject to change."

GODEBUG=	What it emits	Use it to see
gctrace=1	one line per GC (§3)	GC frequency, pause, heap sizes, GC CPU%
schedtrace=N	one line every N ms summarizing scheduler state	run-queue depth, idle Ps/Ms
schedtrace=N,scheddetail=1	detailed multiline per-P/M/G state every N ms	which goroutines are runnable/blocked
inittrace=1	one line per package with init work (§3)	slow / allocation-heavy package init
scavtrace=1	one line ~per GC cycle from the scavenger	memory returned to the OS, RSS behavior
madvdontneed=0	(behavior toggle) use MADV_FREE not MADV_DONTNEED on Linux	why RSS stays high until memory pressure

Two cautions:

• schedtrace/scheddetail and gctrace are diagnostics, not always-on. scheddetail=1 is verbose; combining several traces interleaves their lines on the same stderr and makes both unreadable. Set one at a time.
• allocfreetrace is gone. It historically emitted a stack trace on every allocation and free — astronomically heavy and only ever a last resort. It has been removed from the modern runtime (no longer present in src/runtime/extern.go or src/runtime/*.go as of the Go 1.26 tree). For per-allocation insight today, use a heap profile (go-perf-pprof-profiling) or the execution tracer (go-perf-execution-tracer), not a GODEBUG knob.

---

3. Reading a gctrace Line Field-by-Field

gctrace=1 "causes the garbage collector to emit a single line to standard error at each collection, summarizing the amount of memory collected and the length of the pause" (runtime). The format (verbatim, "subject to change"):

gc # @#s #%: #+#+# ms clock, #+#/#/#+# ms cpu, #->#-># MB, # MB goal, # MB stacks, #MB globals, # P

Field by field, with the runtime/metrics metric that supersedes it (the doc lists these mappings inline):

Field	Meaning	Modern metric
gc #	the GC number, incremented each GC	—
@#s	seconds since program start	—
#%	percentage of time spent in GC since program start	/cpu/classes/gc/total:cpu-seconds
#+#+# ms clock	wall-clock time for the GC phases	—
#+#/#/#+# ms cpu	CPU time across the phases	/cpu/classes/gc/*
#->#-># MB	heap size at GC start, at GC end, and live heap	/gc/scan/heap:bytes
# MB goal	goal heap size	/gc/heap/goal:bytes
# MB stacks	estimated scannable stack size	/gc/scan/stack:bytes
#MB globals	scannable global size	/gc/scan/globals:bytes
# P	number of processors used	/sched/gomaxprocs:threads

"The phases are stop-the-world (STW) sweep termination, concurrent mark and scan, and STW mark termination. The CPU times for mark/scan are broken down in to assist time (GC performed in line with allocation), background GC time, and idle GC time. If the line ends with (forced), this GC was forced by a runtime.GC() call" (runtime). The #% GC-CPU field and the #->#-># live-heap field are the two to read first: a rising #% means GC is eating throughput, and a live heap that climbs run-to-run means a leak or a GOMEMLIMIT worth setting (act → go-perf-gc-tuning).

inittrace=1 is read the same way — init # @#ms, # ms clock, # bytes, # allocs per package, surfacing a package whose init is slow or allocates heavily (runtime).

---

4. runtime/metrics — The Modern, Low-Overhead Way

For anything programmatic (dashboards, alerts, in-process budgets), read runtime/metrics, not runtime.ReadMemStats. The package "provides a stable interface to access implementation-defined metrics exported by the Go runtime … similar to existing functions like runtime.ReadMemStats … but significantly more general" (runtime/metrics). Metrics are keyed by string names with an embedded unit (/path:unit), discovered at runtime via metrics.All(), so the set grows across Go versions without breaking your code; the Kind of a given metric is guaranteed stable.

package main

import (
	"fmt"
	"runtime/metrics"
)

// sampleGC reads three runtime metrics in one call. Reuse the []metrics.Sample
// slice across calls; metrics.Read fills in each Value in place.
func sampleGC() {
	samples := []metrics.Sample{
		{Name: "/gc/heap/allocs:bytes"},        // cumulative bytes allocated
		{Name: "/sched/goroutines:goroutines"}, // live goroutines
		{Name: "/sched/latencies:seconds"},     // runnable→running wait (histogram)
	}
	metrics.Read(samples)

	for _, s := range samples {
		switch s.Value.Kind() {
		case metrics.KindUint64:
			fmt.Printf("%s = %d\n", s.Name, s.Value.Uint64())
		case metrics.KindFloat64:
			fmt.Printf("%s = %f\n", s.Name, s.Value.Float64())
		case metrics.KindFloat64Histogram:
			h := s.Value.Float64Histogram()
			fmt.Printf("%s = histogram with %d buckets\n", s.Name, len(h.Counts))
		case metrics.KindBad:
			panic("unsupported metric " + s.Name) // name not in metrics.All()
		}
	}
}

Key API (runtime/metrics):

• metrics.Read([]Sample) fills each Sample.Value in place; reuse the slice between calls — the read is cheap and does not stop the world.
• metrics.All() []Description lists every supported metric (Name, Description, Kind, Cumulative). Iterate it rather than hardcoding names, so you skip a KindBad on an older runtime.
• Histograms (KindFloat64Histogram) carry Buckets []float64 and Counts []uint64 — the raw material for a p99 without coordinated omission (methodology → go-perf-tail-latency).

A few metric names worth knowing (from metrics/description.go)

Metric	Kind	Reads
/gc/heap/allocs:bytes	uint64, cumulative	total bytes ever allocated (allocation rate = its slope)
/gc/heap/allocs:objects	uint64, cumulative	total heap objects allocated
/gc/heap/live:bytes	uint64	live bytes marked by the previous GC
/gc/gogc:percent, /gc/gomemlimit:bytes	uint64	the effective GOGC / GOMEMLIMIT
/sched/latencies:seconds	histogram	goroutine runnable→running wait (scheduler saturation)
/sched/goroutines:goroutines	uint64	live goroutines (leak detection)
/sync/mutex/wait/total:seconds	float64, cumulative	global lock-contention trend
/memory/classes/total:bytes	uint64	all memory mapped read-write by the runtime

Deprecation to know: /gc/pauses:seconds is now documented as "Deprecated. Prefer the identical /sched/pauses/total/gc:seconds." (metrics/description.go). The newer GC STW pause histograms are /sched/pauses/total/gc:seconds (full pause) and /sched/pauses/stopping/gc:seconds (just the time to stop all Ps); non-GC STW events live under /sched/pauses/*/other:seconds. Read code emitting /gc/pauses:seconds and migrate it.

---

5. Why runtime/metrics Beats runtime.ReadMemStats

runtime.ReadMemStats(m *MemStats) stops the world to take its snapshot — its implementation begins stw := stopTheWorld(stwReadMemStats) (src/runtime/mstats.go). Calling it on a timer in a hot service injects a global pause on every scrape, and it returns one coarse, fixed MemStats struct that cannot express histograms or grow with the runtime. runtime/metrics is the general, forward-compatible, non-STW replacement (§4). Prefer it for all new code; reach for ReadMemStats only when interfacing with old code that already depends on a specific MemStats field.

---

6. Exposing Metrics Continuously

A one-shot metrics.Read answers "now"; production wants a time series.

• expvar is the zero-dependency option. It "provides a standardized interface to public variables … exposed via HTTP at /debug/vars in JSON format," and importing it for its side effect registers a handler plus cmdline and memstats automatically (expvar). Publish your own with expvar.NewInt("requests") or expvar.Publish(name, v) (v is any Var whose String() returns valid JSON). Bridge runtime/metrics into it with an expvar.Func:

import (
	"expvar"
	"runtime/metrics"
)

func init() {
	const goroutines = "/sched/goroutines:goroutines"
	expvar.Publish("go_goroutines", expvar.Func(func() any {
		s := []metrics.Sample{{Name: goroutines}}
		metrics.Read(s)
		return s[0].Value.Uint64()
	}))
}

• Prometheus is the standard for real fleets: the official client's collectors.NewGoCollector already sources from runtime/metrics, so you get /gc/* and /sched/* on a scrape endpoint with no custom wiring. (Use it over hand-rolled gauges; correctness of the client API is out of scope here — this skill's claim is only which source to pull from: runtime/metrics, not ReadMemStats.)

Note expvar's default /debug/vars handler — like net/http/pprof — should not be on a public port; gate it to an internal interface.

---

7. Routing to Related Skills

This plugin (go-perf-*):

• go-perf-gc-tuning — acting on a bad gctrace line or /gc/* metric: GOGC, GOMEMLIMIT, the pacer, allocation rate as the primary lever.
• go-perf-goroutines-scheduler — interpreting schedtrace//sched/* depth: GMP model, work-stealing, container-aware GOMAXPROCS.
• go-perf-tail-latency — turning /sched/latencies:seconds and /sched/pauses/total/gc:seconds histograms into p99/p999 without coordinated omission.
• go-perf-execution-tracer — when you need a timeline, not a counter: runtime/trace, go tool trace, the flight recorder.
• go-perf-os-tooling — OS-level profilers (perf, flame graphs) when the cost is below the runtime.
• go-perf-methodology — the decision layer: which diagnostic answers which question.

Parent / sibling marketplace (go-*):

• go-slog-logging — structured logging, distinct from metric counters.
• go-version-feature-map — version floors: container-aware GOMAXPROCS (1.25), Green Tea GC default (1.26).

---

8. Don't

• Don't poll runtime.ReadMemStats on a timer — it stops the world every call (mstats.go); use runtime/metrics (§5).
• Don't hardcode /gc/pauses:seconds — it is deprecated in favor of the identical /sched/pauses/total/gc:seconds (description.go).
• Don't reach for allocfreetrace — it was removed; use a heap profile or the tracer (§2).
• Don't leave scheddetail=1 (or several traces) running together — verbose, interleaved stderr that drowns the signal (§2).
• Don't typo a GODEBUG key — unrecognized settings are silently ignored, so the trace just never appears (godebug).
• Don't confuse a GODEBUG compatibility toggle with a trace knob — the first has go.mod-derived defaults and a /godebug/non-default-behavior/* counter; the second is environment-only stderr instrumentation (§1).
• Don't expose /debug/vars (or pprof) on a public port — gate it internally (§6).
• Don't read runtime/metrics and then guess at the fix here — act in go-perf-gc-tuning / go-perf-goroutines-scheduler; this skill only tells you what to look at.

---

9. Reference Files

Wrong/right anti-patterns in LLM-generated runtime-observability code, with citations:

${CLAUDE_SKILL_DIR}/references/common-mistakes.md

Source provenance for every claim in this skill:

${CLAUDE_SKILL_DIR}/references/sources.yaml