go-perf-goroutines-scheduler

name	go-perf-goroutines-scheduler
description	Guides goroutine and scheduler performance — stack cost (~2KB; cheap, but unbounded spawning isn't), the G-M-P model and work-stealing, GOMAXPROCS as the parallelism limit, container-aware GOMAXPROCS (Go 1.25; cgroup CPU limit, GODEBUG containermaxprocs/updatemaxprocs, SetDefaultGOMAXPROCS), schedtrace, syscall/M handoff. Fires on "how many goroutines is too many", "set GOMAXPROCS", "goroutines not parallel", "container CPU limit Go". Routes leaks to go-concurrency-goroutines, pool sizing to go-perf-worker-pools-throughput.
allowed-tools	Read, Glob, Grep
compatibility	Claude Code, Codex CLI, Gemini CLI

Go Performance: Goroutines & the Scheduler

"The scheduler's job is to distribute ready-to-run goroutines over worker threads. … G - goroutine. M - worker thread, or machine. P - processor, a resource that is required to execute Go code." — src/runtime/proc.go header comment

This skill owns the performance layer of goroutines and the scheduler: what a goroutine costs, how the G-M-P machine turns GOMAXPROCS into parallelism, and the runtime knobs that decide throughput in production and in containers. The correctness layer — goroutine lifetime, leaks, "never start a goroutine you can't stop," WaitGroup/errgroup — lives in go-concurrency-goroutines; read it first if the question is will this goroutine ever exit. Pool sizing and bounded fan-out route to go-perf-worker-pools-throughput.

All Go below builds clean under gofmt, go vet, and go test on Go 1.25/1.26.

---

1. What a Goroutine Costs — Cheap, Not Free

A goroutine is not a thread. Since Go 1.4's contiguous stacks, "the default starting size for a goroutine's stack … has been reduced from 8192 bytes to 2048 bytes" (Go 1.4 release notes); the runtime constant is still stackMin = 2048 (src/runtime/stack.go). The stack is not fixed: "When a stack limit is reached, a new, larger stack is allocated, all active frames for the goroutine are copied there, and any pointers into the stack are updated" (Go 1.4) — and it shrinks again during GC. So a goroutine starts at ~2KB and pays only for the depth it actually uses.

That makes a goroutine cheap — launching hundreds of thousands or millions is a supported design — but not free. Each one costs the stack (≥2KB live), scheduler bookkeeping, and GC scan work on its stack. The failure mode is unbounded spawning: one goroutine per inbound item with no ceiling means stacks and run-queue depth grow with load until the box falls over.

// WRONG — unbounded fan-out: one goroutine per item, no ceiling
for _, item := range millionItems {
	go handle(item) // memory + scheduler pressure scales with input size
}

The fix is a bounded pool (errgroup.SetLimit, a semaphore, N workers) — sizing belongs to go-perf-worker-pools-throughput, and the lifetime/leak rules to go-concurrency-goroutines. The point here: "a goroutine is cheap" is a license for concurrency, not for unbounded concurrency.

---

2. The G-M-P Model

The scheduler multiplexes goroutines onto OS threads through three entities (proc.go):

• G — goroutine. A unit of work with its own stack; thousands per thread.
• M — machine. An OS thread. "M must have an associated P to execute Go code, however it can be blocked or in a syscall w/o an associated P" (proc.go).
• P — processor. "A resource that is required to execute Go code." The number of Ps is GOMAXPROCS. A P owns a local run queue of runnable Gs; there is also a global run queue.

Parallelism is bounded by P, not by the number of Gs or Ms: at most GOMAXPROCS goroutines run Go code simultaneously, because only that many Ps exist. When a P's local queue empties, its M steals work from another P's queue (or the global queue) — the work-stealing design that keeps all Ps busy without a single contended global lock (Scalable Go Scheduler Design Doc, linked from proc.go). The practical consequence: spawning more goroutines past GOMAXPROCS does not add parallelism for CPU-bound work — it only deepens the queues the scheduler has to manage.

---

3. GOMAXPROCS Is the Parallelism Limit

runtime.GOMAXPROCS "sets the maximum number of CPUs that can be executing simultaneously" (runtime pkg); the env var "limits the number of operating system threads that can execute user-level Go code simultaneously" (runtime). Stated plainly by the 1.25 blog: "GOMAXPROCS is a parallelism limit. If GOMAXPROCS=8 Go will never run more than 8 goroutines at a time" (container-aware GOMAXPROCS).

So when someone says "my goroutines aren't running in parallel," the first checks are: is GOMAXPROCS > 1, and is the work actually CPU-bound (vs all blocked on one lock or one channel — an off-CPU problem the execution tracer diagnoses, see go-perf-execution-tracer). Manually pinning GOMAXPROCS is rarely the win it looks like — and on Go 1.25 it has a sharp side effect (§4): setting it at all disables the container-aware default.

GOMAXPROCS(0) reads the current value without changing it (passing n < 1 "does not change the current setting" (runtime)) — useful for sizing a CPU-bound worker pool to the live parallelism limit:

// Read the effective parallelism without mutating it.
procs := runtime.GOMAXPROCS(0) // e.g. the cgroup-derived value on Go 1.25
cpus := runtime.NumCPU()       // host logical CPUs — may be much larger in a container
// On Go 1.25 in a limited container, procs < cpus by design; size pools to procs.

---

4. Container-Aware GOMAXPROCS (Go 1.25)

Before Go 1.25, GOMAXPROCS defaulted to runtime.NumCPU() — the host's logical CPU count. In a container with a CPU limit (e.g. Kubernetes resources.limits.cpu), a binary on a 64-core node would set GOMAXPROCS=64 while permitted far less, oversubscribing and getting throttled by the kernel: "Throttling … completely pauses application execution for the remainder of the throttling period. The throttling period is typically 100ms, so throttling can cause substantial tail latency impact" (container-aware GOMAXPROCS).

Go 1.25 fixes the default. Two changes (Go 1.25 release notes):

1. "On Linux, the runtime considers the CPU bandwidth limit of the cgroup containing the process, if any. If the CPU bandwidth limit is lower than the number of logical CPUs available, GOMAXPROCS will default to the lower limit." This corresponds to the Kubernetes "CPU limit" option (not "CPU requests").
2. "On all OSes, the runtime periodically updates GOMAXPROCS if the number of logical CPUs available or the cgroup CPU bandwidth limit change" — so an on-the-fly limit change is picked up automatically.

CPU limits are a throughput limit and can be fractional (e.g. 2.5 CPU); GOMAXPROCS must be a positive integer, and "Go always rounds up to enable use of the full CPU limit" (blog).

The toggles (Go 1.25 release notes):

• Both behaviors "are automatically disabled if GOMAXPROCS is set manually via the GOMAXPROCS environment variable or a call to runtime.GOMAXPROCS." This is the trap: a hard-coded runtime.GOMAXPROCS(n) or an GOMAXPROCS= env var opts out of the container default entirely, including the periodic updates.
• They "can also be disabled explicitly with the GODEBUG settings containermaxprocs=0 and updatemaxprocs=0, respectively." With GODEBUG=containermaxprocs=0, GOMAXPROCS defaults back to runtime.NumCPU (runtime).
• runtime.SetDefaultGOMAXPROCS() "sets GOMAXPROCS to the runtime default value, as if the GOMAXPROCS environment variable is not set" — use it to re-enable the container-aware default after something disabled it, or to force an immediate re-read (runtime).

// On Go 1.25, prefer the default — do NOT hard-code GOMAXPROCS in a container;
// pinning it opts out of the cgroup-aware default and the periodic updates.
// If a library or env var disabled it, re-enable the container-aware default:
runtime.SetDefaultGOMAXPROCS()

For older Go, the uber-go/automaxprocs package was the standard workaround; on 1.25+ it is no longer needed.

---

5. Syscalls and cgo Block an M, Not a P

When a goroutine makes a blocking syscall (or a cgo call), its M blocks in the kernel — but the runtime detaches the P so another M can keep running the remaining goroutines. "There is no limit to the number of threads that can be blocked in system calls on behalf of Go code; those do not count against the GOMAXPROCS limit" (runtime). This is why a server with thousands of goroutines each blocked on I/O still makes progress on GOMAXPROCS cores.

The cost is the handoff: detaching the P, and waking or spinning up a fresh M to adopt it, is real work. Code that hammers short blocking syscalls (or crosses the cgo boundary in a tight loop) pays repeated handoff overhead and can balloon the OS-thread count. Two levers: batch syscalls (see go-perf-buffered-io, go-perf-zerocopy-and-syscalls) so there are fewer of them, and cap threads with runtime/debug.SetMaxThreads as a safety valve. Network I/O on Go's own net types does not block an M — it parks the goroutine on the runtime poller — so this concern is specific to file/blocking syscalls and cgo.

---

6. Observing the Scheduler — GODEBUG=schedtrace

To see whether the scheduler is the bottleneck, read its own trace. "Setting schedtrace=X causes the scheduler to emit a single line to standard error every X milliseconds, summarizing the scheduler state"; adding scheddetail=1 "causes the scheduler to emit detailed multiline info … describing state of the scheduler, processors, threads and goroutines" (runtime).

GODEBUG=schedtrace=1000 ./server          # one summary line per second
GODEBUG=schedtrace=1000,scheddetail=1 ./server   # per-P / per-M / per-G detail

A summary line looks like this (fields are runtime-internal and version-dependent, so read trends, not absolutes):

SCHED 1000ms: gomaxprocs=8 idleprocs=0 threads=21 ... runqueue=137 [12 18 22 ...]

Read it for run-queue depth: runqueue= is the global queue and the bracketed list is each P's local queue. A persistently deep global queue with idleprocs=0 (all Ps busy) means you are CPU-saturated — adding goroutines won't help; you need less work or more cores. The opposite — idle Ps with a near-empty runnable count — means the goroutines are blocked off-CPU (§5, mistake #5), a tracer/contention question, not a GOMAXPROCS one. For latency attribution (how long a goroutine sat runnable before running) the execution tracer and runtime/metrics /sched/latencies:seconds are sharper tools — route to go-perf-execution-tracer and go-perf-godebug-and-metrics.

---

7. GC Assist Steals an Allocating Goroutine's CPU

Scheduler "overhead" is sometimes really GC. When the program allocates faster than the background collector keeps up, the runtime recruits the allocating goroutines to help: runtime.gcAssistAlloc is the "function goroutines enter to yield some of their time to assist the GC with scanning and marking. A large amount of cumulative time spent here (>5%) indicates that the application is likely out-pacing the GC" (GC guide). It shows up as "User goroutines assisting the GC in response to a high allocation rate" (GC guide).

The tell: a CPU profile shows time in gcAssistAlloc, and request latency correlates with GC cycles. The fix is fewer allocations, not a scheduler knob — route allocation reduction to go-perf-allocator-internals / go-perf-gc-tuning and the tail-latency framing to go-perf-tail-latency.

---

8. Goroutine-Per-Request vs Bounded

Goroutine-per-request (one goroutine per connection/request) is idiomatic and correct for servers — net/http does exactly this, and the ~2KB stacks make it scale to very high connection counts. Spawning is fine when the count is bounded by something external you trust (open connections, a fixed work set). Bound it when the count is driven by unbounded input (a queue you drain, a fan-out over a large slice, a recursive explosion) — there, an unbounded go per item is the §1 failure. The decision rule: is there a natural ceiling on how many run at once? If not, impose one. The sizing of that ceiling (CPU-bound near GOMAXPROCS, I/O-bound higher) is go-perf-worker-pools-throughput.

Watch the live count with runtime.NumGoroutine() (or the /sched/goroutines:goroutines runtime/metrics sample): a count that climbs with load and never drains is the tell for unbounded spawning or a leak — the leak case (goroutines that never exit) is go-concurrency-goroutines, the throughput case (too many alive at once) is the bounded-pool fix here.

---

9. The Headline Disciplines

Discipline	The rule	Source
Cheap ≠ free	~2KB start, but bound fan-out over unbounded input	"reduced from 8192 bytes to 2048 bytes" (Go 1.4)
P bounds parallelism	At most GOMAXPROCS Gs run Go code at once	"GOMAXPROCS is a parallelism limit" (blog)
Trust the 1.25 default	Don't pin GOMAXPROCS in a container; it opts out	"automatically disabled if GOMAXPROCS is set manually" (Go 1.25)
Re-enable explicitly	SetDefaultGOMAXPROCS() restores the container default	"as if the GOMAXPROCS environment variable is not set" (runtime)
Syscalls block M, not P	Blocked-in-syscall threads don't count against the limit	"no limit to the number of threads … blocked in system calls" (runtime)
Measure the scheduler	schedtrace before any knob	"emit a single line … summarizing the scheduler state" (runtime)
Assist ≠ scheduler	gcAssistAlloc in a profile is allocation pressure	">5% … out-pacing the GC" (GC guide)

---

10. Don't

• Don't spawn one goroutine per item over unbounded input. Cheap ≠ free; bound the fan-out (go-perf-worker-pools-throughput); lifetime/leak rules in go-concurrency-goroutines.
• Don't hard-code runtime.GOMAXPROCS(n) (or set the env var) in a container on Go 1.25 — it disables the cgroup-aware default and the periodic updates; prefer the default or SetDefaultGOMAXPROCS() (Go 1.25).
• Don't assume GOMAXPROCS = host cores under a CPU limit on pre-1.25 Go — it oversubscribes and the kernel throttles in ~100ms bursts (blog); upgrade or use automaxprocs.
• Don't expect more goroutines than GOMAXPROCS to add parallelism for CPU-bound work — they only deepen run queues (blog).
• Don't blame the scheduler for an off-CPU stall — goroutines blocked on a lock/channel are a tracer/contention question (go-perf-execution-tracer, go-perf-block-mutex-profiles), not a GOMAXPROCS one.
• Don't ignore gcAssistAlloc in a CPU profile — that "scheduler overhead" is GC pressure; reduce allocations (go-perf-gc-tuning).
• Don't hammer short blocking syscalls/cgo in a tight loop — each pays P-handoff and can grow the thread count; batch instead (go-perf-buffered-io).

---

11. Routing to Related Skills

• go-concurrency-goroutines — goroutine correctness: lifetime, leaks, defined exit, WaitGroup/wg.Go, errgroup, synchronous libraries. Read first for "will this exit?".
• go-perf-worker-pools-throughput — bounding fan-out and sizing pools (SetLimit, semaphore, batching).
• go-perf-execution-tracer — scheduler-latency and goroutine-analysis views; the flight recorder for capturing a stall.
• go-perf-godebug-and-metrics — the GODEBUG catalog and runtime/metrics (/sched/goroutines, /sched/latencies:seconds).
• go-perf-tail-latency — p99/p999 and why throttling/GC drive the tail.
• go-perf-gc-tuning / go-perf-allocator-internals — reducing the allocation rate behind GC assist.
• go-perf-atomics-vs-locks, go-perf-contention-and-sharding — when "not parallel" is really lock contention.
• go-version-feature-map — version floors: container-aware GOMAXPROCS 1.25.

---

12. Reference Files

Scheduler/goroutine performance anti-patterns in LLM-generated Go, each with wrong/right contrast and citations:

${CLAUDE_SKILL_DIR}/references/common-mistakes.md

Source provenance for every claim in this skill:

${CLAUDE_SKILL_DIR}/references/sources.yaml