go-concurrency-playbook

name	go-concurrency-playbook
description	Use when designing or debugging high-concurrency Go services, especially worker pools, backpressure, bounded channels, errgroup/WaitGroup, rate limits, load shedding, circuit breakers, GOMAXPROCS, or connection pools; also for load-time OOM, rising goroutines, lock contention, p99 spikes, retry storms, pprof, singleflight, semaphore, or production-only collapse.

🌐 繁體中文（canonical） · English mirror

Go High-Concurrency Judgment Handbook

Prerequisite: single-machine correctness (the error three-way choice, the three classic goroutine leaks, nil interface, context rules, channel vs mutex, project structure) lives in backend-server's references/go.md; this handbook does not repeat it. This handbook picks up from "it runs correctly on one machine" and moves up to "it still holds when ten thousand requests pour in at once."

Core Principles

1. Concurrency is a design decision, not a speed button: before spawning a goroutine, ask "is the bottleneck really here?" Adding concurrency without evidence of a bottleneck only turns bugs into concurrent bugs.
2. Any unbounded buffer is a deferred OOM: every boundary where production might outpace consumption needs to be bounded + have an explicit "what happens when it's full" decision (pick one of block / drop / reject).
3. For high-concurrency problems, look at "waiting" first, not CPU: the investigation order is goroutine profile (leaks/blocking) → lock contention → CPU → trace (scheduling). Most "slowness" is waiting on a lock, on a downstream, or on the scheduler — not slow computation.
4. "Feels slow" is forbidden as a reason to change code directly: profile for evidence before acting; adding goroutines / caches / swapping data structures on a hunch usually fixes the wrong place and makes it slower.
5. Problems that only blow up in production can only be verified with load testing: low-traffic single-machine runs have zero evidentiary weight for "leaks, lock contention, pool saturation, GOMAXPROCS misconfiguration" — these only surface under load at the target RPS.

Kickoff Routing

Situation	Path	Read first
Thinking "spawn a few goroutines to speed it up"	First confirm the bottleneck and whether concurrency is even needed, then decide on a pool	references/concurrency-patterns.md §1
Need a worker pool / bounded concurrency / fan-out	Set worker count and bounds, pick the right primitive	references/concurrency-patterns.md
Multiple subtasks to run in parallel + error propagation/cancellation	errgroup, not a bare WaitGroup	references/concurrency-patterns.md §3
High-concurrency reads on the same key + expensive origin fetch	singleflight to prevent cache stampede	references/concurrency-patterns.md §5
Wiring a channel/queue, fear OOM, setting timeouts	Bounded + backpressure + timeout budget decreasing layer by layer	references/backpressure-limits.md
Adding rate limit / throttling / circuit breaker / load shedding	Selection and the "is it really needed" criterion	references/backpressure-limits.md
Service got slow, need to tune performance	Profile for evidence first, don't change anything yet	references/performance-profiling.md
Suspect lock contention / GC pressure	mutex/block profile, fix priority order	references/performance-profiling.md
Before a release / going to high traffic	Run the mandatory checklist item by item (load testing is a hard requirement)	references/release-checklist.md

Red Lines (Absolutely Forbidden)

• Never use an unbounded channel / unbounded queue / unbounded go func() as a buffer — when production outpaces consumption, memory grows without ceiling and a spike OOMs directly; and before the OOM, GC pressure slows the whole service first.
• Never set an inner timeout ≥ the outer timeout — the inner layer waits to the full duration while the outer layer has long since given up and returned an error; the inner layer is still running = wasted work + held resources, and cancellation semantics are entirely defeated.
• Never optimize performance directly on "feels slow" — optimization without profile evidence is a guess; fixing the wrong place wastes time and may make things slower.
• Never pull worker count / connection pool size out of thin air and fill in 100 — concurrency exceeding downstream capacity (DB max_connections, upstream API quota) only creates queuing and timeouts; the number must be derived backward from downstream capacity.
• Never skip -race verification in high-concurrency code — a data race is intermittent at low traffic, inevitable and non-reproducible at high traffic; -race is the only means that reliably catches it.
• Never retry without backoff and jitter — one downstream hiccup and all clients retry in lockstep, forming a retry storm that amplifies a transient fault into an avalanche.

Failure Signals (Turn Back, Don't Retry)

Symptom	Usually means	Turn back to
goroutine count rises monotonically over time and never falls	Leak (missing ctx exit / channel with no reader)	go.md §2 the three classic leaks + profiling §1
Adding a pool made it slower or timeout more easily	worker count exceeds downstream capacity, stuck queuing	patterns §2 re-set worker count
Memory rises slowly and eventually OOMs	Some channel/queue is unbounded, or a leak	backpressure §1 add bounds
Every added timeout/retry spawns a new timeout	timeout layers not aligned, inner > outer	backpressure §3 re-order the budget
Lock contention persists after shrinking the critical section	Wrong lock granularity (should be sharded, isn't)	profiling §3 fix priority order
p99 spikes but CPU isn't maxed	Waiting on a lock/downstream/scheduler, not slow computation	profiling, re-investigate from goroutine/block profile
Service still avalanches after downstream recovers	Retry storm / no circuit breaker	backpressure §4 backoff+jitter+circuit breaker

References Index

• references/concurrency-patterns.md — when to "not" spawn goroutines, worker pool sizing, errgroup vs WaitGroup, pipeline/fan-out, semaphore for bounded concurrency, singleflight. Read before deciding concurrency structure.
• references/backpressure-limits.md — bounded channels and what-happens-when-full, rate limit selection, timeout budget layering, when load shedding/circuit breaker is really needed, connection pool sizing. Read before wiring a boundary or adding throttling.
• references/performance-profiling.md — pprof investigation order, -race, lock contention fix priority order, GC/sync.Pool criteria, correct benchmark posture. Read before tuning performance.
• references/release-checklist.md — load testing, automaxprocs, ulimit, the four golden signals, graceful shutdown draining, retry storm protection. Tick off item by item before going to high traffic.
• references/test-scenarios.md — the judgment test set, to verify whether the model taking over actually follows along. Do not give to a model mid-execution.