fuzzing-an-api-surface

# Fuzzing an API surface Use this skill when you want to systematically hammer a surface that decodes input you do not control, and have every real failure become a named regression test rather than a transient console reproduction. Evener is the proving ground; the portable core (`fuzz/promoter`, `fuzz/schemagen`) carries unchanged to any project, so the methodology below is project-agnostic and the evener targets are worked examples. Read `docs/research/api-fuzzing-toolkit.md` (§3 oracle soft spots, §5 the promote discipline, §8 the portable split) and `docs/design/fuzzing-toolkit-design.md` before extending the toolkit. ## When to reach for this A surface is fuzz-ready when it has: - **A single dispatch choke point** — one function every input flows through (`tool.Registry.ExecuteCall`, `Router.Dispatch`, `WebServer.Handler()`). One harness then covers the whole surface by varying which operation it drives. - **A machine-readable schema** — per-operation JSON Schema, reflected param types, OpenAPI, protobuf. Schema-driven generation reaches deep validator paths raw bytes never hit (evener hands us tool `Definition.Parameters` and the `appwire.Methods` catalog for free). - **A decode / parse seam** — a byte-level frame/stream parser (`llm.ParseSSE`, `appwire.Message.UnmarshalJSON`). These are the cheapest, highest-value targets: a Go `testing.F` byte fuzzer auto-promotes its own crashers. - **A metamorphic relationship** — two ways to feed the same logical input that must agree (re-chunked reads, reordered/whitespace-padded frames, blocking vs timeout code paths). The relationship *is* the oracle. Do NOT reach for this when the seam's only effect is to **execute** something dangerous (see Safety bounds), when no determinism is achievable offline (the real behavior needs a live LLM/network/subprocess), or when there is no oracle richer than "it returned" — a fuzzer with only a floor oracle on a trivial surface buys little. ## Core rules (non-negotiable) - **Safety bound: never let a fuzzer EXECUTE a dangerous handler.** Shell (`bash -c`), network fetch, file read/write under attacker-controlled paths, subprocess/agent spawn — a temp dir does NOT sandbox these. Fuzz the **decode + validate boundary** instead, or restrict execution to a vetted allowlist of genuinely sandboxable operations. Evener's `FuzzToolArgsValidate` drives decode+`Schema.Validate` and never calls a tool handler; `FuzzWebHandler` allowlists GET-only, non-mutating, non-networked routes and excludes spawn, shell, and provider-probe endpoints. - **Determinism is non-negotiable.** Seed the generator; run offline (no real LLM, network, or subprocess); a flaky target — or a promoter that *could* promote a flake — is a defect, not a nuisance. - **"No panic" is the floor, never the only oracle.** Invest in semantic oracles; the real bugs live there. - **A thin-but-real model beats a broad fake one.** Model only what you can drive deterministically against the real seam, and report the boundary you could not reach. Never fake the unreachable part and assert on the mock — that tests mocked behavior. `TestRouterSeqFuzz` models exactly the transport initialize-gate it can reach offline and states in its doc comment that turn/job *status* lifecycle needs a live agent and is out of scope. ## Procedure: pick a surface and seam (research §8, 7 steps) 1. **Pick a surface and its seam.** Find the single choke point all input flows through. That is what the harness drives. 2. **Find or derive the schema.** Point generation at whatever machine-readable structure exists (`Definition.Parameters`, reflected `appwire.Methods`, OpenAPI). No schema → fall back to byte fuzzing. 3. **Write the thinnest `testing.F` first** and harvest the free auto-promotion: a single `[]byte` or typed-tuple target whose crashers Go saves to `testdata/fuzz/` and re-runs forever. This is ~60–200 LoC and finds the crash-class bugs before you build anything else. 4. **Add oracles beyond "no panic"** (next section). This is where valid-but-wrong bugs surface. 5. **For stateful surfaces, model the state machine** and drive it with property-based testing (`rapid`). Keep the model declarative — evener's `liveOpTable` is a transition table, the reusable artifact; the `rapid` machine just draws op sequences. 6. **Wire the promoter with flake-guard before promote** so non-`testing.F` failures (`rapid` sequences, HTTP op-sequences) become permanent. 7. **Gate it in CI:** seed corpus + saved crashers under `-short` on every run (`make fuzz` / `make test`); the unbounded coverage-guided search runs nightly (`make fuzz-nightly`, 60s/target). ## The oracle taxonomy The portable tags are `promoter.OracleTag`: `Panic`, `Invariant`, `ErrorShape`, `Wedge`, `HTTP5xx`, `PathEscape`. How to spot which fits, each tied to a real evener target: - **Floor — never panic** (`Panic`). The seam crashes on some input. Always present; never sufficient alone. Live panic-hunt wherever validation is *not* `recover()`-wrapped at runtime — e.g. `Schema.Validate` in `FuzzToolArgsValidate`, and `Router.Dispatch` (which propagates handler panics) in `TestRouterSeqFuzz`. - **Metamorphic** (`Invariant`). Two equivalent inputs must produce the same output. Spot it when one logical input has more than one delivery shape: re-chunked reads or blocking-vs-timeout paths (`FuzzParseSSE`), one-byte-at-a-time SSE accumulation (`FuzzOpenAIResponsesMetamorphic`), decode→encode→decode fixed point (`FuzzMessageDecode`, `FuzzMethodParams`). - **Divergence** (`ErrorShape`). A validator says clean but the handler disagrees, or vice versa. Spot it when validation and use are separate steps: `TestToolArgsSchemaFuzz` flags a schema-VALID generated value the real validator rejects — generator and schema disagree. - **Invariant / monotonicity** (`Invariant`). A modeled property must hold across a sequence. Spot it on stateful surfaces with an ordering contract: `TestRouterSeqFuzz` asserts the connection's initialize-gate is monotonic (fresh → initialized, never reverts). - **Never 5xx** (`HTTP5xx`). Bad input may legitimately 4xx, but a 500 from your own handler is a fault. Spot it on HTTP surfaces: `FuzzWebHandler` (it excludes the bare stdlib `http.FileServer`, whose `fs.ErrInvalid`→500 is documented net/http behavior, not handler logic). - **Path escape** (`PathEscape`). Attacker-controlled path/query escapes its intended root. Spot it on any file/path resolver: `FuzzWebHandler` plants a secret one level above the session cwd and fails if any response body contains it. - **Never wedge** (`Wedge`). A call that should return makes no progress. Spot it on synchronous seams that could hang: `TestRouterSeqFuzz` runs each dispatch under a bounded context in a goroutine and reports a wedge on timeout. - **Internal invariant** (`Panic`, under `-tags evenerfuzz`). A load-bearing assumption asserted *inside* production code with `invariant.Hold()` so a logic bug trips at its origin, not at a distant surface. Zero-cost in a normal build; panics under `-tags evenerfuzz` (the fuzz build), so it surfaces via the Panic oracle. Spot it wherever a subsystem relies on a property a reader assumes — e.g. a folded job status never leaves a terminal state; an emitted item carries its turn id. Conditions must be side-effect-free; verify the invariant is TRUE and reached before trusting it. - **Differential** (`Invariant`/`ErrorShape`). Two code paths that must compute the same thing, driven from one input and compared modulo an allow-list. The strongest class — it found both real decoder bugs this codebase has caught. Spot it wherever a value is produced two ways: per-provider streaming vs non-streaming decode (`FuzzStreamVsNonStreamDifferential`), one logical response across all providers (`FuzzCrossProviderDifferential`), a decoder vs a committed snapshot of its own output (`appwire/golden_test.go`), or independent readers of one format (`FuzzTranscriptReadersAgree`). A divergence outside the allow-list is a real bug — never widen the allow-list to silence it. **Generation technique — structure-aware inputs.** Independent of which oracle you pick: if the surface has a rich grammar, most raw-byte execs die at the first parse. Drive the target from a deterministic byte-`Source` generator (`fuzz/schemagen`) that emits *valid-but-adversarial* inputs (the `Fuzz…Structured` targets), so coverage-guided search reaches deep logic. It typically lifts input acceptance from ~0% to ~90%+. ## The portable core Two packages in the `primeradiant.com/evener/fuzz` module are the travelling tooling. **Nothing in them imports the project under test** — `fuzz/go.mod` declares no evener dependency, so the module will not build if that boundary is violated. That structural guarantee *is* the portability test. **`schemagen.FromJSONSchema(schema map[string]any) *rapid.Generator[any]`** walks the JSON Schema subset a surface actually uses (`type`/`properties`/`required`/ `enum`/`additionalProperties`/`items`) and yields both schema-`Valid` and schema-`Adjacent` (wrong-type, missing-required, out-of-enum, extra-when-closed) values. Use `schemagen.Generator(schema, mode)` when an oracle needs known-valid input. **`promoter.Adapter`** is the four-hook seam a project implements; everything else (flake-guard loop, bucket store, emitter, commit) is generic: ```go type Adapter interface { Minimize(Failure) Failure // usually a passthrough Signature(Failure) Signature // dedup key Replay(context.Context, Failure) (failed bool, sameSignature bool) Emit(Failure) (path string, err error) } ``` A project implements it by carrying enough state to deterministically reproduce a failure. `toolArgsAdapter` holds each tool's compiled validator; `seqAdapter` holds the transition table + handler registrar. `Replay` re-runs the *same* oracle function that found the failure (`toolArgsFailure` / `seqOracleRun`), so the live property and the replay classify identically. `Emit` calls `promoter.WriteGoTest(dir, promoter.GoTest{...})` with an adapter-rendered `ReplayBody` that decodes the minimized artifact and asserts the bug is fixed. `promoter.New(adapter, store, quarantiner, K)` then runs `Promote`. ## The flake-guard / dedup rubric (the load-bearing discipline) `Promoter.Promote` is the whole point. In order: 1. **Dedup first.** `store.Has(sig)` against the committed `BucketStore` (a JSON file mapping `Signature` → test path). A known bug returns `AlreadyKnown` and is never re-promoted. 2. **Flake-guard.** Replay the minimized artifact **K times** (default 5). Promote ONLY if it fails all K with the **same** signature. Any non-reproduction or signature shift → `Quarantined`: logged via the `Quarantiner`, **never written as a test**. This is the rule that keeps the gate trustworthy; budget it generously on timing-sensitive seams. 3. **Emit-only by default.** A surviving failure writes a `TestRegression_*` file with a provenance header (`// Code generated by fuzz/promoter ... DO NOT EDIT`) and records its bucket. Promotion *into the tree* is the human/opt-in step — review the diff before committing. Signatures must distinguish distinct bugs: `Signature{Oracle, Key}` where for `Panic` the Key is the top-N normalized stack frames, and for semantic oracles (`Invariant`/`ErrorShape`) the adapter folds the named invariant / `Detail` into Key. Collapse them and distinct bugs silently dedup as already-known. ## Hard-won lessons - **rapid owns its shrink loop.** `pgregory.net/rapid` reports failures through `testing.T` and Goexits during shrinking; it does **not** hand you the minimized artifact as a return value. Capture the failing artifact into a closure variable inside `rapid.Check`, and promote it in `t.Cleanup` (which runs during the Goexit unwind). `Adapter.Minimize` is then a passthrough — rapid already minimized. Both `TestToolArgsSchemaFuzz` and `TestRouterSeqFuzz` use this `captured *Failure` + cleanup pattern. - **Liberal-in / conservative-out — the fix location matters.** `FuzzMessageDecode` found a real codec asymmetry: the decoder accepted an id-less response/error frame the encoder then re-rendered unreadable, breaking the round-trip fixed point. The fix (commit `6c7f4eac`) was on the **producer** side — `Response.MarshalJSON`/`ErrorResponse.MarshalJSON` omit the empty id — keeping the decoder liberal. On a transport, tightening a decoder turns a tolerable frame into a connection-fatal error; tighten the producer instead. - **Model only what you can drive for real.** When the real status lifecycle needs a live LLM, model the transport/dispatch layer you *can* reach offline and report the boundary in the target's doc comment. Do not fabricate the unreachable part. ## Per-language portability The loop, the triage/promotion discipline, schema-driven generation, and the oracle taxonomy generalize. Only the engine, corpus format, and emitter syntax are per-ecosystem, all behind the same `Adapter`: - **Go:** `testing.F` (coverage-guided, free crasher auto-promotion) for byte/tuple targets; `pgregory.net/rapid` for stateful/structured PBT. Emitter writes a `*_test.go`; corpus is `f.Add` seeds + `testdata/fuzz/`. - **JS:** `fast-check` (property/model-based) for a renderer/protocol surface; its emitter writes a `test-*.js`. - **Python:** `hypothesis` (+ `schemathesis` for HTTP); its emitter writes a `test_*.py`. ## Cross-module wiring (Go specifics) A local `v0.0.0` workspace module is not `go get`-able. To consume `fuzz/promoter` and `fuzz/schemagen` from another module in the same `go.work`: 1. Add `require primeradiant.com/evener/fuzz v0.0.0` (and `pgregory.net/rapid`) to the consumer's `go.mod`. 2. Add `replace primeradiant.com/evener/fuzz v0.0.0 => ./fuzz` in `go.work`, then `go work sync`. 3. Seed third-party checksums the workspace replace hides: `GOFLAGS=-mod=mod GOWORK=off go mod download pgregory.net/rapid`. Add `fuzz` to `GO_MODULES` so its tests are gated. `make fuzz` runs every `Fuzz*` target's seed corpus + saved crashers per module as deterministic tests (no `-fuzz` flag = no random search); `make fuzz-nightly` runs the bounded search.