reduce-test-cases

name

reduce-test-cases

description

Reduce, shrink, or minimize a failing Wasmtime/Cranelift test case (.wasm/.wat, .wast, or .clif) down to the smallest input that still reproduces a bug, crash, trap, panic, or miscompilation. Gives you a predicate/command skeleton per reducer (wasm-tools shrink, Binaryen wasm-reduce, creduce, clif-util bugpoint), a wasm fallback chain for when a reducer can't parse your module, and a manual reduction playbook.

Reduce Test Cases

A minimal reproducer is the most valuable artifact in a bug report. Reduce a test case in two phases:

1. Automated reducers — the approach from docs/contributing-reducing-test-cases.md, extended to .wast and .clif and to a fallback chain for wasm the primary reducer can't parse.
2. Manual reasoning — when a reducer stalls, doesn't apply (e.g. a .clif miscompile, which bugpoint can't touch), or over-shrinks past your bug.

The predicate is the whole game. Every reducer does the same dumb loop: mutate the input, ask a yes/no script "does this still reproduce the bug?", keep the mutation if yes. The reducer is generic; the predicate is where all the per-bug intelligence lives. This skill does not ship a do-everything driver — your bug is not knowable in advance. Instead templates/ holds a skeleton per reducer that you copy and edit to pin your exact symptom:

Reducer	How it hands you the candidate	"Interesting" means	Skeleton
wasm-tools shrink	as $1	script exits 0	templates/predicate-shrink.sh
Binaryen wasm-reduce	writes a fixed --test file	script's stdout and exit code both match the original	templates/command-wasm-reduce.sh
creduce	candidate by basename, fresh temp dir	script exits 0	templates/predicate-creduce-{wat,wast,clif}.sh
clif-util bugpoint	(no predicate)	compile panics	(built in)

For a miscompile — no trap or panic to grep, just a wrong result — the predicate diffs behavior across two configs instead. That shape is orthogonal to the reducer, so it has its own skeleton, templates/predicate-differential.sh, that you drop into the shrink or creduce contract above. In practice this was the most common real-world case, so reach for it first when the bug is a wrong answer.

All paths below are relative to the repo root.

Prerequisites

Install the reducers (wasm-tools ships with the toolchain; the rest are external):

cargo install wasm-tools     # shrink + print (often already present)
brew install creduce         # generic text reducer for .clif / .wast / .wat
brew install binaryen        # wasm-reduce, the alternate wasm reducer

Build the engines your predicates will run. A debug build is required to reproduce debug_assert! panics; a --release build iterates much faster for everything else, which matters when a reducer runs the predicate thousands of times:

cargo build --bin wasmtime                                                    # debug: target/debug/wasmtime
cargo build --release --bin wasmtime                                          # fast: target/release/wasmtime
cargo build --release --manifest-path=./cranelift/Cargo.toml --bin clif-util  # target/release/clif-util

Every skeleton has a WASMTIME= / CLIF_UTIL= line you must set to an absolute path — reducers run from other directories, so a relative path breaks. Print the absolute path and paste it in:

echo "$PWD/target/release/wasmtime"
echo "$PWD/target/release/clif-util"

Workflow

Work in a scratch directory holding your failing input. The loop never changes:

1. Reproduce the bug once, by hand, to pin the exact symptom — the specific trap string, panic message, ISLE error, or diverging output. This is the most important step: a vague predicate reduces to a different bug. For a miscompile, also confirm it's deterministic — run each config a few times and check every run of a config agrees and the two configs disagree; a flaky config makes a differential predicate unsound.
2. Copy the right skeleton from templates/ next to your input.
3. Edit it: set the absolute engine path, and replace the example symptom with yours. Keep it tight — a loose predicate lets the reducer converge on some other, spec-legal failure.
4. Test the predicate on the ORIGINAL. If it doesn't fire here, it never will on a smaller file. (For shrink/creduce: ./pred.sh input should exit 0. For wasm-reduce: the command should print your marker.)
5. Run the reducer (per format, below).
6. Eyeball the result and re-confirm it's still your bug — reducers are happy to find a smaller program that fails for an unrelated reason.
7. Re-verify the final case independently against the predicate — especially anything a subagent/fork produced, or a hand-reduction. Convert every candidate to the same form before comparing sizes (wasm bytes and canonical wasm-tools print line count); raw creduce/hand whitespace isn't comparable.
8. Re-author the winner to be legible, not just small. Give locals/functions readable names, add a one-paragraph comment explaining the mechanism, and for a .wast make it self-checking with assert_trap / assert_return so the file proves the bug on its own. The best artifact teaches the fix.

wasm / wat — a three-tool fallback chain

Each wasm reducer only understands the proposals its own parser implements, and those subsets differ. Work down the chain until one bites:

1. wasm-tools shrink — try this first

Smartest, most structural, produces the smallest output. Uses templates/predicate-shrink.sh (candidate is $1, exit 0 == reproduces):

cp .agents/skills/reduce-test-cases/templates/predicate-shrink.sh pred.sh
# edit pred.sh: set WASMTIME=<abs path>, set the symptom grep
chmod +x pred.sh
./pred.sh input.wasm && echo "reproduces on original"   # sanity check first
wasm-tools shrink ./pred.sh input.wasm -o reduced.wasm
wasm-tools print reduced.wasm

Verified end-to-end: a 273-byte module (noise functions, globals, a table, data segments) reduced to 42 bytes — exactly the trapping core:

(module
  (type (;0;) (func))
  (export "_start" (func 0))
  (func (;0;) (type 0)
    i32.const 42
    i32.const 0
    i32.div_s
    drop))

2. Binaryen wasm-reduce — if shrink can't parse your module

Different proposal coverage from shrink — and the gap is real, not theoretical: Binaryen can't parse wide-arithmetic (i64.mul_wide_s, i64.add128) or some SIMD, while wasm-tools shrink handles them. So for a module using those, skip straight from shrink to creduce; wasm-reduce is a non-starter. It also has a different contract: it writes each candidate to a fixed --test file (not $1), and keeps a candidate only if your command's stdout and exit code both match the original run — so print one canonical marker when the bug reproduces. Uses templates/command-wasm-reduce.sh:

cp .agents/skills/reduce-test-cases/templates/command-wasm-reduce.sh cmd.sh
# edit cmd.sh: set WASMTIME=<abs path>, set the symptom marker
cp input.wasm t.wasm && bash cmd.sh          # sanity check: prints your marker
wasm-reduce input.wasm --command 'bash ./cmd.sh' --test t.wasm --working reduced.wasm -f

The -f matters: before reducing, wasm-reduce round-trips your module through Binaryen's own parser/writer, and if it can't (an unsupported proposal) it prints failed to read and write the binary and stops. -f forces past that. Verified: with -f, the same 273-byte module reduced to 91 bytes and still trapped — larger than shrink's 42 bytes (it kept the table/memory/globals), which is exactly why this is a fallback, not the first choice.

3. Disassemble + creduce — if neither reducer can parse it

creduce reduces text and never re-encodes the wasm, so only the engine has to understand the proposal — the reducer stays out of the way. This always runs, at the cost of being purely textual: because it doesn't understand .wat structure it tends to plateau early on parenthesis-balance constraints (in one real case it stalled at ~43% of the original and no further). Treat its output as a starting point for the manual playbook, not the finish line. wasmtime runs a .wat directly, so no reassembly is needed. Uses templates/predicate-creduce-wat.sh:

wasm-tools print input.wasm -o case.wat          # disassemble
cp .agents/skills/reduce-test-cases/templates/predicate-creduce-wat.sh pred.sh
# edit pred.sh: set WASMTIME=<abs path>, set the symptom; CAND=case.wat
chmod +x pred.sh
mkdir -p /tmp/reduce && cp pred.sh case.wat /tmp/reduce && cd /tmp/reduce
creduce ./pred.sh case.wat                        # reduces case.wat in place

Verified: a 52-line disassembly reduced to two lines that still trap:

(module(export "_start"(func $a))(
    func $a i32.const 2 i32.const 0 i32.div_s drop))

wast — creduce

wasm-tools shrink reduces one module; a .wast is a whole script (multiple modules + assertions), so reduce the text with creduce. Run the script with wasmtime wast. Uses templates/predicate-creduce-wast.sh:

target/release/wasmtime wast input.wast           # reproduce first
cp .agents/skills/reduce-test-cases/templates/predicate-creduce-wast.sh pred.sh
# edit pred.sh: set WASMTIME=<abs path>, set the symptom; CAND=input.wast
chmod +x pred.sh
mkdir -p /tmp/reduce && cp pred.sh input.wast /tmp/reduce && cd /tmp/reduce
creduce ./pred.sh input.wast

Verified: a 30-line, 3-module script (with passing-assertion noise) reduced to a single line that still traps with integer divide by zero:

(module(func $a i32.const 2 i32.const 0 i32.div_s drop)(start $a))

Feature flags come from -W on the CLI (e.g. -W gc,function-references), not from the ;;! feature = true header — that header is only read by cargo test --test wast. creduce will delete the ;;! header (the CLI ignores it), so re-add it if the reduced case is destined for tests/.

clif — bugpoint (panics only) or creduce (everything else)

clif-util bugpoint <FILE> <TARGET> reduces compile panics only. It has no predicate hook — it catches panics internally and treats a clean compile or a verifier error as "not interesting". On a .clif that fails gracefully (the common test compile expect-fail case) it bails:

target/release/clif-util bugpoint be.clif riscv64
# Warning: Given function compiled successfully or gave a verifier error.  (exit 0)

For anything that isn't a hard panic — miscompiles, wrong output, graceful codegen errors, test/run failures — use creduce with a clif-util predicate. Uses templates/predicate-creduce-clif.sh:

cp .agents/skills/reduce-test-cases/templates/predicate-creduce-clif.sh pred.sh
# edit pred.sh: set CLIF_UTIL=<abs path>, pick the sub-command, set the symptom
chmod +x pred.sh
mkdir -p /tmp/reduce && cp pred.sh be.clif /tmp/reduce && cd /tmp/reduce
creduce ./pred.sh be.clif

Verified: a 287-line, 46-function big-endian.clif reduced to one function that still hits the target codegen error:

function % (i64)->i128 { block0(v0 : i64) : v1 = load.i128 big v0 return v1 }

target/release/clif-util compile --target riscv64 be.reduced.clif
# Error: Unsupported feature: should be implemented in ISLE: inst = `v1 = load.i128 big v0`, type = `Some(types::I128)`

The predicate's sub-command is your choice: clif-util compile (codegen; target on the CLI), clif-util test (honors in-file ; run: / ; check: directives), clif-util run (host JIT), or clif-util interpret (backend-independent — good for miscompile differentials). creduce deletes the test/set/target header lines, so re-add them if you want the reduced file to run under clif-util test.

Miscompiles — differential predicates

A miscompile has no trap or panic to grep; "interesting" means the answer is wrong, which you detect by running two configs and checking they disagree. This was the most common real-world case. The reducer doesn't care — it's just a predicate shape — so templates/predicate-differential.sh drops into either the shrink ($1) or creduce (basename) contract. The load-bearing detail is the guard:

a=$("$WASMTIME" -O opt-level=0 --invoke f "$CAND" 2>/dev/null); ra=$?
b=$("$WASMTIME"                --invoke f "$CAND" 2>/dev/null); rb=$?   # default == opt-level=2
[ $ra -eq 0 ] && [ $rb -eq 0 ] && [ -n "$a" ] && [ -n "$b" ] && [ "$a" != "$b" ]

Both configs must run cleanly and produce output and differ. Drop the exit-0/non-empty guards and a candidate that merely became invalid or started trapping satisfies a != b for free, so the reducer converges on a broken program instead of a miscompile. Real fuzzer repros invoke a named export with arguments and need the right -W feature flags (-Wwide-arithmetic --invoke f "$CAND" 1 -1) — the defaults in the trap skeletons won't cut it. Confirm determinism first (Workflow step 1). For a .wast, prefer the self-checking form: encode the right answer as assert_return / assert_trap and make "interesting" == "opt-level=0 passes but default fails".

Manual reduction playbook

When the reducer stalls, over-shrinks, or the bug is a miscompile no predicate cleanly captures, reduce by hand. Reason from the bug's mechanism back to the minimal trigger, and re-verify after every cut:

• Bisect coarse first. Delete whole functions / modules / assertions, half at a time; keep the half that still reproduces. clif-util bugpoint's own strategy is a good mental model: remove instruction → replace with const → replace with trap → remove block → drop unused entities → merge blocks.
• Then bisect instructions inside the surviving function the same way.
• Neutralize data. Shrink memory/table sizes to the minimum; truncate or zero data/element segments; drop unused globals, types, imports, and exports.
• Simplify operands. Replace computed values with constants; collapse control flow (drop unused blocks/params); inline single-use locals.
• Re-verify each step with wasm-tools print / clif-util cat and by re-running the engine. If a cut kills the repro, revert it and try another.
• Keep a copy of the last-good reduction so a bad cut is always recoverable.

For a miscompile, the manual pass is where you win — automated reducers have no idea which bytes carry the bug, so they grind on structure while the real reduction is semantic. Hunt the diverging value directly:

• Neutralize inputs one at a time, don't delete them. Replace each local.get / global.get (or .clif value) feeding the wrong result with a constant, one at a time; the one whose neutralization makes the divergence vanish is carrying the corruption. Prefer this over deleting/truncating: cutting code changes what the optimizer does, so a cut that "fixes" the bug may have perturbed the pass rather than removed the cause — neutralization keeps the shape and isolates the value.
• Return the suspect directly. Once you've found the diverging value, return (local.get $culprit) (or make it the sole result) collapses the entire downstream computation in one edit — often dropping the bulk of the program at a stroke.
• Constant-fold the setup. Hash/mix stages, loop trip counts, and data segments that only produce the suspect value can be replaced by the constant they compute. Collapse a loop to the smallest trip count that still diverges.
• Then rebuild from scratch. Once the mechanism is clear, hand-writing a fresh minimal module from the understood cause beats continuing to chip at the reduced-but-messy one — and yields the legible artifact you actually want. In one real fuzzer case this took a 649-line module to ~22 lines (4%), where creduce plateaued at 43% and shrink barely moved it.

Race the reducers; don't serialize them. shrink, creduce, and a manual pass are cheap to run at once — start each in its own temp dir (creduce reduces in place and will collide otherwise), let creduce run in the background (it far exceeds a single tool-call timeout), and spin up a fork/subagent for the semantic reduction with full context on the bug's mechanism. Then take the smallest legible result. The manual fork usually wins the hard cases.

The goal is the smallest input where the bug's root cause is still obvious — not just small, but legible.

Gotchas

Battle scars from actually running these:

• wasm-reduce preserves behavior, not "exit 0". It keeps a candidate only if stdout and exit code match the original run — unlike shrink/creduce, which just want exit 0. Collapse "reproduces" to one canonical stdout marker and keep the exit code stable. And it reads the fixed --test file, not $1.
• wasm-reduce needs -f for unsupported proposals. It round-trips the module through Binaryen first and prints failed to read and write the binary if it can't. -f forces past it; if even that gets nowhere, drop to disassemble + creduce. Some proposals it simply can't represent (e.g. wide-arithmetic's i64.mul_wide_s) — for those, -f won't save it; skip to creduce.
• Environment variables don't reliably reach wasm-reduce's --command. Hardcode the absolute engine path in the command script (the skeleton does).
• creduce strips header directives. It deletes anything that doesn't change interestingness — the .clif test/set/target lines and the .wast ;;! feature = true header included. Fine for a repro whose predicate bakes the target/features into the command, but the reduced file won't run under cargo test --test wast or clif-util test unmodified. Re-add the header (or the matching -W / --target / --set flags) afterward.
• creduce runs in a fresh temp dir containing only the candidate, referenced by basename. Predicates must reference the candidate by basename and every tool by an absolute path. If you hand-roll a wrapper that cds, resolve the predicate to an absolute path before changing directory.
• Don't put set -e in a shrink/creduce predicate. A trapping module makes the engine exit non-zero; under set -e the out=$(engine …) line aborts the script before your grep runs, so the predicate never reports "interesting". Let the final grep be the exit status.
• bugpoint is panic-only, and only on the plain compile flow. It ignores verifier and graceful codegen errors ("compiled successfully or gave a verifier error"), and it only drives — a panic that fires only under a pass (e.g. ) is invisible to it. It cannot reduce a miscompile either. For any of these, use .

Troubleshooting

• Predicate fails on the original — it doesn't reproduce yet. Wrong trap string, wrong --invoke export, missing -W feature flag, or wrong engine build. Fix the predicate before reducing anything.
• wasm-reduce says "failed to read and write the binary" / "very unlikely reduction can succeed" — Binaryen can't parse a proposal in your module. Add -f; if that stalls too, disassemble + creduce.
• bugpoint prints "compiled successfully or gave a verifier error" (exit 0) — the input fails gracefully, it doesn't panic. Expected for expect-fail files. Switch to creduce.
• creduce finishes but the file is unchanged — the predicate never passed on any mutation. Usual causes: a relative tool path that broke in creduce's temp dir, a predicate that isn't chmod +x, or set -e swallowing a trapping exit. Re-test the predicate on the original and confirm absolute tool paths.