| name | crawfish-authoring-fixtures-evals |
| description | Author fixtures/ for craw test and gate on a baseline with craw eval — deterministic, mock-by-default, randomness carried in --seed. Load when adding test inputs or an eval gate to a Definition.
|
| user-invocable | false |
| allowed-tools | Read, Grep, Bash |
Authoring fixtures/ & evals
Derived from docs/specs/craw-code/authoring/fixtures-evals.md. Golden:
demo/craw-code-golden/fixtures/.
fixtures/ feeds craw test (eval-as-test): each fixtures/*.json is one input set the
Definition runs against. Evals gate against a saved baseline (craw eval --baseline).
The fixture shape
{"inputs": {"project": "acme", "ticket_body": "the login button does nothing"}}
A fixture is {"inputs": {...}, "expected": <optional>}. inputs maps each declared
Parameter name to a value. With no expected, a fixture passes when the run executes
cleanly; with expected, the Output value must match.
craw test definitions/<name> --fixtures fixtures
Determinism: mock-by-default
Fixtures run on the MockRuntime / the command profile's recorded transport, never a live
model, so the suite is deterministic and free. All randomness is carried in --seed; the
same seed replays byte-identically. Promote to --live only deliberately and always under
--budget.
Spine rule (determinism-mock-default): Fixtures run mock-by-default; randomness is
carried in --seed.
The eval baseline gate
craw eval --baseline <ref> compares per-metric scores against a saved baseline within a
--tolerance, and reports the deltas plus the cost band (total_usd / expected_usd /
worst_case_usd). Sinks fire only in eval mode; an optimize/search run never fires a
consequential action. Read retryable on a craw.error.v1 to decide retry-vs-stop — a
security rejection is retryable:false.
Fixtures never carry secrets. Reference credentials by name in .env.example, never inline.
