메뉴
Author fixtures/ for craw test and gate on a baseline with craw eval — deterministic, mock-by-default, randomness carried in --seed. Load when adding test inputs or an eval gate to a Definition.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Author definition.py — typed inputs/outputs (Parameter, static vs fluid) and the team shape (lead/coordination). Load when writing or editing a Definition's typed boundary. Static is the deliberate consequential choice; fluid is the untrusted default.
Author instructions.md and agents/*.md — front-matter (role, delegates_to, tools, model) over a markdown prompt. Load when writing a lead prompt or a subagent. Fluid inputs are data the prompt analyzes, never instructions to obey.
Attach knowledge with Wiki / with_context / SkillRef — summoned content is tainted data, never instructions, and is pinned by content hash. Load when giving an agent a rubric, corpus, or reference it should consult.
Author mcp/*.py — MCPConnection instances; auth is a secret reference (env-var name), never inline. Load when adding an external tool server. A new connection re-enters the consent gate (craw code grant).
Optimize a Crawfish component — set a baseline (craw eval), search knobs (craw tune), iterate (craw refine), self-version (craw learn). Train vs eval mode, budget-bounded. Load when improving a component's quality or cost.
Author policies/*.py (module-level Policy instances — consequential static config) and skills/*.md (bundled skills). Load when adding a guardrail/routing/permission policy or a bundled skill. A policy is static-only.
| name | crawfish-authoring-fixtures-evals |
| description | Author fixtures/ for craw test and gate on a baseline with craw eval — deterministic, mock-by-default, randomness carried in --seed. Load when adding test inputs or an eval gate to a Definition. |
| user-invocable | false |
| allowed-tools | Read, Grep, Bash |
fixtures/ & evalsDerived from docs/specs/craw-code/authoring/fixtures-evals.md. Golden:
demo/craw-code-golden/fixtures/.
fixtures/ feeds craw test (eval-as-test): each fixtures/*.json is one input set the
Definition runs against. Evals gate against a saved baseline (craw eval --baseline).
{"inputs": {"project": "acme", "ticket_body": "the login button does nothing"}}
A fixture is {"inputs": {...}, "expected": <optional>}. inputs maps each declared
Parameter name to a value. With no expected, a fixture passes when the run executes
cleanly; with expected, the Output value must match.
craw test definitions/<name> --fixtures fixtures
Fixtures run on the MockRuntime / the command profile's recorded transport, never a live
model, so the suite is deterministic and free. All randomness is carried in --seed; the
same seed replays byte-identically. Promote to --live only deliberately and always under
--budget.
Spine rule (determinism-mock-default): Fixtures run mock-by-default; randomness is carried in --seed.
craw eval --baseline <ref> compares per-metric scores against a saved baseline within a
--tolerance, and reports the deltas plus the cost band (total_usd / expected_usd /
worst_case_usd). Sinks fire only in eval mode; an optimize/search run never fires a
consequential action. Read retryable on a craw.error.v1 to decide retry-vs-stop — a
security rejection is retryable:false.
Fixtures never carry secrets. Reference credentials by name in .env.example, never inline.