| name | braintrust-write-eval-scorer |
| description | Design, implement, edit, or audit narrow eval scorers for LLM applications and agents, including deterministic checks, reference or final-state comparisons, trace and tool-call checks, and anchored LLM-as-judge rubrics. Use when translating one observable criterion into scoring logic, choosing between deterministic and judge-based scoring, repairing a vague rubric, or defining handling for refusals, errors, timeouts, and parse failures. Do not use to validate scorer agreement against human labels or to design the human review workflow. |
Write one eval scorer
Contract: references/interaction-contract.md. Calibration, templates, provenance: references/scorer-patterns.md.
Trigger
- "Write a scorer for this criterion." / "Should this be deterministic or a judge?"
- Turning a behavior-spec clause or evidence-map signal into a check.
- A rubric producing inconsistent scores, or a scorer bundling several qualities.
Do
- Name one criterion, and its output contract: a score (0–1, needs a numeric mapping) or
a classification (one label from a fixed set, needs no-match behavior). If the request
combines several criteria, split them before writing any code or rubric.
- Match the method to the evidence — and let stakes override convenience. Objective →
deterministic. Subjective → anchored rubric or human. A checkable outcome on a
safety-critical path still needs sampled human review, because what is most likely wrong is
the check's scope.
- Set two independent axes and keep them apart. Input scope — span, trace, or group; how
much one evaluator call sees, default trace. Reporting level — per-item scoring localizes
failures, aggregate detects regressions. Most criteria need both reporting levels.
- For rubrics, write criteria as anchored examples, not descriptions, each scored
separately, requiring structured output that carries the evidence used. Have the model emit
, and map classes to scores outside the model.
