| name | braintrust-elicit-eval-criteria |
| description | Extract evaluation criteria out of domain experts and real user desires, and capture them as reusable evaluation assets before any labeling or scoring begins — construct facets, anchored exemplars, adversarial traps, scoring guidance, audit rules, and the signals that reveal what users actually want. Use when nobody can say what "good" means, when a rubric does not exist yet, when expert knowledge lives only in reviewers' heads, or when validating that an eval reflects user desires rather than team assumptions. Do not use to run the labeling workflow or to compare a scorer against finished labels. |
Elicit criteria from experts and users
Contract: references/interaction-contract.md. Calibration, templates, provenance: references/elicitation-guide.md.
Trigger
- No rubric exists, and review is happening on instinct.
- Experts disagree about what "good" means, or agree without being able to say why.
- An eval built entirely from team assumptions about what users want.
Do
- Gather both sources. Expert knowledge: what a qualified reviewer knows that the rubric
does not say. User desire: what users actually want, evidenced rather than assumed.
- Elicit from cases, not abstractions. Put a real trace in front of the expert, ask for a
judgment, then ask why — the reason is the criterion. Repeat until reasons stop being new.
- Capture each facet as an anchored exemplar pair: one output earning the top score, one
that just misses, and the distinguishing reason. A pair beats a paragraph of description.
- Harvest user desire from evidence that already exists — escalations, reopened tickets,
abandoned sessions, thumbs-down, support themes. Where none exists, say so and label the
criteria
Assumed.
- Capture what experts know about failure: the traps, the plausible-looking wrong answer,
what juniors get wrong. Highest-value output, and the part rubrics usually omit.
Avoid
- Do not ask experts to write a rubric from a blank page; elicit from cases.
- Do not substitute team intuition for user desire, or present it as evidence when you do.
- Do not run the labeling workflow or implement scoring here; this stage produces the criteria
those steps consume.
- Do not resolve genuine expert disagreement by averaging it — a contested criterion is a
finding about the construct.
Check
- Every facet has an anchored exemplar pair with the distinguishing reason.
- Failure knowledge captured explicitly, not just success criteria.
- Each criterion labeled
Confirmed (evidenced) or Assumed (team belief).
- User-desire signals named with their source, or their absence stated.
Risk
- Experts articulate what they can defend, not always what they use; the reason given may not be
the reason applied, which is why exemplars beat prose.
- A single expert encodes one house style as a universal standard.
- Available evidence of desire is biased toward users who complain; silent dissatisfaction leaves
no trace.
- Criteria elicited once ossify — they need the dataset's refresh cadence.
Braintrust
Run the elicitation session itself as human review over real traces with per-criterion
scoring and free-text notes — the notes are the output, because they carry the reason.
Promote reviewed traces into a versioned golden dataset so exemplars travel with the
criteria instead of living in a doc. Each facet then becomes its own scorer with a consistent
name; anchored pairs become the few-shot content of a rubric scorer. Flag traps in metadata
(trap: true) so they can be sliced out of headline numbers and reported separately. Treat a
criteria revision like a scorer change: bump the version, and do not compare results across the
boundary.
