| name | braintrust-define-eval-release-gate |
| description | Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine minimum meaningful improvement, statistical significance, regression rate, subgroup consistency, worst-run stability, all-attempts reliability, safety upper bounds, latency, and cost into an explicit ship-or-hold policy, to turn metrics into a CI gate, or to explain why a candidate failed a gate and what evidence would justify reconsideration. Do not use for general result analysis without a deployment decision. |
Define or apply a release gate
Contract: references/interaction-contract.md. Calibration, templates, provenance: references/gate-rows.md.
Trigger
- "Create release gates." / "Should this ship?" / "Why did this candidate fail?"
- A ship decision being made from a single average.
- A gate that exists but has no owner and blocks nothing.
Do
- Identify the shipping claim and the failures that cannot be traded away. Draft the table before
pinning thresholds needing product-owner input — mark those
Needs decision.
- Build a row per property the claim implicitly makes, using the row set in
reference.md:
magnitude, significance, consistency, stability, reliability, safety, latency/cost. A gate
should test every property, not just the average.
- Keep improvement metrics and guardrails structurally separate: guardrails are constraints,
never terms in a weighted sum.
- Require the run-integrity precondition — effective N and error accounting per arm. A gate on a
run with unexplained missing items is a gate-shaped artifact of whichever items survived.
- Enter red-team findings as existence constraints, never averaged scores, re-tested every
round rather than retired once passed.
- Emit both forms: machine-readable rules for CI, and a human-readable rationale per row. On
failure, name the failed rows, what each failure means, and the evidence that would justify
reconsideration.
Avoid
- Do not let an average improvement buy down a safety or reliability constraint.
- Do not gate on point estimates for near-zero rates; size n to the tolerance.
- Do not gate on a scorer not validated for gating, or on the set the candidate was tuned against.
- Do not write thresholds nobody owns; an unowned gate becomes a dashboard.
- Do not compute the statistics here.
Check
- Every row has a rule, a statistical basis, and a threshold with a stated source and owner.
- Guardrails expressed as constraints; reliability and safety rows use bounds and worst cases.
- Effective-N precondition explicit.
- Machine-readable rules match the human-readable rationale exactly.
- Failure interpretation written per row — each failure means something different.
Risk
- Thresholds without uncertainty, or without operational ownership, turn gates into decoration.
- A gate that passes says nothing about inputs the set does not cover; completeness is
unreachable, so gates pair with monitoring rather than replacing it.
- Gates tuned until the current candidate passes are not gates.
Braintrust
Encode each row as a scorer-plus-threshold check on the confirmatory experiment and block
promotion on any failing row. A gate producing a chart instead of a block is not a gate. Reuse
scorers by name and version (references/platform-mechanics.md §5); a gate wired to a
differently-named scorer silently stops testing what it was written for. Check each scorer's
fitness statement first — where a scorer is prohibited from gating, do not attach it at all,
since absence beats a warning nobody reads.
Rows needing a derived value rather than a native score: the paired difference with clustered CI;
per-category win rate and regression rate; the leave-one-out recomputation; pass^k across the K
runs; the Wilson or rule-of-three upper bound for safety; p95 per run then the worst run.
§8 lists the wiring failures these invite — the last two attract them most.
Gate against the experiment marked confirmatory, pinned to the held-out dataset version, on a
cached pull.
