braintrust-size-eval-dataset

name

braintrust-size-eval-dataset

description

Calculate or audit eval sample sizes, minimum detectable effects, confidence interval precision, required repeated runs, and clean-trial counts for bounding rare failures. Use when a user asks how many eval cases, items, scenarios, runs, or safety trials are needed, whether an existing dataset is adequately powered, whether N examples can detect an X-point gain, or how many clean trials certify a low violation rate. Account for paired designs, clustering, target confidence, and practical effect size. Do not use for general dataset composition.

Size the dataset

Contract: references/interaction-contract.md. Calibration, templates, provenance: references/power-analysis.md.

Trigger

• "How many eval cases do we need?" / "Can 200 examples detect a 3-point gain?"
• "How many clean trials certify a low failure rate?"
• A null result that needs distinguishing from an underpowered one.

Do

1. Classify the goal first — it selects the method: detect a difference, estimate a rate, or bound a rare failure.
2. Take the minimum meaningful effect from the release gate's magnitude criterion, not from what the data shows. Ask only if genuinely absent.
3. Compute N with the method and anchors in reference.md, applying the design effect for clustered items — related variants are fewer items than they look.
4. Multiply through the whole matrix: items × arms × runs (K ≥ 3 for gating), and check the product against every quota and credit balance in the path before anyone starts.
5. Report a sensitivity range, not a single number, and close with the plain-language decision implication.

Avoid

• Do not return a sample size without stating the practical effect, baseline rate, risk tolerance, clustering, pairing, confidence, and power assumptions.
• Do not use the normal approximation near 0 or 1 — use Wilson, which is exactly the safety-rate case.
• Do not design content here.
• Do not let a cheap smoke run over a handful of items be quoted as evidence about which option is better. It verifies plumbing, nothing else.

Check

• Goal type identified; method named; inputs shown; every assumption listed with its source.
• Design effect applied for clustered or repeated-variant items.
• Sensitivity range alongside the point answer.
• A plain sentence stating what the resulting N can and cannot establish.

Risk

• Treating correlated variants as independent overstates precision more than any other single error in eval reporting.
• Normal approximations near zero produce impossible bounds and false safety assurances.
• Powering for the effect you hope for rather than the effect worth acting on guarantees an ambiguous result.
• An underpowered eval and a real null produce the identical sentence: "no measurable effect."

Braintrust

Shared mechanics: references/platform-mechanics.md. Pairing is what buys the sample-size reduction, and §3 makes it a platform decision rather than a later statistical one — a design sized for pairing silently reverts to the unpaired requirement if the arms were never pinned. So the N you compute here is only achievable if the experiment design honors that.

Planned N and achieved N diverge routinely, which makes the §2 read-safety checks a sizing concern and not just an analysis one: a truncated pull looks exactly like an underpowered run, and the two call for opposite responses. K runs means K recorded trials (§7); the worst run is what a latency or reliability gate reads.