| name | measurement-validation |
| description | Prove a measurement is sound BEFORE spending money optimizing against it. The gate that decides whether an Improve run is allowed to start, and whether its result is allowed to be believed. Refuse metrics whose noise exceeds the effect, that have no held-out split, or whose evidence is incomplete. |
Measurement Validation — earn the right to optimize
Optimization is only as trustworthy as the measurement under it. This skill is the gate on both ends: before a paid run (is this metric allowed to be optimized?) and after (is this result allowed to be believed?). It is the difference between an Improve button that is a product and one that is a slot machine.
Held by both the orchestrator (improve-conductor) and the builder (eval-architect). It is the shared honesty contract.
Invariant (non-negotiable)
- Refuse to optimize if CV(metric) > the target delta. If the run-to-run noise is bigger than the effect you're paying to move, the metric cannot validate the change — raise reps or fix the metric first. Do not tune against noise.
- Refuse to report a lift over INCOMPLETE or UNPAIRED evidence. Every held-out scenario must have a non-errored cell on both the baseline and the candidate side. Below the paired-n floor (≥3), the run is invalid, not a verdict. A lift computed over survivors is worse than no number.
Enforced by trustVerdicts from @tangle-network/agent-app/eval-campaign (rater-trust dimension: IRR floor + per-item rater spread, within-item never pooled, + survivor floor, each failure named in trustReasons), composed with the agent-eval statistical gates from @tangle-network/agent-eval/campaign: powerPreflight (before-gate for Invariant 1 — refuse to greenlight spend when the metric is underpowered vs the target delta), / (paired-bootstrap CI over the held-out split — the after-gate that would have caught the +47 by refusing an unpaired lift), and + (footprint-matched placebo gate — a held-out lift that does not survive came from added prompt/mount FOOTPRINT, not content, and must not be believed).
