| name | eval-integrity |
| description | MANDATORY whenever evaluating, validating, summarizing, or reporting ANY data — eval results, metrics, gates, benchmarks, experiment or test output, training runs, measurements — and before stating ANY conclusion about what the data shows. Counters overstating results. Forces validating a metric's MEANING (not just its value), tagging every number's provenance, inspecting the ground-truth artifact, applying equal skepticism to passes and fails, and keeping claims within the evidence. Apply it on every single data evaluation, positive or negative. |
Eval integrity — say only what the data shows
Run this on EVERY data evaluation, before stating any conclusion, headline, or recommendation. It exists because a real failure happened (see "The case"): a passing gate number was reported as "human-level" when the underlying behavior actually failed and fell into the void. Recomputing the metric from raw did NOT catch it — because the metric's value was right; its meaning was not what was claimed.
The failure mode this counters
Reporting a metric — especially a PASS or any positive result — as a bigger result than the raw evidence supports, by: checking a number's VALUE but not its MEANING; carrying a number from one run into a claim about another; applying less scrutiny to good news than to bad; and using words ("solved", "confirmed", "human-level", "milestone", "cleared") that outrun the evidence.
Mandatory checks — ALL of them, every eval, before you conclude
- What does the metric EXCLUDE? Open the caveats /
diagnostics_not_gated / *_not_gated / footnote / "what this does not measure" fields — not just the headline score. Write one explicit line: "This pass does NOT prove ___." A gate or proxy passing is never the same as the real goal being achieved.
- Provenance on every number. Tag each figure with the run/config/checkpoint/split/seed it came from (seeded vs unseeded, train vs val, which ckpt, which route). NEVER compare, average, or carry a number from one run into a claim about a different run.
- Ground truth over proxy. Inspect the actual artifact the metric summarizes — the trajectory, the raw rows, the produced output, the real behavior — not just the summary statistic. Measuring movement? Look at the path. Measuring text quality? Read the text. If you cannot see the underlying thing, you cannot conclude about it.
- Symmetric skepticism. Interrogate a PASS exactly as hard as you interrogate a FAIL. Ask, in writing, "What would make this look better than it actually is?" — then go check that specific thing. Decompose positives the way you decompose negatives.
- Words ≤ evidence. Do not write "solved / confirmed / works / cleared / human-level / milestone / first success / near-human" unless the evidence demonstrates that exact claim. Otherwise state precisely what happened: "passed metric X, which measures Y and excludes Z."
- Lead with the limitation. Put the caveat / what-it-doesn't-show BEFORE the headline. No celebratory framing ahead of the evidence.
Every eval report MUST include
- A one-line "What this does NOT prove:"
- Provenance tags on the key numbers (run / seed / split / ckpt / config).
- The claim stated at exactly the strength the evidence supports — no stronger.
Litmus test: if the owner/reviewer would have to say "show me the actual behavior" to catch an overstatement in your report, you skipped check #3 and you are not done.
Red flags — stop and recheck
- You're about to call a passing proxy "the goal," "human-level," or "a milestone."
- A number in your conclusion came from a different run than the one you're concluding about.
- You validated the headline metric but never opened the caveats / diagnostics block.
- You feel more excited (or more relieved) than the raw artifact actually justifies.
The case (why this skill exists)
Reported: "first full-gate clear — the bot bunnyhops dm3 hilljump at 83% human speed, near-human cadence (.25), human-level."
Truth: route% = 89.7 is arc-length progress to the launch ledge, not route completion; the bot fails the ~525 qu/s launch leap and falls into the void — cleared_launch: false, class: REACHED_LEDGE_NO_JUMP, both sitting in the very JSON that had been "validated." The .25 cadence was from the seeded trace; the actual unseeded gate run jumps .068, far below the human's .22.
Three failures, one per broken check above: (1) validated the metric's value, not its meaning [#1]; (2) mixed seeded and unseeded provenance [#2]; (3) words outran the evidence [#5]. It was caught only because the owner said "show me" — which forced pulling the x,y,z trajectory, the ground truth that check #3 demands first. Do the checks so the owner never has to.
