eval-design-forensics

name	eval-design-forensics
description	Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE — adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing LLM judge is conflicted (same model/family as a compared system) or unvalidated (no human-agreement, no bias control) (HP-JUDGE-VALIDITY); a declared condition/metric is dropped or switched to favor the method, or 'best' is chosen with no held-out set (HP-SELECTIVE-REPORTING). Verdict-bearing at L0/L1 from the DESCRIBED protocol — NOT repo-gated like experiment-forensics; L2 only CONFIRMS against split/preprocessing/result files. A fresh cross-model reviewer (gpt-5.6-sol xhigh, read-only, fresh thread per pass) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Leakage and under-reporting are usually HONEST methodological errors — every finding describes a discrepancy to CHECK, never an accusation. An LLM generating GROUND-TRUTH labels is HP-FAKE-GT (experiment-forensics) — routed there, not here. Emits eval-design-forensics.findings.json; computes NO verdict. Detect-only. Triggers: "eval design audit", "evaluation validity", "train/test leakage", "data leakage", "is the score measuring generalization", "LLM judge bias", "is the judge validated", "selective reporting", "cherry-picked results", "评估设计审计", "评测有效性", "数据泄漏", "训练测试集泄漏", "裁判模型有没有验证", "选择性报告".
argument-hint	["paper-dir | claims.json"]
allowed-tools	Bash(*), Read, Write, Grep, Glob, WebSearch, WebFetch, mcp__codex__codex

Eval-Design Forensics — does the evaluation measure what the paper claims?

Audit evaluation-design and reporting validity for: $ARGUMENTS (requires claims.json from /evidence-ledger). Emit span-anchored eval-design-forensics.findings.json. This skill computes no verdict.

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is verdict-bearing input — it proposes the findings the deterministic adjudicator turns into the report. Re-firing it on a wall-clock timer adds no signal: its output changes only when the paper / ledger changes (or a repo arrives, raising the observability level), not with the clock. Schedule the external wait that precedes it — ledger built (or artifacts released → L2) → audit once. (Mirrors ARIS's external-cadence doctrine.)

Adapted from the ML-evaluation-methodology literature — the leakage taxonomy of Kapoor & Narayanan (2023), the LLM-as-judge validity work (MT-Bench self-enhancement, self-preference, position bias), and the "Show Your Work" / reproducibility-checklist reporting norms — reframed to audit a third party's evaluation. A favourite autoresearch shortcut is to report a number that is arithmetically self-consistent (family A), runs real code against a real ground truth (family D), and still does not measure what it claims: the protocol leaks, the load-bearing metric is a conflicted/unvalidated LLM judge, or the reporting quietly drops a declared condition. This skill is the constraint that asks "is this a valid measurement of the claim?", pointed at a submission, and it stays honest — leakage and under-reporting are usually honest methodological errors, so every finding is a discrepancy to clarify, never an accusation.

Why this exists

An optimizing pipeline (or rushed human) treats the evaluation as a number to make go up, not a measurement to keep valid. The repeatable failure modes — distinct from "is the number real?" (family D) — are:

• Leakage — the train/test boundary is broken (preprocessing fit before the split, no held-out set, duplicates across splits, a random split over time-ordered data, the same subject in both splits, an evaluated LLM that saw the benchmark in pretraining), so the reported score may not measure generalization at all. HP-EVAL-LEAKAGE
• Judge validity — the headline rests on an automatic LLM judge that is (the same model/family as a compared system, so its preference for that system is the "evidence") or (no human-agreement correlation, no position/length bias control).