Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE — adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing LLM judge is conflicted (same model/family as a compared system) or unvalidated (no human-agreement, no bias control) (HP-JUDGE-VALIDITY); a declared condition/metric is dropped or switched to favor the method, or 'best' is chosen with no held-out set (HP-SELECTIVE-REPORTING). Verdict-bearing at L0/L1 from the DESCRIBED protocol — NOT repo-gated like experiment-forensics; L2 only CONFIRMS against split/preprocessing/resu
Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE — adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing LLM judge is conflicted (same model/family as a compared system) or unvalidated (no human-agreement, no bias control) (HP-JUDGE-VALIDITY); a declared condition/metric is dropped or switched to favor the method, or 'best' is chosen with no held-out set (HP-SELECTIVE-REPORTING). Verdict-bearing at L0/L1 from the DESCRIBED protocol — NOT repo-gated like experiment-forensics; L2 only CONFIRMS against split/preprocessing/result files. A fresh cross-model reviewer (gpt-5.6-sol xhigh, read-only, fresh thread per pass) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Leakage and under-reporting are usually HONEST methodological errors — every finding describes a discrepancy to CHECK, never an accusation. An LLM generating GROUND-TRUTH labels is HP-FAKE-GT (experiment-forensics) — routed there, not here. Emits eval-design-forensics.findings.json; computes NO verdict. Detect-only. Triggers: "eval design audit", "evaluation validity", "train/test leakage", "data leakage", "is the score measuring generalization", "LLM judge bias", "is the judge validated", "selective reporting", "cherry-picked results", "评估设计审计", "评测有效性", "数据泄漏", "训练测试集泄漏", "裁判模型有没有验证", "选择性报告".
Eval-Design Forensics — does the evaluation measure what the paper claims?
Audit evaluation-design and reporting validity for: $ARGUMENTS (requires
claims.json from /evidence-ledger). Emit span-anchored
eval-design-forensics.findings.json. This skill computes no verdict.
🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is
verdict-bearing input — it proposes the findings the deterministic adjudicator
turns into the report. Re-firing it on a wall-clock timer adds no signal: its
output changes only when the paper / ledger changes (or a repo arrives,
raising the observability level), not with the clock. Schedule the external wait
that precedes it — ledger built (or artifacts released → L2) → audit once.
(Mirrors ARIS's external-cadence doctrine.)
Adapted from the ML-evaluation-methodology literature — the leakage taxonomy of
Kapoor & Narayanan (2023), the LLM-as-judge validity work (MT-Bench
self-enhancement, self-preference, position bias), and the "Show Your Work" /
reproducibility-checklist reporting norms — reframed to audit a third party's
evaluation. A favourite autoresearch shortcut is to report a number that is
arithmetically self-consistent (family A), runs real code against a real ground
truth (family D), and still does not measure what it claims: the protocol
leaks, the load-bearing metric is a conflicted/unvalidated LLM judge, or the
reporting quietly drops a declared condition. This skill is the constraint that
asks "is this a valid measurement of the claim?", pointed at a submission, and
it stays honest — leakage and under-reporting are usually honest methodological
errors, so every finding is a discrepancy to clarify, never an accusation.
Why this exists
An optimizing pipeline (or rushed human) treats the evaluation as a number to make
go up, not a measurement to keep valid. The repeatable failure modes — distinct
from "is the number real?" (family D) — are:
Leakage — the train/test boundary is broken (preprocessing fit before the
split, no held-out set, duplicates across splits, a random split over time-ordered
data, the same subject in both splits, an evaluated LLM that saw the benchmark in
pretraining), so the reported score may not measure generalization at all.
HP-EVAL-LEAKAGE
Judge validity — the headline rests on an automatic LLM judge that is
(the same model/family as a compared system, so its preference for
that system is the "evidence") or (no human-agreement correlation,
no position/length bias control).
conflicted
unvalidated
HP-JUDGE-VALIDITY
Selective reporting — a dataset / baseline / metric / seed-count the setup
explicitly declares is dropped from the results, the metric is switched
across tables to keep the method ahead, or "we report the best run/prompt/
checkpoint" with no held-out selection set (selecting on the test set).
HP-SELECTIVE-REPORTING
None of these is inherently misconduct — they are what an agent does when nothing
forces a valid evaluation. The stated version is decidable at L0/L1 from
the described protocol; the verified version (real split/preprocessing/result
files) deepens at L2. What this skill will not do is guess: three leakage
subtypes are undecidable even with the repo and are handed off as
needs_external_check, not invented (see below).
Core principle
Ledger-anchored, span-verified, reviewer≠adjudicator, honest about what it cannot
settle. Four properties:
Anchor to a PAPER claim. Every above-info finding cites a ledger claim_id
and quotes a verbatim span of that claim's text_span
(references/integrity-forensics-contract.md rules 1–2). The leak/judge/reporting
tell lives in the protocol / setup-description — usually method and scope
claims, with comparison / number for the judge metric and
caption / table_cell / baseline for reporting. The anchor is whichever paper
claim the finding undermines; a split-file file:line, a config, or a leaderboard
date is forensic context for the description, never the anchor.
The executor assembles facts; the reviewer judges. At L2 the executor gathers
mechanical split/preprocessing/judge/result facts (grep/hash — listing what
exists is a fact, not a judgment) and may record one public-record date fact
(a benchmark's release vs a model's cutoff, for the contamination FP guard). It
passes paths + the ledger + those facts + the checklist to the reviewer and
never pre-declares "this leaks" (references/reviewer-independence.md). The model
proposes; tools/adjudicate_findings.pydecides. This skill computes no
verdict.
Undecidable leakage subtypes → hand off, don't guess. An illegitimate-proxy
feature, sampling bias in the test set, and pretraining/benchmark
contamination are domain / black-box judgments not settleable from the PDF or
the repo. Emit verdict_local: needs_external_check + requires_external_check: true (contract rule 6); name the external methods a domain check would use —
exchangeability (Oren 2023), Min-K% Prob (Shi 2023), Time-Travel (Golchin 2023),
BIG-bench canary strings — and never run them.
Verdict-bearing at L0/L1; observability still caps the L2-confirm. Unlike
experiment-forensics (no eval code at L0/L1 ⇒ info-only), a stated-tell here
is decided from the described protocol and emits
observability_level_required: 0. The L2 confirmation of the same leak/
omission is a separate finding with observability_level_required: 2 that
is marked as needing L2 on a PDF-only run (references/observability-levels.md). So a
PDF-only run keeps the stated-tell as a flag and the verification as an info
"confirm-at-L2" pointer — never the reverse.
How this differs from the other auditors (route correctly)
This skill is the L0/L1-stated / L2-verified sibling of
baseline-comparison-audit and proof-derivation-forensics (both verdict-bearing
without a repo) — not the L2-only experiment-forensics.
Auditor
Question it answers
Level
eval-design-forensics (this)
Is the evaluation a VALID measurement of the claim, and is the reporting complete? (train/test leakage, conflicted/unvalidated LLM judge, declared-but-unreported / metric-switch / best-without-held-out)
L0/L1 stated · L2 verified
experiment-forensics
Are the reported numbers what the code computes? (fake/derived GT, self-norm, phantom, dead metric)
L2
consistency-audit
Does the paper contradict ITSELF / described method = evaluated method? (owns HP-AGG-DRIFT, HP-APPENDIX-CONTRA, text-only HP-SCOPE-INFLATE)
L0
baseline-comparison-audit
Are the right baselines present, fairly tuned, and is "SOTA" earned? (owns HP-MISSING-BASELINE, HP-SIG-OVERLAP)
Strongest evidence-bound rejection memo (no verdict weight)
any
Do NOT raise here (hand off instead):
An LLM generating the GROUND-TRUTH labels/targets (not judging outputs) →
experiment-forensicsHP-FAKE-GT (L2). The clean split: a judge whose
preference IS the reported metric is HP-JUDGE-VALIDITY (here, L0/L1 stated);
a model that fabricates the reference the metric is computed against is
HP-FAKE-GT (there, needs the code, L2). When unsure which, prefer the L2 route
and set needs_external_check.
best-reported-as-mean (the aggregation lies) → consistency-auditHP-AGG-DRIFT; thin overall scope with no comparison → consistency-auditHP-SCOPE-INFLATE; appendix-vs-main disagreement on the same quantity →
consistency-auditHP-APPENDIX-CONTRA.
A never-mentioned expected SOTA baseline (completeness) →
baseline-comparison-auditHP-MISSING-BASELINE; a "consistently/across-the-board"
comparison resting on one dataset → baseline-comparison-audit's single-dataset
HP-SIG-OVERLAP.
Whether a reported number matches the code (fake GT, self-norm, phantom) →
experiment-forensics (L2); whether a cited paper exists / is used in context
→ citation-forensics; surface / AI-flavor → presentation-signals.
HP-SELECTIVE-REPORTING is scoped to declared-but-unreported / cherry-picked-
among-shown — the gap between what the setup promised and what the tables
deliver. It never re-emits the four patterns above.
The Kapoor & Narayanan leakage taxonomy (adopted — paraphrased)
HP-EVAL-LEAKAGE adopts the eight leakage types in three categories of Kapoor &
Narayanan (2023), paraphrased. The reviewer maps each finding to one type and records
it in the description.
K&N category (the leakage TYPE)
The tell (subtypes)
This repo's observability
Common false positive
L1 — no clean train/test separation
(a) no held-out test set at all; (b) preprocessing (scaling / imputation / resampling) fit on all data before the split; (c) feature selection fit before the split; (d) duplicate / near-duplicate records across splits
L0 stated / L2 verified
a transductive / semi-supervised design where overlap is intended and declared; preprocessing fit on train only, then applied to test (the correct pattern)
L2 — illegitimate (proxy) feature
a feature that stands in for the target, or would be unavailable at prediction time
needs_external_check (domain judgment)
a "proxy-looking" feature that is genuinely available at prediction time
L3 — test set not from the distribution of interest
(a) temporal leakage (random split over time-ordered data / training on the future); (b) non-independence (same subject / patient / group in both splits); (c) sampling bias in the test set
the evaluated model may have seen the public benchmark during pretraining
needs_external_check (black-box) — name Oren 2023 (exchangeability), Shi 2023 (Min-K%), Golchin 2023 (Time-Travel), BIG-bench canary; never run them
a benchmark released after the model's training cutoff, or a corpus documented to exclude it
⚠️ Two scales — do not conflate them. K&N's L1 / L2 / L3 are leakage-type
labels (severity-ordered categories of leak). This repo's L0 / L1 / L2 are
observability levels (what you can see: PDF / +source / +repo). They are
orthogonal. A K&N-L1 preprocessing leak that is stated in the protocol is
decidable at observability-L0. Every finding carries both: the K&N type in
description, the observability in observability_level_required.
Constants & Reviewer Calling Convention
REVIEWER_MODEL = gpt-5.6-sol # different family from executor (Claude)
REVIEWER_REASONING = xhigh # always; effort never lowers reviewer quality
REVIEWER_SANDBOX = read-only # detect-only; never mutate the paper
REVIEWER_CWD = <paper-dir> # so it can read claims.json + the protocol/source directly
THREAD_POLICY = fresh mcp__codex__codex per PASS (and per entry on fan-out);
NEVER mcp__codex__codex-reply across passes/entries (the bias guard)
TAXONOMY_VERSION = 0.5 # references/hack-pattern-taxonomy.md (family H)
LEAKAGE_TAXONOMY = Kapoor & Narayanan 2023 # 8 types / 3 categories — adopted, paraphrased
PATTERNS_OWNED / ALLOWED = HP-EVAL-LEAKAGE, HP-JUDGE-VALIDITY, HP-SELECTIVE-REPORTING # emit ONLY these
DIMENSION = evaluation # SKILL_TO_DIMENSION["eval-design-forensics"]
FINDINGS_FILE = eval-design-forensics.findings.json
FINDING_ID_NAMESPACE = ED### # distinct from F###/NUM###/HL### (consistency), EF### (experiment), BC### (baseline), PD### (proof)
VERDICT_BEARING_AT = L0/L1 (stated-tells) # NOT repo-gated; L2 only CONFIRMS
TRACE_POLICY = forensic (never silently dropped)
TRACE_DIR = .aris/traces/eval-design-forensics/<YYYY-MM-DD>_run<NN>/
Executor (Claude) builds none of the judgment: it locates the ledger, extracts
the evaluation surface (protocol / judge / declared-condition claims), at L2 gathers
mechanical split/preprocessing/judge/result facts (grep/hash) and at most one
public-record date fact for the contamination guard, passes **paths + the ledger
those facts + the checklist** to the reviewer, validates the reviewer's spans, and
writes the findings file. It never summarizes the paper, pre-judges "this leaks", or
leaks an opinion into the prompt (reviewer-independence.md). Passing a public
release date (with its source) is the same allowed division citation-forensics
(canonical metadata) and baseline-comparison-audit (leaderboard dates) use —
reference facts, not hunches.
Reviewer (codex / gpt-5.6-sol) reads claims.json and the source (and, at L2, the
split/preprocessing/judge/result files) directly from its cwd, decides which
evaluations leak / rest on a conflicted-or-unvalidated judge / under-report, applies
the known false-positive cases, and self-reports false_positive_risk. It is the
evidence-extractor, not the judge.
Fresh thread per pass. Leakage (Step 3) and judge-validity + selective-reporting
(Step 4) are separate freshmcp__codex__codex calls. On — effort: max or many
evaluation tracks, fan each track entry out into its own fresh call — never
codex-reply carrying one entry's conclusion into another (the bias guard).
codex-reply is intentionally absent from allowed-tools.
Step 0 — Preconditions: locate the ledger, read the run level
The ledger is the only structure this skill reasons over. Resolve it and read the
observability level L and paper_id it was built at (each Bash block is
self-contained — shell state does not persist, so re-derive paths every block):
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
# $ARGUMENTS is a paper-dir OR a claims.json path:
LEDGER="$ARGUMENTS"; [ -d "$LEDGER" ] && LEDGER="$LEDGER/claims.json"
# Only the NO-ARGUMENT case defaults to the CWD ledger. An EXPLICIT argument that
# resolves to a missing claims.json must NOT silently fall back to $(pwd) — that
# could audit the wrong paper; let the NO_LEDGER check below fire instead.
[ -z "$ARGUMENTS" ] && LEDGER="$(pwd)/claims.json"
python3 - "$LEDGER" <<'PY'
import json, sys, os, collections
p = sys.argv[1]
if not os.path.isfile(p):
sys.exit("NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST "
"(it writes artifact_manifest.json + claims.json).")
d = json.load(open(p, encoding="utf-8"))
claims = d.get("claims", [])
by = collections.Counter(c.get("type") for c in claims)
print("LEDGER =", os.path.abspath(p))
print("PAPER_DIR =", os.path.dirname(os.path.abspath(p)) or ".")
print("PAPER_ID =", d.get("paper_id", "?"))
print("RUN_LEVEL_L =", d.get("observability_level", 0))
print("CLAIMS =", len(claims), dict(by))
# applicability signal — method/scope carry the protocol; comparison/number the judge
# metric; caption/table_cell/baseline the reported conditions:
rel = sum(by.get(t, 0) for t in ("method", "scope", "comparison", "number", "table_cell", "caption", "baseline"))
print("APPLICABLE =", "yes" if rel else "low (no protocol/scope/comparison/table claims)")
PY
Failure handling. If NO_LEDGER is printed, stop and tell the user to run
/evidence-ledger first — this skill never re-reads the raw PDF and invents its own
structure (contract rule 1). Carry L, PAPER_ID, and the absolute LEDGER /
PAPER_DIR into every step below.
Step 1 — Extract the evaluation surface from the ledger (decide whether to run)
Pull the claims this audit reasons over — the protocol (leakage anchors), the
judge (validity anchors), and the declared conditions (reporting anchors) —
and decide if there is anything to audit. This is a mechanical surface scan; the
reviewer decides validity:
LEDGER="<abs path to claims.json from Step 0>"
python3 - "$LEDGER" <<'PY'
import json, re, sys, collections
d = json.load(open(sys.argv[1], encoding="utf-8"))
claims = d.get("claims", [])
LEAK = re.compile(r"\b(train(?:ing)?[\s/_-]*(?:and[\s/_-]*)?test|train[\s/_-]*test|split|held?[\s-]*out|"
r"cross[\s-]*validat|k-?fold|preprocess|standardi[sz]|normali[sz]|imput|"
r"resampl|oversampl|smote|feature[\s-]*select|leak|duplicat|de-?dup|"
r"temporal|time[\s-]*(?:series|order)|contaminat|pre-?train|data\s+split)\b", re.I)
JUDGE = re.compile(r"\b(LLM[-\s]*as[-\s]*a?[-\s]*judge|as\s+(?:a\s+)?judge|automatic(?:ally)?\s+(?:judg|evaluat|scor|rat)|"
r"GPT-?4o?|GPT-?3\.5|Claude|Gemini|win[\s-]*rate|pairwise|preference|"
r"rated\s+by|scored\s+by|judged\s+by|LLM\s+(?:judge|evaluator|grader))\b", re.I)
DECLARE = re.compile(r"\b(we\s+(?:evaluate|report|test|measure|use)|datasets?|benchmarks?|metrics?|"
r"seeds?|over\s+\d+\s+(?:seed|run)|best\s+(?:checkpoint|prompt|run|model|epoch)|"
r"five|four|three|\{[^}]*\})\b", re.I)
leak_a, judge_a, report_a = [], [], []
for c in claims:
t, span = c.get("type"), c.get("text_span", "")
sec = c.get("location", {}).get("section", "?")
if t in ("method", "scope") and LEAK.search(span):
leak_a.append((c["claim_id"], t, sec, span[:160]))
if t in ("comparison", "scope", "method", "number") and JUDGE.search(span):
judge_a.append((c["claim_id"], t, sec, span[:160]))
if t in ("scope", "method", "caption", "table_cell", "baseline") and DECLARE.search(span):
report_a.append((c["claim_id"], t, sec, span[:160]))
print(f"LEAKAGE anchors: {len(leak_a)} JUDGE anchors: {len(judge_a)} REPORTING anchors: {len(report_a)}")
for tag, rows in (("leak", leak_a), ("judge", judge_a), ("report", report_a)):
for cid, t, sec, sp in rows[:30]:
print(f" [{tag}:{t}] {cid} [{sec}] {sp!r}")
print("APPLICABLE =", "yes" if (leak_a or judge_a or report_a) else "no -> write [] and stop")
PY
Branch. If APPLICABLE = no (no protocol / judge / declared-condition claims),
this skill is not applicable: write an empty eval-design-forensics.findings.json
([]), record a one-line NOT_APPLICABLE reason in the trace (Step 7), and stop.
Silent skip is forbidden — the orchestrator globs *.findings.json and expects
the file to exist. Otherwise record the three anchor lists for the prompts. A purely
mechanical grep of the source helps surface the protocol/table language (do not
judge validity here — that is the reviewer's job):
LEDGER="<abs path to claims.json from Step 0>"
grep -rInE 'train[ /_-]*test|split|held[ -]*out|cross[ -]*valid|preprocess|standardi|normali|impute|leak|duplicat|as a judge|win rate|pairwise|we (evaluate|report) on|best (checkpoint|prompt|run)' \
"$(dirname "$LEDGER")" --include='*.tex' --include='*.txt' 2>/dev/null | head -60
Create the run's trace dir now — its first use is the facts file written just below,
so it must exist before Step 7. Reuse this exact RUNDIR in Steps 3–7 (do not
create a second one):
DATE=$(date +%Y-%m-%d); N=1
while [ -d ".aris/traces/eval-design-forensics/${DATE}_run$(printf %02d $N)" ]; do N=$((N+1)); done
RUNDIR=".aris/traces/eval-design-forensics/${DATE}_run$(printf %02d $N)"; mkdir -p "$RUNDIR"
echo "RUNDIR = $RUNDIR" # carry this exact path forward (shell state does not persist)
At L2 only (repo + result files present), gather raw, uninterpreted facts — paths
grep/hash only, the same executor/reviewer division as experiment-forensics. Skip
this block at L0/L1 (there is no code to read — the reviewer block gets "L<2: ..."):
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
PAPER_DIR="<abs PAPER_DIR from Step 0>"; L="<L from Step 0>"; RUNDIR="<the RUNDIR above>"
if [ "$L" = "2" ]; then
# (a) split / preprocessing / dedup ordering FACTS (the K&N-L1 / L3 tells) -> leakage_grep.txt
grep -rInE 'train_test_split|StratifiedKFold|KFold|GroupKFold|TimeSeriesSplit|\.split\(|train/val|holdout|hold-out|'\
'StandardScaler|MinMaxScaler|fit_transform|\.fit\(|SimpleImputer|impute|SMOTE|resample|oversampl|'\
'SelectKBest|feature_select|drop_duplicates|duplicated\(|dedup|shuffle=True|random_state' \
"$PAPER_DIR" --include='*.py' --include='*.ipynb' 2>/dev/null | head -80 > "$RUNDIR/leakage_grep.txt"
# (b) LLM-judge calling code FACTS -> judge_grep.txt
grep -rInE 'as_judge|llm_judge|judge_model|gpt-?4|gpt-?3\.5|claude|gemini|openai|anthropic|'\
'pairwise|win_rate|preference|rate_response|score_response|annotate' \
"$PAPER_DIR" --include='*.py' --include='*.ipynb' --include='*.yaml' --include='*.json' 2>/dev/null | head -60 > "$RUNDIR/judge_grep.txt"
# (c) which DECLARED conditions actually produced result files (the selective-reporting L2 confirm) -> reporting_grep.txt
{ find "$PAPER_DIR/results" "$PAPER_DIR/outputs" "$PAPER_DIR/logs" -type f \( -name '*.json' -o -name '*.csv' \) 2>/dev/null | sort | head -60
echo "## metric/dataset keys present in result files:"
grep -rIhoE '"(dataset|benchmark|metric|seed|split|task)"[^,}]{0,40}' \
"$PAPER_DIR/results" "$PAPER_DIR/outputs" "$PAPER_DIR/logs" 2>/dev/null | sort -u | head -60; } > "$RUNDIR/reporting_grep.txt"
# (d) reproducibility anchors: hash each discovered file (space-safe; tolerant of zero matches)
{ grep -rIlE 'split|scaler|judge|metric' "$PAPER_DIR" --include='*.py' 2>/dev/null | head -n 20
find "$PAPER_DIR" -maxdepth 3 -path '*results*' -name '*.json' 2>/dev/null | head -n 20; } \
| while IFS= read -r ff; do shasum -a 256 "$ff" 2>/dev/null; done > "$RUNDIR/hashes.txt"
echo "L2 facts -> $RUNDIR/{leakage_grep,judge_grep,reporting_grep,hashes}.txt"
else
echo "L<2: stated-tell pass only (no split/judge/result files to read)."
fi
Optional contamination date fact (the FP guard, not a detector). If a benchmark is
named and an evaluated model's training cutoff is knowable, you MAY record one
public-record date fact — WebSearch/WebFetch for " release date" and
" training cutoff" — and write it (URL + access date) to
$RUNDIR/contamination_dates.json. This is a fact that suppresses a false
contamination flag (benchmark released after the cutoff → legitimate); it is never
a contamination detector. The skill never runs Min-K% / exchangeability / Time-Travel.
Failure handling. No network → skip the date fact; the reviewer treats
contamination as needs_external_check regardless. Empty L2 greps (a thin repo) → pass
"L2 but no split/judge/result files found" to the reviewer so it does not invent a leak.
Open a freshmcp__codex__codex thread (Reviewer Calling Convention). The reviewer
reads claims.json from its cwd for the described protocol and, at L2, the split/
preprocessing files; every finding anchors to a ledger claim_id. Send EXACTLY (fill
every [ ... ]):
mcp__codex__codex:
model: gpt-5.6-sol
config: {"model_reasoning_effort": "xhigh"}
sandbox: read-only
cwd: <absolute PAPER_DIR from Step 0>
prompt: |
You are a train/test-LEAKAGE forensics reviewer. You judge ONE thing: given the
EVALUATION PROTOCOL this paper describes (and, at L2, the split/preprocessing code
it ships), is there a leak that means the reported score may NOT measure
generalization? You do NOT judge whether numbers are real (that needs the code and
is another auditor) and you do NOT grade the paper. Describe a discrepancy to
CHECK/CLARIFY, never an accusation — leakage is most often an HONEST methodological
error. Hand off what you cannot ground.
INPUTS (in your working directory — read them directly):
- claims.json — the evidence ledger. The PROTOCOL/SPLIT/PREPROCESSING language
lives in type:"method" and type:"scope" claims; the ONLY structure you reason
over. Each claim = {claim_id, type, text_span (VERBATIM), location, value?}. You
MAY re-open a source file (and, at L2, the split/preprocessing code) to confirm a
span is real, but you may NOT introduce a claim not in the ledger.
LEAKAGE ANCHOR TARGETS (protocol/scope claims — claim_id + verbatim text_span):
[paste the LEAKAGE anchors from Step 1]
L2 SPLIT/PREPROCESSING FACTS (raw grep/hash — uninterpreted; empty if L<2):
[inline RUNDIR/leakage_grep.txt + RUNDIR/hashes.txt, or "L<2: no repo/code available"]
CONTAMINATION DATE FACT (public record, if gathered — GIVEN data, not a verdict):
[inline RUNDIR/contamination_dates.json, or "none gathered"]
RUN OBSERVABILITY LEVEL L = <L from Step 0>.
THE LEAKAGE TAXONOMY you map onto (Kapoor & Narayanan 2023 — 8 types / 3 categories;
paraphrase the type in your description). ⚠️ K&N's L1/L2/L3 below are leakage-TYPE
labels — they are NOT this repo's observability L0/L1/L2 (what you can SEE). Set
observability_level_required from what you can SEE, and name the K&N type in text:
- K&N L1 (no clean separation): (a) no held-out test; (b) preprocessing fit on ALL
data BEFORE the split; (c) feature selection before the split; (d) duplicate /
near-duplicate rows across splits. -> observability 0 (stated) / 2 (verified).
- K&N L2 (illegitimate/proxy feature): a feature that proxies the target or is
unavailable at prediction time. -> needs_external_check (domain judgment).
- K&N L3 (test not from the distribution of interest): (a) temporal leakage (random
split over time-ordered data / training on the future); (b) non-independence
(same subject/patient/group in both splits) -> observability 0 / 2; (c) sampling
bias in the test set -> needs_external_check.
- Pretraining/benchmark CONTAMINATION of an evaluated LLM (the model may have seen
the public benchmark in pretraining) -> needs_external_check. You may NAME the
external methods a human would use (exchangeability, Oren 2023; Min-K% Prob, Shi
2023; Time-Travel, Golchin 2023; BIG-bench canary) but you do NOT run them.
HARD RULES (a finding that breaks any of these is worthless):
1. ANCHOR. Every finding above "info" MUST carry >=1 evidence {claim_id, span} where
claim_id EXISTS in claims.json and span is a VERBATIM whitespace-normalized
SUBSTRING of THAT claim's text_span (no paraphrase). The anchor is the protocol/
split/preprocessing claim the leak undermines; a code file:line goes in
`description`, never as the anchor. ALWAYS anchor — even a needs_external_check
finding — so it stays navigable.
2. DISCREPANCY, NOT ACCUSATION. Never "reject", "fabricated", "the authors cheated".
3. OBSERVABILITY. A leak visible in the DESCRIBED protocol => observability_level_
required = 0 (this is verdict-bearing from a PDF). A leak only CONFIRMABLE from the
split/preprocessing files => a SEPARATE finding with observability_level_required = 2
(an L0/L1 run reports it as needing L2 — that is correct). NEVER put the stated tell at 2.
4. HAND OFF THE 3 UNDECIDABLE SUBTYPES. illegitimate-proxy feature, sampling bias, and
pretraining/benchmark contamination are NOT decidable from the PDF or the repo:
set verdict_local "needs_external_check", requires_external_check true, severity
"info", false_positive_risk "high", and name what a human should check.
5. HONEST FP. A declared transductive/semi-supervised overlap, a standard fixed
benchmark split, preprocessing fit on TRAIN ONLY then applied to test, a correctly
time-respecting split, a benchmark released AFTER the model's cutoff — these LOOK
like leaks but are legitimate. Say so; if the protocol is under-described, prefer
needs_external_check over a flag.
6. pattern_id MUST be HP-EVAL-LEAKAGE.
SEVERITY DECISION (HP-EVAL-LEAKAGE):
- unambiguous stated leak (e.g. "standardize all features, then split") that
plausibly invalidates the HEADLINE generalization claim -> "critical", FP "low",
observability 0, requires_external_check false.
- a leak affecting a NON-headline result, or a stated tell that needs the code to
confirm -> "major" (FP "medium"); the L2 confirmation is a separate observability-2
finding.
- proxy / sampling-bias / contamination -> "info", needs_external_check, FP "high".
OUTPUT: a single JSON array and NOTHING ELSE (no prose, no code fence). Each element
conforms to schemas/finding.schema.json:
{"finding_id":"ED001","skill":"eval-design-forensics","pattern_id":"HP-EVAL-LEAKAGE",
"title":"short, neutral","description":"which K&N type + the protocol span it
undermines + (L2) the split/preprocessing file:line","severity":"critical|major|
minor|info","observability_level_required":0,
"evidence":[{"claim_id":"C0xx","span":"verbatim substring",
"location":{"file":"...","section":"..."}}],
"verdict_local":"fail|warn|clean|needs_external_check",
"requires_external_check":true|false,"false_positive_risk":"low|medium|high",
"alternative_explanation_checked": "<criticals ONLY — the benign readings you explicitly ruled out: rounding/display precision, unit or metric convention, statistical reporting convention, scope difference. Anchoring proves the text exists, NOT that your interpretation is right; a critical without this field demotes to major>",
"recommended_reviewer_action":"what to CHECK or ASK — never 'reject'"}
An empty array [] is a valid, honest result (the protocol shows no leak).
Persist the raw response to the trace dir (Step 7) before parsing. Failure
handling: MCP stall → re-invoke the identical prompt as a fresh
mcp__codex__codex (never codex-reply). Prose instead of JSON → the Step 5 validator
extracts the outermost [...]; if none, re-ask once "Output ONLY the JSON array." If
the L2 facts were empty, bias every code-confirm item toward needs_external_check —
never invent a split file.
A separate, newmcp__codex__codex thread. Send EXACTLY (fill every [ ... ]):
mcp__codex__codex:
model: gpt-5.6-sol
config: {"model_reasoning_effort": "xhigh"}
sandbox: read-only
cwd: <absolute PAPER_DIR from Step 0>
prompt: |
You are an EVALUATION-VALIDITY and REPORTING-COMPLETENESS forensics reviewer. You
judge two things: (1) does a headline metric rest on an LLM JUDGE that is conflicted
or unvalidated? (2) does the reporting DROP a condition the setup declared, SWITCH a
metric to favor the method, or select "best" with no held-out set? PROPOSE findings
only — do NOT grade the paper; describe a discrepancy to CHECK/CLARIFY, never an
accusation. You do NOT judge whether the numbers are real (another auditor).
INPUTS (read directly in your working directory):
- claims.json — the JUDGE protocol lives in type:"method"/"comparison"/"scope"/
"number"; the DECLARED conditions in type:"scope"/"method"; the reported tables in
type:"table_cell"/"caption"/"number"/"baseline". The ONLY structure you navigate
by; each claim = {claim_id, type, text_span (VERBATIM), location, value?}. You MAY
re-open a source file (and, at L2, the judge/result files) to confirm a span.
JUDGE ANCHOR TARGETS (claim_id + verbatim text_span):
[paste the JUDGE anchors from Step 1]
REPORTING ANCHOR TARGETS (declared-condition / table claims — claim_id + verbatim):
[paste the REPORTING anchors from Step 1]
L2 JUDGE + RESULT-FILE FACTS (raw grep — uninterpreted; empty if L<2):
[inline RUNDIR/judge_grep.txt + RUNDIR/reporting_grep.txt, or "L<2: no repo available"]
RUN OBSERVABILITY LEVEL L = <L from Step 0>.
HARD RULES:
1. ANCHOR every finding above "info" to a real claim_id + a VERBATIM substring of that
claim. For JUDGE-VALIDITY anchor to the judge-protocol claim naming the judge model
(and the compared-systems claim showing the family overlap, as extra evidence). For
SELECTIVE-REPORTING anchor to the setup-DECLARATION claim (what was promised). A
config/result file:line is forensic detail for the description — NOT a valid anchor.
2. DISCREPANCY, NOT ACCUSATION. Say what to CHECK/ASK. Never "reject"/"faked".
3. OBSERVABILITY. Judge identity + the absence of reported validation are read off the
described protocol => observability_level_required 0 (1 if only the source shows it).
A declared-but-unreported condition is stated => 0; CONFIRMING the condition actually
ran but went unreported needs the result files => a SEPARATE observability-2 finding.
4. HONEST FP. (judge) a judge validated against human agreement with bias controls
reported; a judge from a clearly DIFFERENT family than every compared system AND not
load-bearing (corroborated by human eval / standard metrics); a calibrated standard
protocol. UNVALIDATED-ONLY is HIGH-FP — missing validation *reporting* is not proof
none was done (may be in an appendix / a cited standard). (reporting) a declared
condition omitted but explicitly justified ("full grid in the repo"); different
tasks legitimately using different standard metrics; best-selection on a DECLARED
held-out validation set; an honestly-labeled pilot.
5. pattern_id MUST be one of: HP-JUDGE-VALIDITY, HP-SELECTIVE-REPORTING.
CHECKLIST (one finding per concrete discrepancy):
1. JUDGE VALIDITY [HP-JUDGE-VALIDITY] — a headline comparison rests on an automatic
LLM judge, and either:
(a) CONFLICTED — the judge is the same MODEL or model FAMILY as a compared system
(especially the proposed one), so its preference for that system IS the
evidence (self-enhancement / self-preference). This is the lower-FP STRUCTURAL
case (family overlap is checkable) -> severity major, false_positive_risk
"medium". observability 0.
(b) UNVALIDATED — the LLM judge is load-bearing yet the paper reports NO
human-agreement validation (no correlation / kappa vs humans) AND NO bias
control (no position-swap, no length/verbosity control) -> severity major if
the headline rests on it, minor otherwise; false_positive_risk "high" (caps at
minor). observability 0.
ROUTING: an LLM that generates the GROUND-TRUTH labels/targets (not judging
outputs) is HP-FAKE-GT (experiment-forensics, L2) — do NOT raise it here; if
unsure which, set verdict_local "needs_external_check".
2. SELECTIVE REPORTING [HP-SELECTIVE-REPORTING] — one of:
(a) a dataset / baseline / metric / seed-count the SETUP EXPLICITLY DECLARES is
then omitted from the results and is not in the appendix;
(b) METRIC-SWITCHING across tables (Table 2 reports M where the method leads;
Table 3 quietly switches to M' where it also leads) in a way that consistently
favors the proposed method;
(c) "we report the best checkpoint / prompt / run" with NO held-out selection set
(selecting on the test set).
severity major; critical if the omission/switch/selection is what PRODUCES the
headline (false_positive_risk "low" when the declared-vs-reported gap is
unambiguous); minor if peripheral. observability 0 (stated) / 2 (the result file
shows the condition ran but went unreported — a SEPARATE finding).
DE-DUP (do NOT emit these — route them): best-reported-as-mean -> HP-AGG-DRIFT
(consistency-audit); thin overall scope with no comparison -> HP-SCOPE-INFLATE
(consistency-audit); a never-mentioned expected baseline -> HP-MISSING-BASELINE
(baseline-comparison-audit); appendix-vs-main on the SAME quantity ->
HP-APPENDIX-CONTRA (consistency-audit). This pattern is ONLY declared-but-unreported
/ cherry-picked-among-shown.
OUTPUT: a single JSON array and NOTHING ELSE (schemas/finding.schema.json), same shape
as the leakage prompt; set finding_id "ED0xx", skill "eval-design-forensics". An empty
array [] is valid and honest. Set requires_external_check only when you genuinely
cannot settle a point at this level.
Deepen at L2. When L == 2, the judge/result facts let the reviewer promote a
text-only suspicion to a confirmed finding (keep observability_level_required: 0 if
the text already showed the tell; use 2 only for what the files reveal). Fan-out
(optional, breadth). On — effort: max or many evaluation tracks, issue the relevant
checklist item per track as a separate fresh mcp__codex__codex call and
concatenate the arrays — never codex-reply. Persist each raw reply to the trace
(Step 7). Failure handling: identical to Step 3.
Step 5 — Validate + anchor (the anti-hallucination gate)
The executor enforces the ANCHOR gate (the one tools/adjudicate_findings.py
re-applies, so an anchored finding you keep is not silently rejected downstream) plus
eval-design-specific owned-pattern + schema-hygiene + external-check pre-filters,
before keeping anything. The span must be a verbatim, whitespace-normalized
substring of the cited claim (span in base, never base in span — appending
hallucinated text to a real claim must fail). Pass every saved raw reviewer response
(leakage + judge/reporting + any per-track fan-out files); they merge into one findings
file with one ED### namespace:
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
LEDGER="<abs path to claims.json>"
OUT="$(dirname "$LEDGER")/eval-design-forensics.findings.json"
# args: LEDGER OUT then each saved raw reviewer response file from Steps 3–4:
python3 - "$LEDGER" "$OUT" "<resp_leakage.md>" "<resp_judge_reporting.md>" <<'PY'
import json, re, sys, os
ledger_path, out_path = sys.argv[1], sys.argv[2]
resp_paths = [p for p in sys.argv[3:] if p and os.path.isfile(p)]
def nw(s): # mirror adjudicator _norm_ws (whitespace only)
return " ".join((s or "").split())
OWNED = {"HP-EVAL-LEAKAGE", "HP-JUDGE-VALIDITY", "HP-SELECTIVE-REPORTING"} # ALLOWED set
ABOVE = {"critical", "major", "minor"}
# OBS map — the canonical observability each owned pattern is decidable at (documentation;
# the REVIEWER sets observability_level_required per finding, and the adjudicator does the
# real req>run_level downgrade). stated-tell = 0; repo confirm = 2; the 3 leakage external
# subtypes carry no level (needs_external_check -> info).
OBS = {"HP-EVAL-LEAKAGE": "0 stated / 2 verified (proxy|sampling|contamination -> needs_external_check)",
"HP-JUDGE-VALIDITY": "0/1 stated (2 may corroborate)",
"HP-SELECTIVE-REPORTING": "0 stated / 2 verified"}
ledger = json.load(open(ledger_path, encoding="utf-8"))
base = {c["claim_id"]: nw(c.get("text_span", "")) for c in ledger.get("claims", [])
if c.get("claim_id")}
proposed = []
for p in resp_paths:
raw = open(p, encoding="utf-8").read()
m = re.search(r"\[.*\]", raw, re.S) # tolerate prose / code-fence wrapping
try:
chunk = json.loads(m.group(0) if m else raw)
proposed += chunk.get("findings", []) if isinstance(chunk, dict) else chunk
except Exception as e:
print(f"WARN: could not parse {p}: {e}", file=sys.stderr)
kept, demoted, not_owned, ext = [], 0, 0, 0
for f in proposed:
if not isinstance(f, dict):
continue
# OWNED gate: this skill emits ONLY family-H patterns — a missing/non-owned pattern_id is
# DROPPED (not demoted), so nothing foreign can ride the `evaluation` dimension or survive
# above-info on a malformed/absent id.
if f.get("pattern_id") not in OWNED:
not_owned += 1
continue
f["skill"] = "eval-design-forensics"
# ANCHOR: keep only evidence whose span is a verbatim ws-normalized substring of its claim.
# Guard malformed evidence (non-list / non-dict items can't anchor).
anchored = [ev for ev in (f.get("evidence") if isinstance(f.get("evidence"), list) else [])
if isinstance(ev, dict) and ev.get("claim_id") in base and nw(ev.get("span", "")) and
nw(ev["span"]) in base[ev["claim_id"]]] # span IN claim, never claim IN span
f["evidence"] = anchored
# ANCHOR gate: above-info needs >=1 anchored span
if f.get("severity") in ABOVE and not anchored:
f["severity"] = "info"; f.setdefault("_demotions", []).append("unanchored"); demoted += 1
# OBSERVABILITY hygiene — record, never rescore. The adjudicator annotates the level a
# finding declares against the level this run had; a missing/invalid declaration is
# flagged so a forgotten level-2 asymmetry is visible rather than quietly passing as
# confirmed. type() not isinstance() so JSON booleans (True==1) are rejected.
olr = f.get("observability_level_required")
if f.get("severity") in ABOVE and (type(olr) is not int or not (0 <= olr <= 3)):
f.setdefault("_demotions", []).append("undeclared-observability")
# FP-RISK hygiene — false_positive_risk is the REVIEWER's self-assessment, reported as a
# column; the executor never guesses it. A missing/invalid value is flagged, not rescored.
if f.get("severity") in ABOVE and f.get("false_positive_risk") not in ("low", "medium", "high"):
f.setdefault("_demotions", []).append("undeclared-fp-risk")
import os as _aris_os
RESOLVED_MODEL = _aris_os.environ["ARIS_RESOLVED_MODEL"] # exported by the executor from the call that ACTUALLY ran
RESOLVED_REASONING = _aris_os.environ["ARIS_RESOLVED_REASONING"] # (fail LOUD if unset — never stamp a target default)
f["reviewer"] = {"model": RESOLVED_MODEL, "reasoning": RESOLVED_REASONING, "deterministic": False}
# needs_external_check is the auditor saying it could not settle this itself. Record it;
# the adjudicator reports it in its own column and the human weighs it. Do NOT rescore.
if f.get("verdict_local") == "needs_external_check" or f.get("requires_external_check") is True:
f["requires_external_check"] = True; ext += 1
kept.append(f)
for k, f in enumerate(kept, 1): # one namespace, sequential
f["finding_id"] = f"ED{k:03d}"
json.dump(kept, open(out_path, "w", encoding="utf-8"), indent=2, ensure_ascii=False)
above = sum(1 for x in kept if x.get("severity") in ABOVE)
seen = sorted({f.get("pattern_id") for f in kept if f.get("pattern_id") in OWNED})
print(f"validated {len(kept)} eval-design findings (above_info={above}; {demoted} demoted->info "
f"(unanchored/undeclared); {not_owned} not-owned dropped; {ext} needs-external-check->info) -> {out_path}")
print("OBS map for patterns seen:", {p: OBS[p] for p in seen})
PY
Scope of this gate: anchoring + owned-pattern + schema hygiene + external-check
pinning only (schema hygiene = the presence/validity of the reviewer's required
fields — a missing/invalid observability_level_required or false_positive_risk on an
above-info finding fails closed to info, never a guessed default). Do not
re-implement the adjudicator's verdict-bearing gates (the observability LEVEL downgradereq > run_level, the FP-risk cap{high:minor, medium:major, low:critical}, the
verdict) — those need the run level and belong to tools/adjudicate_findings.py, the
single reporter. needs_external_check findings (the 3 leakage external subtypes, and any
unsure judge/GT call) are marked, not rescored — the report gives that its own column. The remaining judgments are the
reviewer's, per the prompt — the K&N type, the conflicted-vs-unvalidated split, the
observability_level_required: 2 tag for a code-only confirm; if a kept finding plainly
violates one, re-run that pass with the correction noted (never hand-fabricate). A finding
that loses all evidence is kept as info (the forensic record stays).
Worked HP-EVAL-LEAKAGE (stated preprocessing-before-split, headline, critical, L0):
{
"finding_id": "ED001",
"skill": "eval-design-forensics",
"pattern_id": "HP-EVAL-LEAKAGE",
"title": "Preprocessing fit before the split may leak the test set into the reported accuracy",
"description": "Claim C006 describes the protocol as 'we standardize all features, then split 80/20'. Standardizing before the split fits the scaler on the test rows (Kapoor & Narayanan leakage type L1 — no clean train/test separation), so the headline accuracy may not measure generalization. Discrepancy to verify: confirm whether the scaler was in fact fit on the training partition only and applied to test.",
"severity": "critical",
"observability_level_required": 0,
"evidence": [
{"claim_id": "C006", "span": "we standardize all features, then split 80/20",
"location": {"file": "main.tex", "section": "method"}}
],
"verdict_local": "fail",
"requires_external_check": false,
"false_positive_risk": "low",
"recommended_reviewer_action": "Ask the authors whether the scaler/imputer was fit on TRAIN ONLY and then applied to test; if it was fit on all data before the split, the reported accuracy should be re-measured with a leak-free pipeline."
}
Worked needs_external_check (pretraining contamination — hand off, do NOT guess):
{
"finding_id": "ED002",
"skill": "eval-design-forensics",
"pattern_id": "HP-EVAL-LEAKAGE",
"title": "Benchmark may be contaminated in the evaluated model's pretraining (not decidable here)",
"description": "Claim C012 reports a headline score on a widely-published benchmark using a frontier LLM. Whether the model saw this benchmark during pretraining is a black-box question undecidable from the PDF or the repo — NOT an allegation. A domain check would use an exchangeability test (Oren 2023), Min-K% Prob (Shi 2023), Time-Travel (Golchin 2023), or BIG-bench canary strings; this tool does not run them.",
"severity": "info",
"observability_level_required": 0,
"evidence": [
{"claim_id": "C012", "span": "achieves 91.2 on the public benchmark",
"location": {"file": "main.tex", "section": "experiments"}}
],
"verdict_local": "needs_external_check",
"requires_external_check": true,
"false_positive_risk": "high",
"recommended_reviewer_action": "Have a domain expert run a contamination probe (Min-K% / exchangeability / Time-Travel) or confirm the benchmark post-dates the model's training cutoff."
}
Worked HP-JUDGE-VALIDITY (conflicted judge, major, L0):
{
"finding_id": "ED003",
"skill": "eval-design-forensics",
"pattern_id": "HP-JUDGE-VALIDITY",
"title": "Headline win-rate rests on a judge that shares a family with the proposed system",
"description": "Claim C021 reports 'our GPT-4-based agent wins 78% of pairwise comparisons, judged by GPT-4'. The judge shares a model family with the proposed system (self-enhancement / self-preference: an evaluator favors its own family's generations), and no human-agreement number is reported. Discrepancy to verify: the 78% is the load-bearing evidence yet the judge is conflicted and unvalidated.",
"severity": "major",
"observability_level_required": 0,
"evidence": [
{"claim_id": "C021", "span": "wins 78% of pairwise comparisons, judged by GPT-4",
"location": {"file": "main.tex", "section": "experiments"}}
],
"verdict_local": "warn",
"requires_external_check": false,
"false_positive_risk": "medium",
"recommended_reviewer_action": "Ask for a human-agreement validation of the judge (correlation/kappa) and a family-disjoint or position-swapped judge; report the win-rate under a judge that does not share a family with the proposed system."
}
Worked HP-SELECTIVE-REPORTING (declared-but-unreported datasets, L0):
{
"finding_id": "ED004",
"skill": "eval-design-forensics",
"pattern_id": "HP-SELECTIVE-REPORTING",
"title": "Two declared evaluation datasets are not reported and not in the appendix",
"description": "Claim C008 declares 'we evaluate on five datasets {A,B,C,D,E}', but every results table reports only {A,B,C} and no appendix reports D or E. Discrepancy to verify: ask for the D/E results, or an explicit reason for the omission. (De-dup: this is declared-but-unreported, not best-as-mean (HP-AGG-DRIFT) or a missing expected baseline (HP-MISSING-BASELINE).)",
"severity": "major",
"observability_level_required": 0,
"evidence": [
{"claim_id": "C008", "span": "we evaluate on five datasets",
"location": {"file": "main.tex", "section": "experiments"}}
],
"verdict_local": "warn",
"requires_external_check": false,
"false_positive_risk": "low",
"recommended_reviewer_action": "Ask the authors for the results on datasets D and E (or an explicit justification for omitting them); confirm the headline survives once the declared conditions are all reported."
}
Step 6 — Emit (one file)
Step 5 wrote the validated array to eval-design-forensics.findings.json (a bare
JSON array conforming to schemas/finding.schema.json), in the ledger's directory. If
both passes found nothing — or this skill was not applicable (Step 1) — the file is
[]; write it anyway (silent skip is forbidden; the orchestrator's *.findings.json
glob and the standalone adjudicate command both expect it at a predictable path). This
skill writes exactly one findings file and runs no deterministic tool of its own
(unlike consistency-audit): every finding is a cross-model proposal, span-anchored,
gated, and handed to the adjudicator.
Step 7 — Trace (forensic; never silently dropped)
Save every reviewer call under the same RUNDIR created at the start of Step 2
(.aris/traces/eval-design-forensics/<YYYY-MM-DD>_run<NN>/). This repo ships no
save_trace.sh, so write files directly — reuse that path; do not re-run the bump
loop here (it would allocate a second empty dir):
RUNDIR="<the RUNDIR printed in Step 2>" # e.g. .aris/traces/eval-design-forensics/<date>_run01
mkdir -p "$RUNDIR" # idempotent — the dir already exists from Step 2
Populate it:
.aris/traces/eval-design-forensics/<date>_run<NN>/
run.meta.json # {skill, paper_id, run_level_L, taxonomy_version:"0.5", leakage_taxonomy:"K&N 2023", generated_at, not_applicable?}
leakage_grep.txt / judge_grep.txt / reporting_grep.txt / hashes.txt # Step 2 L2 facts (if L2)
contamination_dates.json # optional public-record date fact (if gathered)
001-leakage.request.json # the EXACT Step-3 prompt sent (paths + ledger + facts + taxonomy + checklist)
001-leakage.response.md # the FULL raw reviewer response (input to Step 5)
001-leakage.meta.json # {model:"gpt-5.6-sol", reasoning:"xhigh", thread_id, sandbox:"read-only"}
002-judge-reporting.request.json
002-judge-reporting.response.md
002-judge-reporting.meta.json
# 00N-track-<entry>.* for each per-track fan-out call, if used
Each request.json is the independence audit trail — it must show the executor sent
only paths + the ledger + the K&N taxonomy + the mechanical facts + the checklist,
never a digest or pre-judgment of the paper (references/reviewer-independence.md).
Step 8 — Hand off, or adjudicate standalone
Within /anti-autoresearch, stop here: the orchestrator globs every
*.findings.json, runs the adjudicator once over the union, and emits REPORT.md +
report.json. When running this skill alone, you may produce the report yourself —
--ledger is required (it verifies each finding quotes a real ledger span; without
it every above-info finding fails closed to info):
The adjudicator applies, in order: ANCHOR → OBSERVABILITY → FP-RISK → MEMO → SURFACE →
EXTERNAL-CHECK gates, then computes overall_verdict ∈ {CLEAN_GIVEN_EVIDENCE,
SOFT_FLAGS, HARD_FLAGS} (any span-anchored critical decidable at L → HARD_FLAGS).
No model is in the final decision; this skill rolls up under the evaluation
dimension via SKILL_TO_DIMENSION. needs_external_check findings carry no severity
weight — they surface as open questions for a human, never as a flag. (Use the
eval-design.* output names when standalone so you do not clobber the orchestrator's
combined report.json / REPORT.md.)
Output contract
This skill always writes, into the ledger's directory:
eval-design-forensics.findings.json — a JSON array (schemas/finding.schema.json);
the validated leakage + judge-validity + selective-reporting findings (or []). Each
above-info finding carries evidence[].claim_id + a verbatim span, an integer
observability_level_required, a pattern_id ∈ {HP-EVAL-LEAKAGE,
HP-JUDGE-VALIDITY, HP-SELECTIVE-REPORTING}, and an honest false_positive_risk.
The three undecidable leakage subtypes (illegitimate-proxy / sampling-bias /
contamination) and any unsure judge-vs-GT call appear as
verdict_local: needs_external_check + requires_external_check: true, never a
guessed leak.
.aris/traces/eval-design-forensics/<date>_run<NN>/ — the mechanical-facts files +
the raw reviewer call(s).
It writes no verdict and no report of its own — report.json / REPORT.md come
only from tools/adjudicate_findings.py (Step 8 / the orchestrator). It writes no
second deterministic file and never edits the audited paper.
Key rules
Verdict-bearing at L0/L1; the L2-confirm is separate. A stated leak / conflicted
judge / declared-but-unreported condition is decidable from the described protocol →
observability_level_required: 0 (this is the opposite of experiment-forensics,
which is info-only below L2). The L2 confirmation of the same tell is a separate
observability_level_required: 2 finding that is marked as needing L2 on a PDF-only run.
No span → no severity. Reject unanchored/paraphrased findings to info here (the
adjudicator re-enforces). span in claim, whitespace-normalized — never claim in span. The anchor is a PAPER claim (the protocol / judge / declared-condition
sentence); the split-file file:line, config, or leaderboard date is forensic detail
in the description, never the anchor.
Two scales — never conflate. K&N leakage types L1/L2/L3 are not this repo's
observability levels L0/L1/L2. Carry both: the K&N type in description, the
observability in observability_level_required.
Hand off the 3 undecidable leakage subtypes. Illegitimate-proxy feature, sampling
bias, and pretraining/benchmark contamination → needs_external_check; name the
external methods (Oren 2023 / Min-K% / Time-Travel / canary) but never run them.
Judge ≠ ground-truth generator. A judge whose preference IS the metric is
HP-JUDGE-VALIDITY (here); a model that fabricates the reference the metric is
computed against is HP-FAKE-GT (experiment-forensics, L2). Conflicted (family
overlap) caps at major (FP medium); unvalidated-only caps at minor (FP high).
Selective-reporting stays in lane. Only declared-but-unreported / metric-switch /
best-without-held-out. best-as-mean → HP-AGG-DRIFT; thin scope → HP-SCOPE-INFLATE;
never-mentioned baseline → HP-MISSING-BASELINE; appendix-vs-main on the same quantity
→ HP-APPENDIX-CONTRA. Do not double-emit.
Discrepancy, not accusation. Leakage and under-reporting are usually honest
methodological errors; output asks a reviewer to check/clarify, never to reject. The
absence of a reported validation is not proof none was done (high-FP care).
Reviewer ≠ adjudicator. The model proposes findings; adjudicate_findings.py
decides the verdict. This skill emits findings only.
Cross-model, fresh thread per pass/track. Reviewer is a different family (gpt-5.6-sol
xhigh); leakage and judge/reporting are separate fresh mcp__codex__codex threads;
codex-reply is never used (absent from allowed-tools).
Detect-only. Never edit the audited paper (reviewer sandbox is read-only).
Reproducible. Same ledger + same findings → same verdict; the mechanical facts and
any date fact (with URL + date) are traced so the expectation is auditable.
When NOT to use this skill
No claims.json yet → run /evidence-ledger first; this skill never invents
structure from the raw PDF.
The paper describes no evaluation protocol / judge / declared conditions
(APPLICABLE = no in Step 1) → write [], record "eval-design-forensics": "not_applicable" in the run's coverage.json (when orchestrated), and stop.
An LLM generating the ground-truth labels/targets (not judging outputs) →
/experiment-forensicsHP-FAKE-GT at L2.
best-reported-as-mean / appendix-vs-main / thin in-text scope with no comparison
→ /consistency-audit (HP-AGG-DRIFT / HP-APPENDIX-CONTRA / HP-SCOPE-INFLATE).
A never-mentioned expected baseline, or "SOTA / first / beats prior work" →
/baseline-comparison-audit (+ /citation-forensics), handed off via
needs_external_check.
Whether a reported number matches the repo / code (fake GT, self-norm, phantom)
→ /experiment-forensics at L2.
An AI-text / "looks machine-written" verdict → out of scope; surface hints live in
/presentation-signals (auxiliary, surface-class). This repo is not an AI-text
classifier.
On a timer → never /loop / /schedule / CronCreate this skill; re-fire only
when the paper, ledger, or observability level changes (see the fence at the top).
Review tracing
Forensic policy — never silently skipped. The exact RUNDIR layout and the
reviewer-independence audit-trail rule for each request.json (only paths + ledger +
the K&N taxonomy + mechanical facts + checklist were sent — no digest, no pre-judgment)
are specified once in Step 7; follow it for every mcp__codex__codex call (leakage,
judge/reporting, and any per-track fan-out).
Acknowledgements
The taxonomy this skill operationalizes is reframed from the ML-evaluation-methodology
literature for third-party forensics — ledger-anchored findings, observability
tiers, and reviewer ≠ adjudicator:
Kapoor, S. & Narayanan, A. (2023), "Leakage and the reproducibility crisis in
machine-learning-based science," Patterns — the eight leakage types / three
categories that HP-EVAL-LEAKAGE adopts and paraphrases (priority ack).
Contamination methods are named only, never run: Oren et al. (2023) exchangeability
test; Shi et al. (2023) Min-K% Prob; Golchin & Surdeanu (2023) Time-Travel; the
BIG-bench (Srivastava et al. 2022) canary-string convention.
Zheng et al. (2023), "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" —
self-enhancement bias and the human-agreement bar behind HP-JUDGE-VALIDITY;
Panickssery et al. (2024), self-preference (evaluators favor their own
generations); Wang et al. (2024), position bias in LLM evaluators.
Dodge et al. (2019), "Show Your Work: Improved Reporting of Experimental Results,"
and Pineau et al. (2021), the ML Reproducibility Checklist (NeurIPS 2019
Reproducibility Program report) — the under-reporting norms behind
HP-SELECTIVE-REPORTING.
