| name | verify-results |
| description | Check whether reported paper metrics are consistent with outputs from the local code artifact. Use for result reproduction checks, artifact evaluation, reproducibility checklists, code-versus-paper audits, table or claim metric comparisons, and badge-readiness review. |
Verify Results
Close the loop between what the paper claims and what the artifact produces.
This skill helps the author confirm their reported numbers reproduce: it locates
the experiment code, helps stand up a clean/sandboxed run, runs the artifact's
own tests, and does a consistency audit — comparing the metrics the run
produces against the paper's tables and claims, within a tolerance that does not
change the paper's conclusions. It reports mismatches (paper says X, code
produces Y) and missing reproduction steps, and audits the artifact against
current reproducibility-badge expectations.
It is a copilot: it sets up and guides, and the author runs anything heavy
(training, long evals) in their own environment. It never fabricates a number,
never executes destructive commands, and never claims a result was
independently reproduced — a clean audit means consistent, not reproduced.
When to use
- "Do my results reproduce?" / "Does my code match the paper's tables?"
- "Check my reproducibility" / "verify my experiments" / "reproduce my numbers".
- Prepping an artifact for an evaluation track (ACM AE, USENIX, OSDI, SOSP,
SIGMOD ARI, ETAPS, NeurIPS/ICML/ACL reproducibility).
- Filling a reproducibility checklist (NeurIPS Paper Checklist, ACL Responsible
NLP, ML Code Completeness) and wanting an honest read on each item.
- After a results table changes and you need to confirm the code still produces it.
Inputs
- The artifact / experiment code (a directory; a repo URL the author has
cloned locally — this skill reads local files, it does not clone for you).
- The paper
.tex whose tables/claims are being checked (or the specific
\input file that holds the results table).
- The target venue's artifact track, if any — its current Call for
Artifacts decides which badges exist and what hosting they require.
- Optionally, a metrics file from a prior run (JSON/CSV) to compare without
re-running.
Process
This skill follows plan → set up → run (author) → audit, with the verification
step grounded in external, measurable signals (test pass/fail, a numeric
diff against a file the run produced) — never the model's own judgment that the
numbers "look right".
-
Locate the experiment code and the claims. Confirm where the code lives
and which paper tables/claims it is supposed to produce. Extract the paper's
reported numbers into a reviewable claims ledger:
python3 scripts/extract_claims.py paper.tex --ledger claims.json
The ledger is a starting point, not ground truth — walk it with the author:
drop spurious numbers (years, citation counts, the top-1=1 from a \\ row),
fix metric labels, mark each kept claim confirmed. The author is the author.
-
Audit the artifact for completeness and badge-readiness.
python3 scripts/audit_repo.py path/to/artifact --blind <single|double|none>
This inventories the repo against the ML Code Completeness Checklist
(dependencies, training code, evaluation code, pretrained models, a README
with a results table + exact reproduce command), checks for the artifact's
own tests, flags missing reproduction steps, and warns when the only
hosting is a GitHub/personal URL (badge tracks want an archival DOI —
Zenodo/FigShare/Dryad/Software Heritage). It runs nothing. Under
--blind double it also scans the README for de-anonymizing emails/URLs.
This is a fast pre-comparison gate, not the deep version. Don't re-do work
the sibling skills own: making the code run-ready/deterministic and the
repro-essentials audit belong to test-research-code;
packaging, the badge taxonomy, and the archival DOI belong to
prepare-artifacts; the deep double-blind
sweep belongs to anonymize-paper /
refactor-research-code. The unique job
of this skill is the consistency audit (step 5) — does the run's output
match the paper's tables. Use audit_repo.py only to confirm there is enough
to run before comparing, then hand deep gaps to the owner skill.
-
Re-verify the badge rules against the live Call for Artifacts — mandatory.
Badge offerings change per venue, per year (e.g. one cycle a venue offers
all three badges; another, only Artifacts Available). The terms Reproduced
vs Replicated were swapped by ACM after 2020-05-14 — pre-2020 papers use
the inverse meanings. Do not state any badge requirement, hosting rule, or
deadline from memory: fetch the venue's current CFA and confirm it with a
source URL and access date.
Reproducibility standards and the badge taxonomy are in
references/repro-standards.md — treat it as a
map of what to verify, not as current truth.
-
Set up a clean, sandboxed run — then hand the author the commands. A
reproduction must run from a pinned, isolated environment, not the author's
polluted shell. Help build the recipe (fresh venv/conda/container from the
dependency spec; seeds fixed; the exact command from the README), but the
author runs anything heavy. See
references/sandbox-and-run.md. First have the
author run the artifact's own tests (pytest, make test, the repo's
harness) — a concrete pass/fail gate before any metric comparison. Never run
destructive commands; never auto-install into the author's base environment.
-
Consistency audit: compare produced metrics to the paper. Point the run's
output (a metrics JSON/CSV the author generated) at the confirmed ledger:
python3 scripts/compare_metrics.py --ledger claims.json --metrics run.json \
--rel-tol 0.01 --abs-tol 0.005 [--map test_acc=c1 ...]
It reports MATCH / MISMATCH / MISSING per claim with a tolerance that
does not change the paper's claim — never bit-exact (ACM, SIGMOD ARI,
and ETAPS all require only agreement within tolerance / "similar behavior").
Tune --rel-tol/--abs-tol to the metric's scale and use --map when names
differ. A metric/no-produced-value is a missing repro step (the paper
reports it; the run didn't emit it).
-
Decide the verification outcome with explicit stop conditions. Map each
compared claim to: match (consistent within tolerance), mismatch
(paper says X, code produces Y — reconcile: stale table? wrong seed? different
split? selective reporting?), or unverified (could not run / metric not
emitted — say so, never paper over it). Do not loop indefinitely: stop
when all confirmed claims are match-or-explained, or escalate to the author
when a mismatch needs a judgment call (which number is right). Escalation is a
feature, not a failure (working-principle #4).
-
Write the reproduction report to
paper-workspace/review/reproduction-report.md and append a line to
paper-workspace/INDEX.md. Lead
with the verdict (N of M claims consistent), then the mismatch table (claim,
paper value, produced value, |diff|, likely cause), the artifact-completeness
checklist with each item's status, missing repro steps, and badge-readiness
per the live CFA. State plainly what was and was not actually run.
Output
A reproduction-report.md: verdict (consistent claims / total) → mismatch
table (paper vs produced, with diffs and suspected cause) → artifact
completeness checklist (5 items + tests + hosting) → missing reproduction steps
→ badge-readiness against the live CFA (with source links and dates). Plus the
machine-readable claims.json ledger and the --json outputs if requested.
Adapt to your discipline
The metric heuristics target ML/systems papers (accuracy, F1, BLEU, latency,
speedup...). For other fields, the ledger is just {metric, value} records —
hand-author it for any quantitative claim (effect sizes, p-values, runtimes) and
compare_metrics.py still does the tolerance-aware audit. Non-code artifacts
(datasets, proofs) use steps 2–3 only.
Related skills (don't duplicate them)
This skill's one unique job is the consistency audit: does the run's output
match the paper's reported numbers. Everything adjacent has an owner — hand it off
rather than re-doing it.
audit_repo.py here is a lightweight pre-comparison gate, not a replacement for
test-research-code's repro_check.py or prepare-artifacts' badge work.
Guardrails
- Consistency is not reproduction. A clean audit says the produced numbers
match the paper within tolerance — it does not mean the result was
independently reproduced or replicated. Never claim a badge is earned; that is
a committee's call against the live CFA.
- Never fabricate a number, a metric, or a "passing" run. If a run did not
happen or a metric was not emitted, report it as unverified — do not infer it.
- Don't trust the model's own read of correctness. The verification signal
is external: a test exit code, a numeric diff against a file the author
produced. Self-reflection validates hallucinations — do not use it as the gate.
- Run nothing heavy or destructive. The author runs training/long evals in
their own sandbox; this skill sets up and guides. No installs into the base
env, no
rm, no network side effects.
- Anonymization-aware. Under double-blind, flag identifying content in the
artifact and prefer an anonymized mirror; never expose the author's identity.
- Never submit the paper or the artifact to any system on the author's behalf.
Memory
Uses the shared .paper-memory/ convention (full spec:
paper-memory-convention.md).
- At start: read
.paper-memory/lessons.md (and profile.yml for the
contribution type — a dataset/system paper is judged on the artifact more
heavily). Lead with any recurring repro habits (e.g. "tables drift from the
code between drafts", "unpinned dependencies").
- At end: append each finding worth remembering as one dated entry in the
shared format
- [YYYY-MM-DD] (verify-results | <scope>) issue -> recommendation (use reflect-and-improve's reflect_log.py append, which
dedupes and dates). Tag a cross-paper habit recurring, a one-off this-paper.
- Create
.paper-memory/ on demand and offer to add it to .gitignore. It is
local-only; never upload it or copy it into this repo.
