verify-evals

# Verify Evaluations on a Benchmark This skill audits **existing** evaluation rows on a configured benchmark leaderboard. It does not add new scores — use `add-evals` for that. The goal is to confirm every row on a benchmark page is **apples-to-apples**: the same metric, the same split, the same evaluation protocol, a score faithful to the source paper or release, and an accurate **open vs closed-source** tag (`is_open`). Leaderboards rank models against each other. A row that uses validation AP50 next to hidden-test AP50, or a backbone ablation next to a full submission, misleads users even when the number itself is "correct" for some table in the paper. ## Related skills | Skill | Role in verification | |-------|----------------------| | **`hf-cli`** | Read arXiv papers (`uv run hf papers read`), paper metadata, model cards (`uv run hf models card`), and model info JSON. Primary source for checking reported numbers. | | **`add-evals`** | Defines how rows *should* have been imported — metric filter, split expectations, model naming, `methodology`, `is_open`, `num_parameters`. Verification is essentially running those rules in reverse on existing rows. | | **`add-benchmark`** | Use only if the leaderboard itself is misconfigured (wrong metric linked, missing task association). Do not fix bad rows by changing the leaderboard definition during verification. | ## Pre-flight Run from the repo root: ```bash uv run --env-file backend/keys.env ... ``` Confirm the API is up (any dataset id works; 404 on missing id still means the server responded): ```bash curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8000/api/v1/datasets/276 ``` Resolve the benchmark from the user URL or name. Example: `https://paperswithcode.co/benchmark/scannet?task=3d-instance-segmentation` → dataset slug `scannet`, task slug `3d-instance-segmentation`. ```python from sqlalchemy import text from backend.database.connection import get_engine dataset_slug = "scannet" task_slug = "3d-instance-segmentation" with get_engine().connect() as conn: ds = conn.execute(text( "SELECT id, name, slug, introducing_paper_id, paper_url FROM datasets WHERE slug = :slug" ), {"slug": dataset_slug}).mappings().first() task = conn.execute(text( "SELECT id, name, slug FROM tasks WHERE slug = :slug" ), {"slug": task_slug}).mappings().first() metrics = conn.execute(text( "SELECT m.name, m.direction, td.supports_harness " "FROM task_dataset_metrics tdm " "JOIN metrics m ON m.id = tdm.metric_id " "JOIN task_datasets td ON td.task_id = tdm.task_id AND td.dataset_id = tdm.dataset_id " "WHERE tdm.task_id = :task_id AND tdm.dataset_id = :dataset_id" ), {"task_id": task.id, "dataset_id": ds.id}).mappings().all() print(dict(ds), dict(task), [dict(m) for m in metrics]) ``` Record the **leaderboard contract** before reviewing rows: - `dataset_id`, `task_id` - Allowed metric name(s) — usually one primary metric (e.g. `AP50`) - `supports_harness` — whether `evaluations.harness` should be populated - Expected split and protocol for this benchmark (see Step 1 — do not assume hidden test without checking) ## Step 1: Define the apples-to-apples contract Every row on a leaderboard must match the same contract. Infer it from the benchmark's official definition, existing rows, domain convention, and (for older CV/3D leaderboards) legacy PwC — then apply it uniformly. ### Split (most common failure mode) Check whether the leaderboard is meant to compare: | Split | Typical use | |-------|-------------| | **Hidden test** | Official benchmark server results (ScanNet, many vision leaderboards) | | **Public validation** | Ablation tables, development comparisons, papers that never submit to the server | | **Train / other** | Almost never belongs on a public SOTA leaderboard | **Rule:** Do not mix splits on one ranked leaderboard. Every row must use the **same** split (and the same score from that split). Read the paper table caption carefully: "validation set", "test set", "hidden test", "leaderboard submission", "we report on val in the main paper and test in the appendix". #### Resolving the split when convention is unclear Do **not** jump straight to "hidden test because ScanNet has a benchmark server." Many curated pages — including legacy PwC — used **validation** for most papers and test only for benchmark submissions. 1. **Official source** — benchmark homepage / server docs (e.g. `kaldir.vc.in.tum.de/scannet_benchmark/`) → hidden test is the server leaderboard. 2. **Existing rows** — tag each row's split from `methodology` and flag mixing before per-row source checks: ```python # Quick split-mixing audit for r in rows: m = (r["methodology"] or "").lower() split = "test" if "hidden test" in m or " test " in m else ( "val" if "validation" in m or " val " in m else "unclear" ) print(r["model_name"], r["metrics"], split) ``` 3. **Legacy PwC score-matching** — stream `evaluation-tables.json` (never load the full file on the VPS; use `ijson`). Match each legacy score to known val/test cells in papers or READMEs to infer which split legacy used. Legacy rows carry **no split label** — only score equality proves it. ```python import ijson, re from pathlib import Path path = Path("data/legacy_pwc_data/evaluation-tables.json") with open(path, "rb") as f: for item in ijson.items(f, "item"): if item.get("task") != "<Task Name>": continue for ds in item.get("datasets", []): if "<dataset slug fragment>" not in ds.get("dataset", "").lower(): continue for row in ds.get("sota", {}).get("rows", []): print(row.get("model_name"), row.get("metrics"), row.get("paper_url")) ``` Legacy PwC was **not** split-consistent (e.g. ScanNet Mask3D mixed val mAP with test AP50 in one row; ScanNet200 used validation for ODIN/ISBNet/MSTA3D but test mAP for Mask3D). Treat legacy as **historical evidence**, not a mandate to keep mixing. 4. **User choice** — when official docs, legacy, and current rows disagree, report the options with evidence and let the user pick the contract (e.g. "validation for all rows except Mask3D on hidden test"). Document the chosen contract in the audit report and apply it to every row. **Exception rows:** a contract like "validation except paper X on test" is valid when mirroring legacy or when only one method has an official test submission. List exceptions explicitly; every other row must still match the dominant split. ### Metric The stored `best_metric` and the sole key in `metrics` must match a metric linked in `task_dataset_metrics` for this `(task_id, dataset_id)`. Also verify the **source table column** maps to that metric: - `AP50` / `mAP@50` / `mAP 50` → usually `AP50` when that is the configured metric - Do not accept component subscores (e.g. MME-P on an `Score`-only leaderboard) - Do not accept a different quantity (F1 imported as Accuracy, percent OCRBench score without scale normalization) Cross-check `add-evals` Step 5 — the same exclusion rules apply in reverse. ### Evaluation setup / input modality Many benchmarks allow only one input representation. Examples: - ScanNet instance segmentation: mesh-sampled point cloud on hidden test (standard); sensor RGB-D with feature transfer is a **different setup** even on the same split - Image-text retrieval: I→T vs T→I are separate leaderboards — never mix directions If setups differ, rows are not comparable even when metric and split match. Prefer **delete** over keeping a row with a long methodology disclaimer unless the user explicitly wants a mixed setup page. ### Model identity The score must be for the model named in `evaluations.model_name`, using the setup that row claims: - Primary method vs backbone-only ablation (e.g. `PTv3 + PointGroup` backbone table vs `Mask3D` end-to-end method) - Correct variant (size, mode, decoder/head suffix when the score is for a downstream system) - Not a comparator/baseline row unless the leaderboard intentionally tracks comparators (default: paper's **own** best eligible result only, per `add-evals`) ### One best eval per model (no variant duplicates) A leaderboard should show **each model family once**, with its **best** eligible score — not multiple rows for the same releasing paper under different config names (e.g. `Claude Opus 4.8 (High)` and `Claude Opus 4.8 (Max)` both on paper 97913). This is **not** caught by the DB unique key `(paper_id, task_id, dataset_id, model_name, harness, best_metric)` because `model_name` differs per variant. The benchmark page also keys display rows on `model_name`, so variant duplicates appear as separate ranked entries even when they represent the same model release. **Default contract:** one row per `paper_id` on a leaderboard, keeping the highest score that still matches the split/metric/setup contract. Keep the surviving row's `model_name` as the variant that achieved that score (e.g. keep `Claude Opus 4.8 (Max)` at 34.08%, delete `Claude Opus 4.8 (High)` at 33.80%). **When multiple rows per paper are OK:** only when variants are genuinely different comparable models on this benchmark — e.g. `ViT-B/16` vs `ViT-L/16` are separate size tiers, not reprompted vs non-reprompted runs of the same release. When in doubt, report both options; default to one row per paper for agent/runtime-variant leaderboards (PostTrainBench, DeepSWE, FrontierSWE, etc.). **Quick variant-duplicate audit** (run in Step 2 before per-row verification): ```python from collections import defaultdict from sqlalchemy import text from backend.database.connection import get_engine task_id = 14 dataset_id = 810 with get_engine().connect() as conn: rows = conn.execute(text(""" SELECT e.id, e.paper_id, e.model_name, e.best_metric, e.metrics, e.harness, e.best_rank, p.title FROM evaluations e JOIN papers p ON p.id = e.paper_id WHERE e.task_id = :task_id AND e.dataset_id = :dataset_id ORDER BY e.paper_id, (e.metrics->>e.best_metric)::float DESC NULLS LAST """), {"task_id": task_id, "dataset_id": dataset_id}).mappings().all() by_paper = defaultdict(list) for r in rows: by_paper[r["paper_id"]].append(r) for paper_id, group in sorted(by_paper.items()): if len(group) <= 1: continue print(f"\npaper_id={paper_id} ({group[0]['title']}) — {len(group)} rows:") for r in group: score = r["metrics"].get(r["best_metric"]) print(f" id={r['id']} rank={r['best_rank']} {r['model_name']} {score}") print(" → keep highest score; delete or skip lower variants") ``` For external leaderboard imports that re-sync from a JSON feed, fix the import script to consolidate by `paper_id` before insert — see `_best_entry_per_paper` in `backend/database/evaluations_setup/import_posttrainbench_external_evals.py`. Re-running the import after that change is preferred over hand-deleting rows one by one. **Worked fix (PostTrainBench, agents task):** five papers had two agent-config variants each (High vs Max, reprompted vs not, etc.). Kept only the highest Accuracy per `paper_id` — e.g. Opus 4.8 (Max) 34.08% over Opus 4.8 (High) 33.80%. Leaderboard dropped from 23 to 18 rows with no duplicate model families. ### Paper attachment Each row's `paper_id` must be the paper that **introduced or released** the evaluated model, not a survey or unrelated technique paper. The score must appear in that paper (or its official README/model card linked from it). ## Step 2: Inventory all rows on the leaderboard ```python from sqlalchemy import text from backend.database.connection import get_engine task_id = 327 dataset_id = 276 with get_engine().connect() as conn: rows = conn.execute(text(""" SELECT e.id, e.model_name, e.best_metric, e.metrics, e.methodology, e.paper_id, e.harness, e.is_open, e.hf_model_url, e.num_parameters, e.external_source_url, e.best_rank, p.arxiv_id, p.title, p.source FROM evaluations e JOIN papers p ON p.id = e.paper_id WHERE e.task_id = :task_id AND e.dataset_id = :dataset_id ORDER BY (e.metrics->>e.best_metric)::float DESC NULLS LAST, e.model_name """), {"task_id": task_id, "dataset_id": dataset_id}).mappings().all() for r in rows: print(r["id"], r["best_rank"], r["model_name"], r["metrics"], f"open={r['is_open']}", r["arxiv_id"]) ``` Also fetch via API for rank sanity: ```bash curl -s "http://localhost:8000/api/v1/evaluations/?dataset_id=<dataset_id>&task_id=<task_id>&page_size=100" ``` Flag duplicates on the unique key `(paper_id, task_id, dataset_id, model_name, harness, best_metric)`. Also run the **variant-duplicate audit** from Step 1 (group by `paper_id`, flag papers with >1 row). Resolve these before per-row source checks so you do not waste time verifying rows that should be deleted anyway. ### Quick open/closed tag scan Run before per-row verification to catch obvious mis-tags. The API defaults `is_open` to `true`, so missing or wrong closed tags are a common import gap — users rely on `is_open === false` to show the closed-source badge and to hide proprietary rows via Settings. ```python # Heuristic pre-scan — follow up with source checks in Step 3 CLOSED_NAME_MARKERS = ( "claude", "gpt-", "gpt ", "gemini", "grok", "o1-", "o3-", "o4-", ) for r in rows: name = (r["model_name"] or "").lower() stored = r["is_open"] hf_url = r.get("hf_model_url") # add to Step 2 SELECT if auditing deeply if stored is not False and any(m in name for m in CLOSED_NAME_MARKERS): print(f"REVIEW is_open: id={r['id']} {r['model_name']} stored={stored} (proprietary name)") if stored is False and hf_url: print(f"INCONSISTENT: id={r['id']} is_open=false but hf_model_url={hf_url}") if stored is not False and not hf_url and any(m in name for m in CLOSED_NAME_MARKERS): print(f"LIKELY_WRONG_IS_OPEN: id={r['id']} {r['model_name']}") ``` Also check **consistency across rows for the same model variant**: every eval for the same `(paper_id, model_name)` on different benchmarks should share the same `is_open`. Flag when one leaderboard row is closed and a sibling row for the identical variant is open. ```python from collections import defaultdict from sqlalchemy import text from backend.database.connection import get_engine