add-evals

# Add Evaluations This skill imports evaluation results into the paperswithcode database from any of the common sources that ship benchmark tables: Hugging Face model cards, a paper on Arxiv/Hugging Face/paperswithcode.co, or a Github or project page URL. It is intentionally conservative: it only writes evaluation rows for benchmarks that already exist in the database with a task association and a configured leaderboard metric. It never invents new datasets, tasks, or metrics during the import, and it never imports rows for source-only metric labels that are not already configured on the exact leaderboard. ## Why this exists Model releases (especially flagship LLMs / VLMs / OCR models) advertise benchmark numbers across several surfaces — the HF model card, the technical report or arXiv paper, the GitHub README and external blog posts/project pages. Any of them can be the only source where a particular table appears, so the import workflow has to be source-agnostic but consistent. The natural workflow is: - Reuse existing benchmark datasets in the DB by matching names from the source. - Skip source benchmarks that are not yet in the DB (curate those separately later, not as a side effect). - Add evaluation rows for the model only on benchmarks that are fully wired up (dataset linked to any task with the specific leaderboard metric being imported). - Keep imports idempotent and easy to re-run. This avoids polluting the schema with half-configured benchmarks, false-positive name matches, duplicate evaluation rows, or mixed-metric leaderboards where incomparable component scores are ranked together with canonical scores. ## Pre-flight Always run from the repo root using the project environment: ```bash uv run --env-file backend/keys.env ... ``` Confirm the backend is reachable and the admin token is set: ```bash curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8000/api/v1/datasets/1 grep -E '^PWC_ADMIN_API_TOKEN=' backend/keys.env | head -1 | sed 's/=.*/=<redacted>/' ``` If the server is not running, start it (or restart it without asking confirmation per AGENTS.md). ## Common Shortcuts From Recent Curation ### Transformers model docs When the user gives a Transformers model doc URL such as `https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/<model>.md`, or points at `data/transformers_model_doc_papers.csv`, treat it as a normal source surface but use the CSV as a resolver: 1. Look up the model doc slug or CSV line in `data/transformers_model_doc_papers.csv`. 2. Use the CSV arXiv/paper mapping when it clearly identifies the source paper. 3. Fetch the raw model doc markdown and parse its evaluation tables. 4. If the doc describes a release with no matching arXiv paper, run `add-external-paper` first and attach the evals to that external row. For "next N rows" or "lines X-Y of the CSV", process each row independently through this skill and return a compact summary: source row, paper id, rows added, skipped benchmarks, and any blocker. For larger batches, it is fine to spawn one worker subagent per row with this skill loaded, but the parent agent should still merge the final summary and avoid duplicate writes. ### Misattributed or moved evaluations If the user says existing evals landed on the wrong paper, first create or resolve the correct source row, then move only the matching rows in one transaction. Scope the update by `old_paper_id`, target `model_name` or benchmark set, and optional `external_source_url`; never move all rows from a paper unless the user explicitly asks. ```bash uv run --env-file backend/keys.env python - <<'PY' from sqlalchemy import text from backend.database.connection import get_engine old_paper_id = <old_id> new_paper_id = <new_id> model_pattern = "%<model name>%" new_source_url = "<canonical source URL>" # or None to preserve row-level sources with get_engine().begin() as conn: before = conn.execute(text(""" SELECT id, model_name, dataset_id, best_metric FROM evaluations WHERE paper_id = :old_paper_id AND model_name ILIKE :model_pattern ORDER BY id """), {"old_paper_id": old_paper_id, "model_pattern": model_pattern}).mappings().all() print("moving", len(before), "rows") conn.execute(text(""" UPDATE evaluations SET paper_id = :new_paper_id, external_source_url = COALESCE(:new_source_url, external_source_url) WHERE paper_id = :old_paper_id AND model_name ILIKE :model_pattern """), { "old_paper_id": old_paper_id, "new_paper_id": new_paper_id, "model_pattern": model_pattern, "new_source_url": new_source_url, }) PY ``` After moving rows, verify zero matching rows remain on the old paper, the expected count appears on the new paper, and duplicate keys were not created. ## Workflow ### 1. Identify and fetch the evaluation source Different starting points should converge on the same triple — a model identity, an evaluation table, and a paper row to attach it to. Pick the first source below that the user actually gave you, and then follow the fallback chain until you have an evaluation table: 1. **HF model repo / card** — fetch the card and inspect the metadata. Always invoke `hf` via `uv run` so the venv-installed CLI is used: ```bash mkdir -p /tmp/<repo_slug> uv run hf models card <repo_id> --text > /tmp/<repo_slug>/card.md uv run hf models info <repo_id> > /tmp/<repo_slug>/info.json ``` The model card is the primary surface. If it has evaluation tables, that is your source. If it does not, read the `info.json` metadata: the card's `arxiv` tag (e.g. `arxiv:2401.12345`) or `papers` field points at a linked paper, and the card body or repo description often contains a GitHub URL. Fall through to step 2 / 3 with those identifiers. 2. **Paper (arXiv, HF, PDF report, or PapersWithCode paper with an arXiv ID)** — when the source is a paper directly, or the model card was missing evals, read the paper as markdown and inspect linked artifacts: ```bash uv run hf papers read <arxiv_id> > /tmp/<paper_slug>/paper.md uv run hf papers info <arxiv_id> > /tmp/<paper_slug>/paper-info.json ``` Extract evaluation tables from the paper body first. If the paper itself has no usable table, look for official artifacts linked from the paper text and metadata. For PDF reports that need an external paper row, inspect the extracted first pages for a real abstract/opening summary and pass that text verbatim to `add-external-paper`; do not generate a replacement summary when the PDF already contains one. Also distinguish official GitHub repositories from cited dependencies or frameworks before creating the external row, because official repo URLs should be marked official while incidental GitHub links should not be. For papers with an arXiv ID, also look for HF models tagged with that paper. The current `uv run hf papers info <arxiv_id>` output does not reliably expose linked models, so use the Hugging Face model search page directly: ```bash curl -L "https://huggingface.co/models?other=arxiv:<arxiv_id>" > /tmp/<paper_slug>/hf-models.html ``` Treat matching HF model cards as candidate sources only. A model can appear in this search simply because its README/model card mentions the arXiv ID, not because the paper introduced that model. Before parsing evals from any discovered model card, be 100% sure the model was introduced by the paper itself: the model card should clearly present the paper as the model's own technical report/release paper, share the same authors/organization/release context, and not merely cite the paper as related work, a base model, a benchmark source, or background. If this is not certain, do not use that model card as a source; ask the user or skip it. For each confirmed introduced model, fetch the model card and metadata as in step 1 and use its evaluation tables as candidate source rows. If the paper has no usable table but lists a GitHub repo, fall through to step 3. 3. **GitHub repository** — when the model card or paper points to a GitHub repo, or the user provided one directly, fetch the README via the `gh` CLI and use it as the source: ```bash gh repo view <owner>/<repo> --json description,url,homepageUrl > /tmp/<repo_slug>/repo.json gh api repos/<owner>/<repo>/readme -H "Accept: application/vnd.github.raw" > /tmp/<repo_slug>/README.md ``` Repo READMEs often contain the most complete benchmark tables for new releases (especially for OCR/VLM/robotics models). Treat README tables exactly like card tables for the rest of the workflow. GitHub repositories are also an important source-availability signal. Do not mark a row closed merely because the paper page has no direct HF model link. Inspect the official repo for: - direct checkpoint links, release assets, Git LFS pointers, or download scripts; - training/evaluation scripts and configs that reproduce the reported checkpoint from public data or listed public artifacts; - linked Hugging Face model or dataset repos in the README, docs, shell scripts, config files, or release notes. If the official repo clearly releases weights, or clearly provides reproducible scripts for the evaluated model/checkpoint, treat the evaluated row as open (`is_open = true`) even when `hf_model_url` stays unset. Example: a paper whose official GitHub repo contains scripts to reproduce the reported policy checkpoint should not be marked closed just because Hugging Face has no linked model card. When the README or repo files mention HF repos that are not linked from the HF paper page, inspect them directly. HF repos without model cards, without `arxiv:` tags, or without README metadata can still be official if the source GitHub repo links them. Use these links for `papers.hf_models` and `evaluations.hf_model_url` only when the source ties them to the evaluated variant. 4. **Benchmark chart image** — when a card, README, or release page has no text score for a benchmark but embeds a results chart, download the image and read it (image viewer / vision). Import only values printed on the chart as numbers for bars clearly labeled as the introduced model; never estimate from bar height or axis gridlines. Apply the same version, split, setting, metric, and scale rules as for a table, and cite the image URL plus the exact panel title, bar label, and printed value (for example `Chart image on https://huggingface.co/org/model: panel "SWE-bench Verified", bar "Model-X (thinking)", printed "74.2"`) as the row's evidence. Prefer a text table whenever one exists. The remote curator follows the same procedure in `remote-paper-curation/SKILL.md`. You can combine sources when each one carries part of the picture (e.g. the card has the "main" table, the paper has an ablation, the README has the final leaderboard). Track per-row context — which source the score came from, model variant / mode / shot count / metric label / backbone head or task framework / harness notes — so model names and methodology lines you write later are accurate and traceable. While reading each source, also pin down the **raw model parameter count** for every model row you intend to import, since the database now stores it on each evaluation (`evaluations.num_parameters`, BIGINT). Check, in order: the HF model info JSON (`safetensors.total` or an explicit parameter field), the model card / paper / README headline that states the model size (e.g. "7B parameters", "355B-A22B", "1.5 billion parameters"), and any size column already in the source's own benchmark table. Convert size labels to the raw integer (`7B` -> `7_000_000_000`, `355B-A22B` -> `355_000_000_000` total parameters not active, `1.5B` -> `1_500_000_000`, `355M` -> `355_000_000`). Mixture-of-experts models should use **total** parameters by default; if you store the active count instead, mention it in `methodology` so the meaning of the stored value is unambiguous. If the source does not state the parameter count clearly for that specific variant, do not guess from sibling models, blog headlines, or organization conventions — leave the field unset for that row and report it as missing. Accuracy matters more than coverage here. While reading each source, also try to pin down the **official Hugging Face model repo URL** for every model row you intend to import, since the database stores it per evaluation (`evaluations.hf_model_url`, TEXT). This is strictly opt-in: only set it when you are **100% certain** the repo is the exact evaluated model — never guess, and prefer leaving it blank over risking a false positive. Apply these rules: - **Certainty gate.** Only populate `hf_model_url` when the source itself ties the score to that specific repo: the score table is on that model card, the card/README/paper explicitly links the evaluated checkpoint to the repo, or the repo id unambiguously matches the evaluated variant name and size. If a benchmark number could belong to a sibling model, a different size, a base vs instruct variant, or a comparator you are not 100% sure about, leave `hf_model_url` unset for that row. - **Prefer the official organization.** Use the repo under the model's official HF organization (e.g. `MiniMaxAI/MiniMax-M3`, `Qwen/...`, `deepseek-ai/...`), not community re-uploads, mirrors, finetunes, or third-party conversions, even when those rank higher in search. If only a community mirror exists and no official repo, leave the field unset rather than linking the mirror. - **Prefer the non-quantized weights.** When the official org publishes several variants of the same checkpoint that differ only by precision/quantization (e.g. a base repo plus `-FP8`, `-float16`, `-INT4`, `-GGUF`, `-AWQ`, `-GPTQ` repos), link the full-precision / non-quantized base repo. Only link a quantized repo when the evaluated row is specifically that quantized variant. - **Distinct model variants get their own URL.** Different evaluated variants (size, mode, backbone) that have their own official repos should each point at their own repo; do not reuse one repo across rows that are genuinely different models. - **HF paper linkage is not required.** A repo can be official even when it is not returned by `hf papers info`, has no model card, lacks an `arxiv:` tag, or is absent from the HF paper page. If the paper's official GitHub/project page links the repo and the repo contains the evaluated checkpoint/files, it is eligible for `hf_model_url`. Closed-source API models (Claude/GPT/Gemini and similar) usually have no downloadable weights repo — leave `hf_model_url` unset for those rows. This field is independent of the paper-level `papers.hf_models` array; keep appending official model URLs there for traceability regardless. Do not extract Elo scores. Cards, papers, and READMEs increasingly include LLM-judged Elo rankings (e.g. Gemini-judged Elo over olmOCR-Bench / OmniDocBench / XDocParse / OCRArena). These are pairwise-comparison rankings, not the benchmark's native metric, and they are tied to the specific judge model and pool of comparators used at the time of publication. They do not map to any leaderboard metric in this database and they are not comparable across sources, so skip every Elo table outright and only ingest the underlying native-metric tables (Accuracy, Edit Distance, ANLS, Pass@1, Score, etc.). Mention the skipped Elo table in the final report so the user knows it was intentional. Do not decide harness handling from benchmark names alone. The database marks each task-dataset leaderboard with `task_datasets.supports_harness`. Only populate `evaluations.harness` when the selected leaderboard has `supports_harness = TRUE`. For Terminal Bench 2.0, SWE-Bench Verified, and SWE-Bench Multilingual, preserve named runtime/framework values such as `Terminus-2`, `OpenHands`, `MiniSWE-Agent`, or `Claude Code` when the source reports them. If the selected leaderboard supports harness values and the primary source plus official linked artifacts do not name the runtime/framework, set `harness = "Not reported"` rather than leaving it blank or using `unknown`. For non-supporting leaderboards, put runtime/framework/tool details in `methodology`, or use a concise `model_name` suffix for clearly distinct non-harness setups such as `(w/ tools)`, `(Agent Swarm)`, or `(w/ python)`. ### 2. Confirm import scope with the user Sources typically contain several target-model variants and many comparator models. By default, import the best reported result from a model introduced by the source **for each benchmark leaderboard independently**, not one globally selected flagship row and not every ablation row. Benchmark pages rank results per dataset/metric, so the selected model may legitimately vary by benchmark: if one introduced variant wins ImageNet-O and a different introduced variant wins CLEVR Distance, import those two winning rows with their own model names. This keeps benchmark pages focused on the strongest reported result for the paper while avoiding comparator rows and lower-performing ablations. Before importing anything, agree with the user on: - Which target-model rows are eligible. Sensible defaults: - Include only models/configurations introduced by the source paper or release. Exclude baseline/comparator rows such as Opus, GPT, Gemini, K2, GLM, CLIP, OpenCLIP, or previous versions unless the source itself introduced that model or the user explicitly asks for comparators. - **No external evals by default.** Every imported row must use the **source paper's `paper_id`** for the model that paper introduces. Do **not** scrape comparison-table baseline scores and attach them to other papers' `paper_id` rows, even when the source paper (e.g. a benchmark paper or SOTA comparison table) reports those numbers. That pattern — scores from paper A posted on paper B with `external_source_url` pointing back at A — is an **external eval** and is out of scope unless the user explicitly asks to import comparators or external evals. Phrases like "do the same for this paper" or "import the eval table" still mean **only the introducing model(s) on that paper**, not every column in the table. When skipping comparator columns, say so in the final report (e.g. `skipped external eval: RealBasicVSR scores from comparison table`). - For every source benchmark that survives the DB/metric filter, choose the highest score among eligible target rows using the configured leaderboard metric and direction. Do this per benchmark; do not assume the largest, newest, or most prominently named model is best everywhere. - If the best eligible row differs by benchmark because of mode, model size, backbone, compression level, decoding setting, tool use, or runtime setup, preserve that distinction in `model_name` or `harness` as appropriate. The same source paper can therefore produce evaluation rows with different model names across different benchmarks. - For multi-mode releases (e.g. Non-Think / High / Max / Thinking), use the best eligible mode per benchmark unless the source marks modes as incomparable for that leaderboard. If the source clearly presents one mode as the only intended comparable leaderboard setting, use that mode and mention the choice in `methodology`. - For multi-size product lines (e.g. Flash / Pro) or backbone/data-scale tables (e.g. ViT-B/32 vs ViT-L/14, 1B vs 2.5B), pick the benchmark-wise best eligible target variant by default. Import multiple rows for the same benchmark only when the user asks for family coverage or the source treats variants as separate primary models worth comparing side-by-side. - For base-model tables, exclude unless explicitly requested. - Which benchmark tables to consider (base model, instruct, frontier comparison, modes-comparison) and which surface they come from (card / paper / README). Ask the user once if anything is ambiguous; otherwise proceed with the defaults above and report exactly what was included and excluded at the end. ### 3. Resolve the source paper Every evaluation row attaches to a `papers` row, regardless of which surface the numbers came from. Resolve or create the right source row before posting evaluations. Two common cases (for external papers the `source_url` should be the paper PDF when one exists, with the project page as a project link; see `add-external-paper`): - ArXiv-backed: locate the paper row by `arxiv_id`. The release is usually already imported. - External tech report / model release (HF model card, HF PDF, blog, vendor page, GitHub release page): use the `add-external-paper` skill first to create or update a `source = 'external'` paper row keyed by the canonical release URL, then return here and attach evals to that row. **Always check HF model metadata `arxiv:` tags before falling back to an external row.** When the source is a HF model card (or a paper/README linked to a HF model), `uv run hf models info <repo_id>` returns a `tags` list that includes entries like `arxiv:2410.00037`. Each tag is the model's own declared canonical paper. Follow this resolution order: 1. Look up every `arxiv:` tag in the local `papers` table by `arxiv_id` (filtering on `is_latest_version`). If exactly one tag resolves to a local paper, use that paper as the source row and append the HF model URL (`https://huggingface.co/<repo_id>`) to its `hf_models` array for traceability. This is the common case — most flagship model cards list a single canonical paper — and it should be preferred over creating a new external row. 2. If multiple `arxiv:` tags resolve to local papers (ambiguous, e.g. a finetune that tags both its own paper and the base-model paper), pick the one the model card / README clearly identifies as the technical report for this release. If you cannot tell, stop and ask the user; do not guess. 3. If no `arxiv:` tag resolves to a local paper (none present, or all tags are unimported), fall back to the external-row path below. An ArXiv tag pointing at a paper we do not have is not, on its own, a reason to import a new ArXiv row in this skill — defer to the standard ingestion tooling and use an external row in the meantime if needed. Use an ArXiv row only when it is actually the source for the evaluated model and scores. Do not attach a model-card / README evaluation table to a related technique paper just because the source cites it. If a user says a release "should be external" but the source's own HF metadata, model search, or report clearly resolves to a local canonical ArXiv paper for the same release, use the ArXiv row and mention that choice rather than creating a duplicate external row. If the source is the primary surface for an unreleased or not-yet-paper-backed model, **pause this workflow and run the `add-external-paper` skill before any evaluation import**. Follow that skill's scope decision and metadata rules to create or update the external paper row with `source = 'external'`, a canonical `source_url`, title, abstract or source-grounded summary, `date_published`, official GitHub/project links, and full HF artifact URLs. After `add-external-paper` verifies the external row, resume this workflow using that row's `paper_id`. Do not post evaluations to a related paper, leave the source row missing, or invent a minimal paper record inside this skill. Always populate `papers.date_published` for the source row. Scatter plots use this date to place benchmark progress over time, so do not leave external source rows undated. For ArXiv rows, use the paper's publication date. For external reports/blogs/model cards/README releases, use the source publication date from the page metadata, visible page date, release notes, or trusted source metadata; if you cannot determine it confidently, stop and ask the user instead of importing undated evaluations. Look up the paper id with a read-only query, e.g.: ```python from sqlalchemy import text from backend.database.connection import get_engine with get_engine().connect() as conn: rows = conn.execute(text( "SELECT id, arxiv_id, source, source_url, title FROM papers " "WHERE arxiv_id = ANY(:arxiv_ids) OR title ILIKE :t " "ORDER BY date_published DESC NULLS LAST LIMIT 5" ), { "arxiv_ids": ["<arxiv id from HF tags>", "..."], "t": "%<model family>%", }).fetchall() for r in rows: print(dict(r._mapping)) ``` If no paper row exists for an ArXiv-backed source, use the standard ArXiv/import tooling or ask the user how to proceed; do not silently create an external duplicate for a real ArXiv paper. If no paper row exists for a non-ArXiv source, run `add-external-paper` immediately and only continue once it has created or reused a verified external row. Store HF artifacts as absolute URLs in `hf_models`, `hf_datasets`, and `hf_spaces`, not bare repo ids, because the frontend uses these fields directly as links. When a release has a canonical ArXiv-backed model search on Hugging Face, prefer that search URL in `hf_models` over enumerating every individual model repo if it better represents the linked model family. Use the form `https://huggingface.co/models?other=arxiv:<arxiv_id>` for a paper-scoped model list, especially for external rows that summarize a collection or release with several official variants. Use individual model URLs such as `https://huggingface.co/org/model` when the source row is truly about one specific model, or when the user explicitly asks to link the exact repos. ### 4. Inventory existing benchmarks For each benchmark name from the source, check the DB for an existing dataset, any existing task association, and existing leaderboard metrics. The semantic task page is not the comparability boundary for eval import: a result for `DocVQA` should attach to the already-configured `DocVQA` leaderboard even if that leaderboard currently lives under `Image Understanding` rather than a newer or more specific task. Use a single read-only query: ```python from sqlalchemy import text from backend.database.connection import get_engine source_benchmarks = ["MMLU-Pro", "GPQA Diamond", "HLE", ...] # exact labels from the source with get_engine().connect() as conn: