| name | leaderboard |
| description | Publish the nurb model leaderboard from merged benchmark submissions. Sanity-checks every run landed since the last regeneration, writes or refreshes the editorial verdicts, regenerates the benchmarks repo's REPORT.md and this repo's site/benchmarks.html, and opens the publish PRs. Use when the user says "update the leaderboard", "publish the benchmarks", "regenerate the benchmark page", or after merging submission PRs. |
nurb leaderboard
Two generated surfaces, one editorial layer, two repos. The benchmark (tasks, scorer, submissions, REPORT.md) lives in Shpigford/nurb-benchmarks; the public page (site/benchmarks.html, behind nurb.dev/benchmarks) lives here. Both surfaces regenerate mechanically from that repo's submissions/; the verdict sentences and subscription labels live in its src/nurb_evals/site.py and are written by a person. Submission PRs are pure additions and merge freely; nothing reaches the public page until this skill runs. That gap is deliberate: it is where the sanity check and the verdicts happen, so publishing is an editorial act, not a side effect of merging.
Step 0: Get the benchmarks checkout
Clone or update Shpigford/nurb-benchmarks somewhere outside this repo (for example ../nurb-benchmarks, or a temp directory), on its main. Everything below that touches submissions, verdicts, or REPORT.md happens in that checkout; only the final site/benchmarks.html lands here.
Step 1: Find what is new
The runs added since the page was last regenerated. In this repo, git log -1 --format=%ci -- site/benchmarks.html dates the last publish; in the benchmarks checkout, list the submission directories whose merge commits landed after it (git log --since=<date> --diff-filter=A --name-only -- submissions/ | grep results.jsonl).
Each directory under is one run: holding plus per-trial gzipped transcripts and part sources. If nothing is new, say so and stop.
