post-leaderboard-result

name	post-leaderboard-result
description	Post a completed benchmark run to the leaderboard site (S3 trajectory upload + leaderboard.json entry) via a branch + PR. Triggers on: upload results, post to leaderboard, add this run to the board, publish this run.

Post a run to the leaderboard

Ships a completed results/<framework>/<uuid>/ run to nearai.github.io/benchmarks (scoreboard) and the S3 trajectory viewer. Two independently-loaded systems, both need updating; see CLAUDE.md ("Leaderboard Site") for the site's own build.

Hard rule: never push straight to main. Even though this only touches data files (leaderboard.json, optionally baselines/), it goes through a branch + PR like any other change — no exceptions, no "it's just JSON." Also never commit on top of whatever branch happens to be checked out; that may be someone's in-progress feature branch full of unrelated changes.

Prefer the automated workflow — you probably don't need AWS credentials

.github/workflows/post-leaderboard-result.yml does everything below (S3 upload, leaderboard.json splice, PR) via the same OIDC role CI already uses — no personal AWS access required, and it's triggerable by anyone with write access to nearai/benchmarks (same gate as /benchmark):

# 1. On a branch, stage the run's artifacts (NOT the gitignored results/ path):
mkdir -p .leaderboard-intake/<uuid>
cp results/<framework>/<uuid>/run.json      .leaderboard-intake/<uuid>/
cp results/<framework>/<uuid>/tasks.jsonl   .leaderboard-intake/<uuid>/   # optional
git add .leaderboard-intake && git commit -m "stage <suite> run for leaderboard" && git push -u origin <branch>

# 2. Dispatch the workflow — framework/suite/run_id are read from run.json itself:
gh workflow run post-leaderboard-result.yml --repo nearai/benchmarks \
  --ref <branch> -f run_dir=.leaderboard-intake/<uuid>

# Add -f official=true if this should be marked an official/curated row.

The workflow uploads to S3, splices the entry into leaderboard.json, removes the staging files, and opens (or comments on) a PR for review. Only fall back to the fully manual steps below if you specifically need something the workflow doesn't support yet (e.g. also adding a baselines/ entry — still a manual follow-up, see step 5).

Manual fallback

0. Gather inputs

Ask (or infer from context) for:

• framework — e.g. ironclaw, openclaw, hermes, claude-code
• suite — e.g. claw_swe_bench, pinchbench
• uuid — the run directory under results/<framework>/<uuid>/
• whether this should also become a baselines/ entry (a standing reference number, not just a one-off board row)

Sanity-check results/<framework>/<uuid>/run.json before doing anything else:

• model field — ironclaw runs sometimes record model="ironclaw" (the framework label) when --model wasn't passed through. Fix this to the real model id before posting, or the board will misgroup/mislabel the row.
• finished_at is set (run actually completed, not a partial/killed run).

1. Isolate the work

git fetch origin main
git worktree add -b leaderboard/<suite>-<framework>-<short-uuid> /tmp/lb-<short-uuid> origin/main
cd /tmp/lb-<short-uuid>

Do everything below inside that worktree.

2. Build trajectories + manifest (for the S3 viewer)

FRAMEWORK=<framework> SUITE=<suite> RUN_UUID=<uuid> \
  python3 scripts/format-trajectories.py \
    results/<framework>/<uuid>/tasks.jsonl \
    <path-to-diff.json-or-any-nonexistent-path-if-no-baseline> \
    /tmp/lb-<short-uuid>-traj

python3 scripts/write-manifest.py \
  --run-json results/<framework>/<uuid>/run.json \
  --out /tmp/lb-<short-uuid>-traj/manifest.json

(format-trajectories.py buckets into pass/fail when there's no diff/baseline arg to compare against — that's fine and the common case.)

3. Upload to S3

Bucket is s3://nearai-benchmark-runs (singular "benchmark", not "benchmarks" — easy to get wrong). This is a live, public-read bucket that other people's viewer links depend on — before syncing, double check the <framework>/<suite>/<uuid> path doesn't collide with an existing run, then confirm with the user before running the sync if there's any doubt about the inputs above.

aws s3 cp results/<framework>/<uuid>/run.json      s3://nearai-benchmark-runs/runs/<framework>/<suite>/<uuid>/run.json
aws s3 cp results/<framework>/<uuid>/tasks.jsonl    s3://nearai-benchmark-runs/runs/<framework>/<suite>/<uuid>/tasks.jsonl
aws s3 sync /tmp/lb-<short-uuid>-traj/ s3://nearai-benchmark-runs/runs/<framework>/<suite>/<uuid>/

Verify with a plain curl -I on the manifest URL → expect 200.

4. Hand-merge site/public/data/leaderboard.json

Never run build-data.ts raw and commit its output — it walks all of local results/, and many live board entries (older trajectory runs) have no corresponding local results//baselines/ dir anymore, so a raw regen silently drops them. Instead:

1. In a scratch scratchpad, run npx tsx site/scripts/build-data.ts to get a freshly-computed RunSummary for just this run (easiest: temporarily copy only this run's results/<framework>/<uuid>/ into an empty scratch results/ tree, or run the full build-data and pluck out the one object matching this uuid/run_id).
2. Splice that one object into the existing committed site/public/data/leaderboard.json's runs array — append, don't replace the array. Set has_trajectories: true on it so LeaderboardTable.tsx deep-links to the S3 viewer (/runs/<framework>/<suite>/<uuid>) instead of the local-only route.
3. If frameworks/models/suites/datasets registries in the same file don't already list this run's framework/model/suite/dataset, add entries (mirrors site/scripts/frameworks.json conventions).
4. Confirm the diff is clean and append-only: git diff --stat site/public/data/leaderboard.json should show insertions only, ideally 0 deletions.

5. Optional: add a baselines/ entry

Only if this run should become the standing reference number for <model> on <suite>/<framework> (not just a one-off board row).

<hash> is the short git commit hash from run.json's commit_hash field (7-8 chars, e.g. 2ebea05a) — check existing sibling dirs under baselines/<suite>/ for the exact slugging convention already in use (model name lowercased/dash-separated, framework only included when a suite has multiple frameworks, e.g. baselines/clawbench/qwen-qwen3-5-122b-a10b-ironclaw-2ebea05a/) and match it rather than inventing a new one:

baselines/<suite>/<model-slug>-<framework>-<hash>/run.json
baselines/<suite>/<model-slug>-<framework>-<hash>/tasks.jsonl   # skip if > 100MB (GitHub limit) — full data already lives on S3

6. Commit, push, open a PR — do not push to main

git add site/public/data/leaderboard.json baselines/ 2>/dev/null
git commit -m "leaderboard: post <framework>/<suite> run <short-uuid>"
git push -u origin leaderboard/<suite>-<framework>-<short-uuid>
gh pr create --repo nearai/benchmarks --base main \
  --title "leaderboard: post <framework>/<suite> run <short-uuid>" \
  --body "$(cat <<'EOF'
## Summary
- Adds <framework> <model> run <uuid> on <suite> to the leaderboard (pass_rate=..., avg_score=...)
- Trajectories synced to s3://nearai-benchmark-runs/runs/<framework>/<suite>/<uuid>/

## Test plan
- [ ] `leaderboard.json` diff is append-only (insertions only)
- [ ] Manifest URL returns 200
- [ ] Row renders on a local `npm run dev` and deep-links to the S3 viewer
EOF
)"

Report the PR URL back. Deploy triggers automatically once it's merged into main (deploy-site.yml on push touching results//baselines//site/; it ships the committed leaderboard.json directly — no build-data step in CI).

Clean up

git worktree remove /tmp/lb-<short-uuid>