Menu
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Basé sur la classification professionnelle SOC
Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.
Use when the user wants to run the local multi-architecture transcription benchmark on FLEURS Latin-American Spanish + Brazilian Portuguese across every supported model — Whisper variants (including the Surus LATAM fine-tune), Voxtral, Qwen3-ASR, and Canary. All inference runs on the user's own hardware; no cloud APIs or paid endpoints. Covers single-model smokes, full-panel runs across the two-venv setup, and reading the comparative WER / CER table the runner produces. Triggers on: "run the transcription benchmark", "benchmark asr across models", "compare voxtral canary qwen whisper", "asr panel", "fleurs panel", "latam asr benchmark".
| name | push-to-latamboard |
| description | Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed. |
Publishes new benchmark results to latamboard.surus.lat.
benchy eval ← runs the model
└─ outputs/benchmark_outputs/<run_id>/<model>/
merge_and_publish.py ← leaderboard pipeline
├─ downloads existing scores from HuggingFace
├─ merges new results on top
└─ pushes merged data back to HuggingFace
HuggingFace dataset ← source of truth
mauroibz/leaderboard-results
└─ leaderboard_table.json ← one row per model
latamboard frontend (runtime fetch) ← live, no redeploy
└─ fetches leaderboard_table.json from HF on every page load
The frontend fetches leaderboard_table.json live from HuggingFace with
cache: 'no-store'. Once benchy pushes, the leaderboard updates on the
next browser refresh — no frontend build or deploy required.
One-time setup: add your HuggingFace token to .env in the benchy root:
echo "HF_TOKEN=hf_..." >> .env
The token needs write access to mauroibz/leaderboard-results.
Verify the dataset name in configs/config.yaml matches what the frontend
pulls:
# configs/config.yaml
datasets:
results: "mauroibz/leaderboard-results" # must match latamboard's fetch URL
Note: benchy was historically configured with
LatamBoard/leaderboard-results. The latamboard frontend actually fetches frommauroibz/leaderboard-results. If these differ, updateconfigs/config.yamlto match the latamboard frontend.
If you don't have a completed run yet, use the run-benchmark or evaluate
skill to produce one. The run ID is printed at the start of every benchy eval
run and also appears in outputs/benchmark_outputs/.
benchy eval --config <model-config> --tasks latam_board --run-id <run_id>
Confirm the run passed:
cat outputs/benchmark_outputs/<run_id>/<model_name>/run_outcome.json | python3 -m json.tool | grep '"status"'
# expected: "status": "passed"
python -m src.leaderboard.merge_and_publish --run-id <run_id>
This runs the full pipeline:
outputs/publish/summaries/leaderboard_table.json (all models, all columns)Expected output ends with:
🎉 Done! View the dataset at:
https://huggingface.co/datasets/mauroibz/leaderboard-results
If the run was already processed and outputs/publish/ is up to date:
python -m src.leaderboard.merge_and_publish --skip-process
Open latamboard.surus.lat and hard-refresh
(Cmd+Shift+R). The new model row should appear immediately.
To verify the HuggingFace side directly:
python3 -c "
import json
from huggingface_hub import hf_hub_download
p = hf_hub_download('mauroibz/leaderboard-results', 'leaderboard_table.json', repo_type='dataset')
data = json.load(open(p))
print([m['model_name'] for m in data])
"
Transcription results are fully supported. Scores are stored as
word_accuracy = 1 - WER (higher is better, consistent with the rest of the
leaderboard). Subtask columns are transcription_fleurs_es_latam and
transcription_fleurs_pt_br.
After pushing a transcription run, the latamboard frontend will:
Both happen automatically — no frontend deploy needed.
| File | Fetch method | Auto-updates? |
|---|---|---|
leaderboard_table.json | Live HF fetch on every page load | ✅ Yes — instant |
tasks_groups.json | Served from /public/ (build artifact) | ❌ Needs redeploy |
tasks_list.json | Served from /public/ (build artifact) | ❌ Needs redeploy |
You only need to redeploy the latamboard frontend when task definitions change (new tasks, new task groups, updated descriptions). For score updates alone, Step 2 is sufficient.
❌ HF_TOKEN not set — Add HF_TOKEN=hf_... to .env in the benchy root.
Model shows 0.0 scores — The run likely used the new _adhoc_structured_*/main/
output layout. Confirm with:
ls outputs/benchmark_outputs/<run_id>/<model_name>/
If you see _adhoc_structured_*/ dirs instead of structured_extraction/,
the processor should handle them automatically (fixed in commit 3c8bdd7).
Model appears with publisher: unknown — The run has no run_config.yaml.
Confirm run_outcome.json exists and contains a "model" key:
cat outputs/benchmark_outputs/<run_id>/<model_name>/run_outcome.json | python3 -m json.tool | grep '"model"'
If the key is present, the fallback in extract_model_info_from_config will
use it (fixed in commit 8740764).
Old models disappeared from the leaderboard — merge_and_publish merges
new results on top of the existing HF dataset; it never deletes old entries.
If scores vanished, the HF download step may have failed silently. Check the
output of Step 2 for ❌ lines.
Dataset name mismatch — If configs/config.yaml has LatamBoard/leaderboard-results
but the frontend fetches from mauroibz/leaderboard-results, pushes go to the
wrong dataset. Update configs/config.yaml to use mauroibz/leaderboard-results.