메뉴
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.
Use when the user wants to run the local multi-architecture transcription benchmark on FLEURS Latin-American Spanish + Brazilian Portuguese across every supported model — Whisper variants (including the Surus LATAM fine-tune), Voxtral, Qwen3-ASR, and Canary. All inference runs on the user's own hardware; no cloud APIs or paid endpoints. Covers single-model smokes, full-panel runs across the two-venv setup, and reading the comparative WER / CER table the runner produces. Triggers on: "run the transcription benchmark", "benchmark asr across models", "compare voxtral canary qwen whisper", "asr panel", "fleurs panel", "latam asr benchmark".
| name | submit-to-latamboard |
| description | Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor. |
Run a benchmark and publish your scores to latamboard.surus.lat via a PR.
You do not need a HuggingFace token. Scores go live automatically when your PR is merged.
benchy --help)github.com/surus-lat/benchyconfigs/models/ (see configure-model skill if not)benchy eval --config configs/models/<your-model>.yaml --tasks latam_board
For transcription models specifically:
benchy eval --config configs/models/<your-model>.yaml --tasks transcription
Note the run_id printed at the start of the run (e.g. 20260622_143022_LIMITED).
Smoke-test first if you want to verify before a full run:
benchy eval --config configs/models/<your-model>.yaml --tasks latam_board --limit 5 --exit-policy smoke
Proceed to full run only if run_outcome.status is passed or degraded.
python -m src.leaderboard.process_all --run-id <run_id>
This parses the benchmark outputs and generates outputs/publish/summaries/<model>_summary.json.
Check it worked:
cat outputs/publish/summaries/<model-name>_summary.json | python3 -m json.tool | head -20
You should see non-zero scores under categories.
python -m src.leaderboard.package_submission --run-id <run_id> --skip-process
This creates:
submissions/<run_id>/
run_manifest.json ← metadata: model list, date, benchy version
README.md ← exact commands to reproduce
models/
<model-name>/
run_outcome.json ← proof of execution
model_summary.json ← processed leaderboard entry
configs/
<model>.yaml ← the config used (reprex)
Multiple models in one PR: If you ran multiple models, package each run separately:
python -m src.leaderboard.package_submission --run-id <run_id_1> --skip-process
python -m src.leaderboard.package_submission --run-id <run_id_2> --skip-process
All of them go in the same PR.
git add submissions/<run_id>/
git commit -m "submission: <run_id> — <model-name>"
git push origin main
Then open a PR against surus-lat/benchy:main. Use the submission PR template — it includes a checklist for reviewers.
The PR description should include:
latam_board, transcription, etc.)The GitHub Action fires automatically. It:
model_summary.json from submissions/leaderboard_table.json back to HFYour model appears on latamboard.surus.lat on the next page load — no deploy needed.
process_all shows 0.0 scores for all tasks
The run likely used _adhoc_structured_*/main/ output layout (modern structured extraction runs). Check that your run's outputs/benchmark_outputs/<run_id>/<model>/ has the expected task directories.
package_submission says "No summary for "
Run Step 2 first (process_all), then package.
package_submission says "No config found for "
The model config in configs/models/ doesn't have a model.name matching the full HuggingFace path in run_outcome.json. Edit the config to set model.name: "org/ModelName" exactly as it appears on HuggingFace.
PR reviewer asks to reproduce They'll run:
benchy eval --config submissions/<run_id>/configs/<model>.yaml --tasks <task_group> --limit 5
Your submissions/<run_id>/README.md has the exact commands.
run_outcome.json shows status: passed or degraded (not failed)submissions/<run_id>/configs/ matches what was actually run