メニュー
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.
Use when the user wants to run the local multi-architecture transcription benchmark on FLEURS Latin-American Spanish + Brazilian Portuguese across every supported model — Whisper variants (including the Surus LATAM fine-tune), Voxtral, Qwen3-ASR, and Canary. All inference runs on the user's own hardware; no cloud APIs or paid endpoints. Covers single-model smokes, full-panel runs across the two-venv setup, and reading the comparative WER / CER table the runner produces. Triggers on: "run the transcription benchmark", "benchmark asr across models", "compare voxtral canary qwen whisper", "asr panel", "fleurs panel", "latam asr benchmark".
| name | submit-to-latamboard |
| description | Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor. |
Run a benchmark and publish your scores to latamboard.surus.lat via a PR.
You do not need a HuggingFace token. Scores go live automatically when your PR is merged.
benchy --help)github.com/surus-lat/benchyconfigs/models/ (see configure-model skill if not)benchy eval --config configs/models/<your-model>.yaml --tasks latam_board
For transcription models specifically:
benchy eval --config configs/models/<your-model>.yaml --tasks transcription
Note the run_id printed at the start of the run (e.g. 20260622_143022_LIMITED).
Smoke-test first if you want to verify before a full run:
benchy eval --config configs/models/<your-model>.yaml --tasks latam_board --limit 5 --exit-policy smoke
Proceed to full run only if run_outcome.status is passed or degraded.
python -m src.leaderboard.process_all --run-id <run_id>
This parses the benchmark outputs and generates outputs/publish/summaries/<model>_summary.json.
Check it worked:
cat outputs/publish/summaries/<model-name>_summary.json | python3 -m json.tool | head -20
You should see non-zero scores under categories.
python -m src.leaderboard.package_submission --run-id <run_id> --skip-process
This creates:
submissions/<run_id>/
run_manifest.json ← metadata: model list, date, benchy version
README.md ← exact commands to reproduce
models/
<model-name>/
run_outcome.json ← proof of execution
model_summary.json ← processed leaderboard entry
configs/
<model>.yaml ← the config used (reprex)
Multiple models in one PR: If you ran multiple models, package each run separately:
python -m src.leaderboard.package_submission --run-id <run_id_1> --skip-process
python -m src.leaderboard.package_submission --run-id <run_id_2> --skip-process
All of them go in the same PR.
git add submissions/<run_id>/
git commit -m "submission: <run_id> — <model-name>"
git push origin main
Then open a PR against surus-lat/benchy:main. Use the submission PR template — it includes a checklist for reviewers.
The PR description should include:
latam_board, transcription, etc.)The GitHub Action fires automatically. It:
model_summary.json from submissions/leaderboard_table.json back to HFYour model appears on latamboard.surus.lat on the next page load — no deploy needed.
process_all shows 0.0 scores for all tasks
The run likely used _adhoc_structured_*/main/ output layout (modern structured extraction runs). Check that your run's outputs/benchmark_outputs/<run_id>/<model>/ has the expected task directories.
package_submission says "No summary for "
Run Step 2 first (process_all), then package.
package_submission says "No config found for "
The model config in configs/models/ doesn't have a model.name matching the full HuggingFace path in run_outcome.json. Edit the config to set model.name: "org/ModelName" exactly as it appears on HuggingFace.
PR reviewer asks to reproduce They'll run:
benchy eval --config submissions/<run_id>/configs/<model>.yaml --tasks <task_group> --limit 5
Your submissions/<run_id>/README.md has the exact commands.
run_outcome.json shows status: passed or degraded (not failed)submissions/<run_id>/configs/ matches what was actually run