菜单
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.
Use when the user wants to run the local multi-architecture transcription benchmark on FLEURS Latin-American Spanish + Brazilian Portuguese across every supported model — Whisper variants (including the Surus LATAM fine-tune), Voxtral, Qwen3-ASR, and Canary. All inference runs on the user's own hardware; no cloud APIs or paid endpoints. Covers single-model smokes, full-panel runs across the two-venv setup, and reading the comparative WER / CER table the runner produces. Triggers on: "run the transcription benchmark", "benchmark asr across models", "compare voxtral canary qwen whisper", "asr panel", "fleurs panel", "latam asr benchmark".
| name | workshop-submit-to-latamboard |
| description | Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop. |
This skill walks a user through Part 3 of the benchy benchmarks workshop: packaging a Part 2 run into a submissions/<run_id>/ folder, committing it on a feature branch in the user's fork, and opening a PR against surus-lat/benchy:main.
Scope: this skill stops at the PR. The leaderboard publish (HuggingFace upload + latamboard.surus.lat refresh) is handled by the workshop organizer because the parse_model_results.py processors for extraction tasks haven't been updated to consume the current benchy output format. The PR captures the evidence; the organizer finishes the pipeline later.
Companion docs: .workshop/parts/part3-submit-to-latamboard.md. Verbose human reference: .agent/skills/submit-to-latamboard/SKILL.md.
Trigger phrases:
outputs/benchmark_outputs/w2_full_<slot>/<model>/run_outcome.json existsorigin git remote pointing at the user's fork of surus-lat/benchygh CLI authenticated (gh auth status)Pick the run to submit:
RUN_ID=w2_full_modelA
Package the run with --skip-process (the extraction processors aren't ready yet, so skip them):
.venv/bin/python -m src.leaderboard.package_submission \
--run-id ${RUN_ID} --skip-process
Inspect the submission:
ls submissions/${RUN_ID}/
cat submissions/${RUN_ID}/run_manifest.json
cat submissions/${RUN_ID}/models/*/run_outcome.json | jq '{status, duration_s, model}'
Expected files: run_manifest.json, README.md, models/<model>/run_outcome.json, configs/<model>.yaml.
Verify configs/<model>.yaml is the Together config, not a vLLM duplicate with the same model name:
grep -A2 "^together:\|^vllm:" submissions/${RUN_ID}/configs/*.yaml
If it picked the wrong one, replace it manually:
cp configs/models/together_<slot>.yaml submissions/${RUN_ID}/configs/
Commit on a feature branch and push to the fork:
git checkout -b submission/${RUN_ID}
git add submissions/${RUN_ID}/
git commit -m "submission: ${RUN_ID} — workshop attendee <name>"
git push -u origin submission/${RUN_ID}
Open the PR against surus-lat/benchy:main:
gh pr create --base main --repo surus-lat/benchy \
--title "submission: ${RUN_ID} — workshop attendee <name>" \
--body "Workshop submission. Task: structured_extraction. Model: <model-name>. Hardware: Together AI."
Capture the PR URL.
Report back:
submissions/${RUN_ID}/package_submission picks a vLLM config with the same model name: the script grabs any config whose model.name matches. Manually replace with the Together config in step 4.No summary for <model>: expected with --skip-process. The submission is valid without a model_summary.json — Part 3's scope is the PR, not the publish.gh pr create opens against the fork: always pass --base main --repo surus-lat/benchy explicitly.main.merge_and_publish --from-submissions can't find an extraction model_summary.json. That's expected; the organizer addresses it when modernizing the processors.The process_all script invokes per-task processors in src/leaderboard/functions/parse_model_results.py that look for legacy results_*.json files. Current benchy writes <model>_<ts>_metrics.json per subtask instead. Only the transcription processor was updated for the new format (commit 8ffbdcd). Extraction processors will need the same modernization before the workshop PRs can publish end-to-end. Until then, the workshop ships at "open the PR" and the organizer carries the publish.