Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.
Use when the user wants to run the local multi-architecture transcription benchmark on FLEURS Latin-American Spanish + Brazilian Portuguese across every supported model — Whisper variants (including the Surus LATAM fine-tune), Voxtral, Qwen3-ASR, and Canary. All inference runs on the user's own hardware; no cloud APIs or paid endpoints. Covers single-model smokes, full-panel runs across the two-venv setup, and reading the comparative WER / CER table the runner produces. Triggers on: "run the transcription benchmark", "benchmark asr across models", "compare voxtral canary qwen whisper", "asr panel", "fleurs panel", "latam asr benchmark".
Use when the user wants to run Qwen3-ASR (any of the Qwen3-ASR-0.6B, Qwen3-ASR-1.7B, or Qwen3-ForcedAligner-0.6B models) locally or via the DashScope cloud API. Covers the right install (NOT transformers — the qwen-asr PyPI package), local Python usage, benchy integration via the qwen3_asr_chat adapter, the DashScope cloud alternative, the vLLM backend, and the version conflict with Voxtral. Triggers on: "run qwen3 asr", "qwen3-asr inference", "transcribe with qwen", "qwen-asr package", "qwen3 asr benchmark".