benchy

Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.

Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.

Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.

Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.

Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.

Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.

Use when the user wants to run the local multi-architecture transcription benchmark on FLEURS Latin-American Spanish + Brazilian Portuguese across every supported model — Whisper variants (including the Surus LATAM fine-tune), Voxtral, Qwen3-ASR, and Canary. All inference runs on the user's own hardware; no cloud APIs or paid endpoints. Covers single-model smokes, full-panel runs across the two-venv setup, and reading the comparative WER / CER table the runner produces. Triggers on: "run the transcription benchmark", "benchmark asr across models", "compare voxtral canary qwen whisper", "asr panel", "fleurs panel", "latam asr benchmark".

Use when the user wants to run Qwen3-ASR (any of the Qwen3-ASR-0.6B, Qwen3-ASR-1.7B, or Qwen3-ForcedAligner-0.6B models) locally or via the DashScope cloud API. Covers the right install (NOT transformers — the qwen-asr PyPI package), local Python usage, benchy integration via the qwen3_asr_chat adapter, the DashScope cloud alternative, the vLLM backend, and the version conflict with Voxtral. Triggers on: "run qwen3 asr", "qwen3-asr inference", "transcribe with qwen", "qwen-asr package", "qwen3 asr benchmark".

Run a Whisper-family transcription benchmark on a Mac, end-to-end and locally. Use when the user asks to benchmark/evaluate/compare ASR or speech-to-text models on FLEURS (Latin American Spanish, Brazilian Portuguese) using the `transformers_audio` provider in this repo. Covers setup, the model panel, Mac-specific MPS gotchas (the large-v3 wedge), result interpretation, and extension. Triggers on: "benchmark whisper", "run ASR panel", "compare STT models locally", "FLEURS transcription on Mac".

Use when proposing or reviewing a design that adds parts — classes, methods, abstractions, files, YAML fields, registries, lifecycle hooks. Especially when justifying additions as "for symmetry," "for flexibility," "we might need it," or "for cleanliness." Triggers on phrases like "let's add a base class," "I'll introduce a Capabilities dataclass," "we need a registry," "this should declare load/unload." Symptoms: every part is reasonable in isolation but the count keeps climbing; defenses cite "best practice" rather than a current concrete consumer; arguing a capability "wasn't really needed" to justify removing a part.

Add a new inference provider to benchy. Covers both OpenAI-compatible endpoints (prefer this path) and custom HTTP systems (SURUS-style). Includes config files, interface code, CLI wiring, capability declarations, and smoke-test verification. Use when asked to add a new model provider, API endpoint, or custom system.

Add a new benchmark task or task group to benchy. Covers handler selection, directory layout, metadata.yaml, task class implementation, __init__.py exports, and smoke-test verification. Use when asked to add a new evaluation task, benchmark, or task group.

Capture which AI system is being evaluated and write the target: section of benchmark.yaml. Stage 3a of the second-layer workflow. Use when a user wants to specify the model or API endpoint to benchmark.

Help the user choose how to grade the AI's output and write the scoring: section of benchmark.yaml. Use after define-task. This is Stage 2 of the second-layer workflow.

Collect what the AI system does and write the task: section of benchmark.yaml. Use when a user describes their AI in plain English and needs help defining a benchmark task. This is Stage 1 of the second-layer workflow.

Run benchy evaluations against models or systems. Covers the canonical smoke→full workflow, config selection, task filtering, exit policies, and reading run_outcome.json. Use when asked to evaluate, benchmark, or run benchy against a model or system config.

Read and interpret benchy run outputs. Covers run_outcome.json structure, status vocabulary, metric summaries, failure diagnosis, per-task/subtask breakdown, and actionable next steps. Use when asked to analyze, explain, or act on benchy evaluation results.

Translate benchmark results into plain English for non-developer users. Read run_outcome.json and run_summary.json and produce a human-readable summary with the 2-3 worst-performing samples and one concrete next step.

Run a complete benchmark from a benchmark spec end-to-end. Validates the spec, runs smoke test, then full run, and hands off to read-results. Use when the user has a complete benchmark spec and wants to evaluate their AI system.

Supply the benchmark test data and write the data: section of benchmark.yaml. Stage 3b of the second-layer workflow. Handles local files, format adaptation, and synthetic generation.

Generate synthetic benchmark examples from a task spec. Invoked from setup-data when there is no data, or directly when the user asks to generate examples. Produces a JSONL file in .data/<benchmark-name>/.

Pre-flight check for a benchmark spec before running. Runs benchy validate, reports errors in plain English, and guides the user to the right skill to fix each one. Use after the three definition stages and before run-benchmark.

Bidirectional algorithm for converting between a full working implementation and a battle-tested design plan. Use when: (1) you have a working implementation on a branch and need to produce a high-fidelity plan/design doc from it (Implementation → Plan), or (2) you have an oracle plan and need to execute it to a working implementation with zero wrong turns (Plan → Implementation). Triggers on: "extract the plan", "write the design", "implement from the plan", "oracle plan", "impl to plan", "document this branch".