Menu
Pre-flight check for a benchmark spec before running. Runs benchy validate, reports errors in plain English, and guides the user to the right skill to fix each one. Use after the three definition stages and before run-benchmark.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.
| name | validate |
| description | Pre-flight check for a benchmark spec before running. Runs benchy validate, reports errors in plain English, and guides the user to the right skill to fix each one. Use after the three definition stages and before run-benchmark. |
Checks that a benchmark spec is complete and ready to run. Use this after finishing the definition stages (define-task, define-scoring, configure-model/setup-data) and before run-benchmark.
benchy eval and wants to diagnose the specbenchy validate # auto-discovers if one spec exists
benchy validate --benchmark benchmarks/my-benchmark.yaml # explicit path
If the project has multiple specs and no --benchmark was specified, list the candidates and ask the user which one to validate.
✓ benchmarks/my-benchmark.yaml is valid and ready to run.
Tell the user: "Your benchmark spec is valid. You're ready to run a smoke test."
Suggest next step:
benchy eval --benchmark <path> --limit 5 --exit-policy smoke
✗ benchmarks/my-benchmark.yaml has N error(s):
• <error 1>
• <error 2>
For each error, translate it into plain English and point to the skill that fixes it:
| Error pattern | Plain English | Fix with |
|---|---|---|
task.output.fields required | "You haven't defined what fields to extract." | define-task |
task.type invalid | "The task type isn't recognized." | define-task |
target.url required | "Your API endpoint URL is missing." | configure-model |
target.model required | "No model has been specified." | configure-model |
data.path not found | "The data file doesn't exist at the path given." | setup-data or synthesize-data |
data.source invalid | "The data source type isn't recognized." | setup-data |
scoring.type invalid | "The scoring method isn't recognized." | define-scoring |
Tell the user which skill to use for each issue. Do not attempt to fix the spec directly unless the user asks — guide them to the right skill.