Menu
Run a complete benchmark from a benchmark spec end-to-end. Validates the spec, runs smoke test, then full run, and hands off to read-results. Use when the user has a complete benchmark spec and wants to evaluate their AI system.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Basé sur la classification professionnelle SOC
Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.
| name | run-benchmark |
| description | Run a complete benchmark from a benchmark spec end-to-end. Validates the spec, runs smoke test, then full run, and hands off to read-results. Use when the user has a complete benchmark spec and wants to evaluate their AI system. |
Runs a benchmark from a spec file end-to-end using the two-stage workflow.
Users never see --config, handler class names, or task group identifiers.
When the project has a single spec (or benchmark.yaml at the root), commands auto-discover it. When multiple specs exist, always pass --benchmark <path> explicitly.
Before running, validate the spec:
benchy validate --benchmark <path-to-spec>
If validation fails, show the errors and stop:
"Your benchmark spec has errors that need to be fixed before running: • [error 1] • [error 2]"
Guide the user to fix them using the appropriate skill (define-task, define-scoring, configure-model, setup-data).
benchy eval --benchmark <path-to-spec> --limit 5 --exit-policy smoke
Then read:
outputs/benchmark_outputs/<run_id>/<model_name>/run_outcome.json
Proceed to Stage 2 only if ALL of the following hold:
0run_outcome.status is passed or degradedrun_outcome.counts.failed_tasks == 0run_outcome.counts.error_tasks == 0run_outcome.counts.no_samples_tasks == 0If smoke fails, diagnose and fix before proceeding:
| Symptom | Likely cause | Fix |
|---|---|---|
no_samples | Dataset not found | Check data.path in the benchmark spec |
connectivity_error | API unreachable | Check target.url and network |
all_invalid_responses | Wrong response format | Check target.response_path |
json_parse_error | Model not returning JSON | Add a system_prompt to target: |
benchy eval --benchmark <path-to-spec> --exit-policy strict
outputs/benchmark_outputs/<run_id>/<model_name>/
├── run_outcome.json ← authoritative result
├── run_summary.json ← compact metrics
└── <task>/
└── <subtask>/
├── *_metrics.json
└── *_samples.json
Always parse run_outcome.json for pass/fail. Never rely on terminal output.
Hand off to read-results:
"The benchmark completed. Run
read-resultsto see a plain-English summary of the results."
Or invoke it directly by reading run_outcome.json and run_summary.json.
Same --run-id resumes a partial run — already-completed tasks are skipped:
benchy eval --benchmark <path-to-spec> --run-id my-run --exit-policy strict