Menu
Help the user choose how to grade the AI's output and write the scoring: section of benchmark.yaml. Use after define-task. This is Stage 2 of the second-layer workflow.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
| name | define-scoring |
| description | Help the user choose how to grade the AI's output and write the scoring: section of benchmark.yaml. Use after define-task. This is Stage 2 of the second-layer workflow. |
Helps the user choose a scoring method and writes the scoring: section of benchmark.yaml.
This is Stage 2 of 4. Requires task: section to already exist.
Present these choices. No math required from the user.
| What to say | scoring.type | When it makes sense |
|---|---|---|
| "Each correct field earns a point (score = correct / total)" | per_field | Extraction tasks with multiple fields |
| "All fields must be correct to earn a point (pass/fail)" | binary | When partial credit doesn't make sense |
| "Close enough counts (fuzzy match)" | semantic | QA, translation, summaries |
For classification tasks (task.output.type: label), always use binary (exact label match). Don't ask — just note it.
per_field:numeric_tolerance: 0.01case_sensitive: true/falsebinary:case_sensitive: true/falsesemantic:No follow-up params needed. F1 score handles variation automatically.
| scoring.type | Benchy internal |
|---|---|
per_field | StructuredHandler + field-level F1 / EQS metrics |
binary | ExactMatch |
semantic | FreeformHandler + F1Score + ExactMatch |
custom | user-provided Python function (advanced — only if explicitly asked) |
# Per-field (extraction)
scoring:
type: per_field
partial_credit: true
case_sensitive: false
numeric_tolerance: 0.01 # optional
# Binary (pass/fail)
scoring:
type: binary
case_sensitive: false
# Fuzzy (semantic)
scoring:
type: semantic
Read the existing benchmark spec (e.g., benchmarks/my-benchmark.yaml). Update or add only the scoring: key. Leave all other sections unchanged.
After writing the scoring: section, tell the user:
Scoring section written. Next: run
configure-modelto specify which AI system to evaluate, andsetup-datato provide the test examples. You can run either one first.
Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.