Menu
Collect what the AI system does and write the task: section of benchmark.yaml. Use when a user describes their AI in plain English and needs help defining a benchmark task. This is Stage 1 of the second-layer workflow.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
| name | define-task |
| description | Collect what the AI system does and write the task: section of benchmark.yaml. Use when a user describes their AI in plain English and needs help defining a benchmark task. This is Stage 1 of the second-layer workflow. |
Collects the task definition from the user and writes the task: section of benchmark.yaml.
This is Stage 1 of 4. After completing this skill, run define-scoring.
Ask in plain English. Never mention handler classes, task types, or benchy internals.
Question 1 — What does your AI do? "Describe what your AI system does in one or two sentences."
Question 2 — What goes in?
Question 3 — What comes out?
Question 4 — If extraction: list the fields.
field_name: description (one per line)
vendor_name: Name of the seller
amount: Total invoice amount
date: Invoice date in YYYY-MM-DD format
Question 4 — If classification: list the possible labels.
positive
negative
neutral
| User says | task.type | output.type |
|---|---|---|
| Extracts fields | extraction | structured |
| Classifies | classification | label |
| Answers questions | qa | text |
| Translates | translation | text |
| Free text output | freeform | text |
| Image or PDF input | (any above) | + input.type: image |
Image input + extraction → structured task with multimodal_input: true in dataset_config.
This is NOT a separate handler — it is the structured type with image enabled.
# Extraction example
task:
type: extraction
input:
type: text # text | image | document
description: "Argentine invoice PDF"
output:
type: structured
fields:
- name: vendor_name
type: string
description: Name of the seller
required: true
- name: amount
type: number
description: Total invoice amount
required: true
- name: date
type: string
format: YYYY-MM-DD
required: false
# Classification example
task:
type: classification
input:
type: text
description: "Customer review"
output:
type: label
labels:
- positive
- negative
- neutral
# QA / Translation / Freeform
task:
type: qa # or translation, freeform
input:
type: text
description: "Question about a document"
output:
type: text
If a benchmark spec already exists (e.g., benchmarks/my-benchmark.yaml or benchmark.yaml at the root), read it and update only the task: section. If it does not exist, create it at benchmarks/<name>.yaml (or benchmark.yaml for a single-benchmark project) with only the task: and name: / description: fields populated.
benchmark:
name: my-benchmark
description: One-line description of what the benchmark tests
task:
# ... content from above
Leave scoring:, data:, and target: empty or absent — the other skills fill them in.
After writing the task: section, tell the user:
Task section written. Next: run
define-scoringto choose how to grade the output.
Based on SOC occupation classification
Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.
