Menu
Supply the benchmark test data and write the data: section of benchmark.yaml. Stage 3b of the second-layer workflow. Handles local files, format adaptation, and synthetic generation.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.
| name | setup-data |
| description | Supply the benchmark test data and write the data: section of benchmark.yaml. Stage 3b of the second-layer workflow. Handles local files, format adaptation, and synthetic generation. |
Supplies the benchmark examples and writes the data: section of benchmark.yaml.
This is Stage 3b — can be run before or after configure-model.
Ask the user which situation applies:
Path 1 — Plug and run (user has a ready file) They have a CSV or JSONL with the right columns.
Required columns:
id — unique identifier (or auto-generated if missing)text — input to the AI (or image_path for image tasks)expected — ground truth / correct outputValidate: check that the file exists and has these columns. If something is missing, give a clear error:
"Your file is missing the 'expected' column. This should contain the correct output for each example."
Path 2 — Adapt format (user has data but in wrong shape) They have data but column names don't match.
Steps:
{their_column → required_name}Path 3 — Synthesize (no data)
They have no data — invoke synthesize-data.
Hard rule for image tasks: if task.input.type is image or document, block this path:
"I can generate expected outputs but not the images themselves. You need real images. Let me help you format them using Path 1 or 2."
For image tasks, the dataset needs image_path pointing to real image files. Guide the user to:
.data/my-benchmark/images/){"id": "1", "image_path": "images/doc1.jpg", "expected": {...}}.data/my-benchmark/train.jsonl# Local file (plug and run)
data:
source: local
path: .data/my-benchmark/train.jsonl
# HuggingFace dataset
data:
source: huggingface
dataset: org/dataset-name
split: test
# Synthetic (will be generated by synthesize-data)
data:
source: generate
count: 30
seed_description: "Realistic Argentine invoices with varying vendors and amounts"
Read the existing benchmark spec (e.g., benchmarks/my-benchmark.yaml). Update or add only the data: key. Leave all other sections unchanged.
After writing, run benchy validate to confirm all four sections are present and valid:
benchy validate --benchmark <path-to-spec>
After writing the data: section:
data.source: generate, tell the user to run synthesize-data to generate the examples.target: and data: are complete, tell the user they're ready to run run-benchmark.