Menu
Generate synthetic benchmark examples from a task spec. Invoked from setup-data when there is no data, or directly when the user asks to generate examples. Produces a JSONL file in .data/<benchmark-name>/.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.
| name | synthesize-data |
| description | Generate synthetic benchmark examples from a task spec. Invoked from setup-data when there is no data, or directly when the user asks to generate examples. Produces a JSONL file in .data/<benchmark-name>/. |
Generates synthetic {text, expected} pairs from the benchmark spec's task spec.
Can be invoked from setup-data (when Path 3 is selected) or directly:
"Generate 30 examples for my invoice benchmark"
If task.input.type is image, document, or pdf — stop immediately:
"I can generate expected outputs but not the images themselves. You need real images to run this benchmark. Use setup-data (Path 1 or 2) to supply your own images."
Do not attempt generation. Return to the user.
task: section)from src.data_generator import generate_data
path = generate_data("<path-to-spec>", count=30)
# → .data/<benchmark_name>/train.jsonl
Or via CLI (if running as a subprocess):
python -c "from src.data_generator import generate_data; generate_data('<path-to-spec>', count=30)"
data: section to point to the generated file:data:
source: local
path: .data/<benchmark-name>/train.jsonl
Generated 30 examples → .data/my-benchmark/train.jsonl
task.type, task.input.description, task.output.fields from the benchmark spec.data/<benchmark-name>/train.jsonlThe generator model is an internal engine detail — never mention it to the user.
.data/<benchmark-name>/train.jsonl — one JSON object per line:
{"id": "0", "text": "Invoice from ...", "expected": {"vendor_name": "ACME Corp", "amount": 1250.00}}
{"id": "1", "text": "Factura de ...", "expected": {"vendor_name": "Distribuidora XYZ", "amount": 875.50}}
Tell the user:
"Generated {N} examples → .data/{benchmark-name}/train.jsonl. Your the benchmark spec has been updated. Run
run-benchmarkto evaluate."
If count < 10, warn:
"Note: {N} examples is a small dataset — results may not be statistically reliable. Consider generating more."