synthesize-data

Estrelas8

Forks0

Atualizado19 de junho de 2026 às 06:01

Generate synthetic benchmark examples from a task spec. Invoked from setup-data when there is no data, or directly when the user asks to generate examples. Produces a JSONL file in .data/<benchmark-name>/.

Instalação

Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.

Fonte

surus-lat

surus-lat/benchy

Abrir repositório GitHub Ver repositórios do creator

Download

Ocupações relacionadasSOC

Baseado na classificação ocupacional SOC

Desenvolvedores de softwareInformática e Matemática·SOC 15-1252

SKILL.md

readonly

name	synthesize-data
description	Generate synthetic benchmark examples from a task spec. Invoked from setup-data when there is no data, or directly when the user asks to generate examples. Produces a JSONL file in .data/<benchmark-name>/.

Synthesize Data Skill

Generates synthetic {text, expected} pairs from the benchmark spec's task spec.

Can be invoked from setup-data (when Path 3 is selected) or directly:

"Generate 30 examples for my invoice benchmark"

Hard Rule: Image Tasks

If task.input.type is image, document, or pdf — stop immediately:

"I can generate expected outputs but not the images themselves. You need real images to run this benchmark. Use setup-data (Path 1 or 2) to supply your own images."

Do not attempt generation. Return to the user.

Steps

Read the benchmark spec (must already have a valid task: section)
Check input type — block image tasks (see above)
Run the data generator:

from src.data_generator import generate_data

path = generate_data("<path-to-spec>", count=30)
# → .data/<benchmark_name>/train.jsonl

Or via CLI (if running as a subprocess):

python -c "from src.data_generator import generate_data; generate_data('<path-to-spec>', count=30)"

Update the benchmark spec's data: section to point to the generated file:

data:
  source: local
  path: .data/<benchmark-name>/train.jsonl

Confirm success:

Generated 30 examples → .data/my-benchmark/train.jsonl

What the Generator Does

Reads task.type, task.input.description, task.output.fields from the benchmark spec
Builds a generation prompt from the spec
Calls the generator model N times in parallel (async)
Validates each result against the field schema
Saves to .data/<benchmark-name>/train.jsonl

The generator model is an internal engine detail — never mention it to the user.

Counting

Default count: 30
User can specify: "Generate 50 examples"
Minimum useful: 10 (fewer makes metrics unreliable)

Output Artifact

.data/<benchmark-name>/train.jsonl — one JSON object per line:

{"id": "0", "text": "Invoice from ...", "expected": {"vendor_name": "ACME Corp", "amount": 1250.00}}
{"id": "1", "text": "Factura de ...", "expected": {"vendor_name": "Distribuidora XYZ", "amount": 875.50}}

After Generation

Tell the user:

"Generated {N} examples → .data/{benchmark-name}/train.jsonl. Your the benchmark spec has been updated. Run run-benchmark to evaluate."

If count < 10, warn:

"Note: {N} examples is a small dataset — results may not be statistically reliable. Consider generating more."

Mais deste repositório

mesmo repositório

workshop-benchmark-together-model

surus-lat/benchy

Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.

2026-06-228

workshop-define-benchmark

surus-lat/benchy

Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.

2026-06-228

workshop-submit-to-latamboard

surus-lat/benchy

Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.

2026-06-228

submit-to-latamboard

surus-lat/benchy

Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.

2026-06-228

push-to-latamboard

surus-lat/benchy

Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.

2026-06-228

rebuild-leaderboard

surus-lat/benchy

Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.

2026-06-218

name	synthesize-data
description	Generate synthetic benchmark examples from a task spec. Invoked from setup-data when there is no data, or directly when the user asks to generate examples. Produces a JSONL file in .data/<benchmark-name>/.

Synthesize Data Skill

Generates synthetic {text, expected} pairs from the benchmark spec's task spec.

Can be invoked from setup-data (when Path 3 is selected) or directly:

"Generate 30 examples for my invoice benchmark"

Hard Rule: Image Tasks

If task.input.type is image, document, or pdf — stop immediately:

"I can generate expected outputs but not the images themselves. You need real images to run this benchmark. Use setup-data (Path 1 or 2) to supply your own images."

Do not attempt generation. Return to the user.

Steps

Read the benchmark spec (must already have a valid task: section)
Check input type — block image tasks (see above)
Run the data generator:

from src.data_generator import generate_data

path = generate_data("<path-to-spec>", count=30)
# → .data/<benchmark_name>/train.jsonl

Or via CLI (if running as a subprocess):

python -c "from src.data_generator import generate_data; generate_data('<path-to-spec>', count=30)"

Update the benchmark spec's data: section to point to the generated file:

data:
  source: local
  path: .data/<benchmark-name>/train.jsonl

Confirm success:

Generated 30 examples → .data/my-benchmark/train.jsonl

What the Generator Does

Reads task.type, task.input.description, task.output.fields from the benchmark spec
Builds a generation prompt from the spec
Calls the generator model N times in parallel (async)
Validates each result against the field schema
Saves to .data/<benchmark-name>/train.jsonl

The generator model is an internal engine detail — never mention it to the user.

Counting

Default count: 30
User can specify: "Generate 50 examples"
Minimum useful: 10 (fewer makes metrics unreliable)

Output Artifact

.data/<benchmark-name>/train.jsonl — one JSON object per line:

{"id": "0", "text": "Invoice from ...", "expected": {"vendor_name": "ACME Corp", "amount": 1250.00}}
{"id": "1", "text": "Factura de ...", "expected": {"vendor_name": "Distribuidora XYZ", "amount": 875.50}}

After Generation

Tell the user:

"Generated {N} examples → .data/{benchmark-name}/train.jsonl. Your the benchmark spec has been updated. Run run-benchmark to evaluate."

If count < 10, warn:

"Note: {N} examples is a small dataset — results may not be statistically reliable. Consider generating more."