Menü
Capture which AI system is being evaluated and write the target: section of benchmark.yaml. Stage 3a of the second-layer workflow. Use when a user wants to specify the model or API endpoint to benchmark.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Basierend auf der SOC-Berufsklassifikation
Add a new Together AI model to benchy and benchmark it on structured_extraction — smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring — copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR — actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately — no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.
| name | configure-model |
| description | Capture which AI system is being evaluated and write the target: section of benchmark.yaml. Stage 3a of the second-layer workflow. Use when a user wants to specify the model or API endpoint to benchmark. |
Captures which AI system to evaluate and writes the target: section of benchmark.yaml.
This is Stage 3a — can be run before or after setup-data.
The word "target" never appears in user-facing output. Use "your AI", "the system", "your API".
Present these options:
Path 1 — Your own API endpoint (HTTP) The user has a custom HTTP service.
Ask:
{{field}} placeholders)data, result.output)Path 2 — A specific model (OpenAI, Anthropic, Together, etc.) The user wants to test a named model.
Ask:
Path 3 — A local model (running on their machine) The user has a local OpenAI-compatible server (vLLM, llama.cpp, Ollama, etc.).
Ask:
| Placeholder | Meaning |
|---|---|
{{text}} | Input text from the dataset |
{{image_path}} | Path to image file |
{{image_path|base64_image}} | Image as base64 data URL |
{{field|json}} | Field value as embedded JSON |
Example: '{"document": "{{image_path|base64_image}}", "extract": true}'
| target.type | Benchy provider |
|---|---|
api | GenericAPIInterface |
model | OpenAIInterface (cloud provider) |
local | OpenAIInterface (local base_url) |
# Path 1 — Custom API
target:
type: api
url: https://api.example.com/extract
body_template: '{"image": "{{image_path|base64_image}}"}'
response_path: data # dot-notation; omit if the full response is the answer
name: my-pipeline-v1 # label for output folders
# Path 2 — Named model
target:
type: model
provider: together # openai | anthropic | together | google | alibaba
model: google/gemma-4-31B-it
system_prompt: "Extract invoice fields and return as JSON."
# Path 3 — Local server
target:
type: local
url: http://localhost:8000/v1
model: meta-llama/Llama-3.1-8B-Instruct
Remind the user to set the required environment variable in .env:
OPENAI_API_KEYANTHROPIC_API_KEYTOGETHER_API_KEYGOOGLE_API_KEYapi_key_env in target:Read the existing benchmark spec (e.g., benchmarks/my-benchmark.yaml). Update or add only the target: key. Leave all other sections unchanged.
After writing the target: section, tell the user:
Target configured. If you haven't set up your data yet, run
setup-datanext. Once both target and data are ready, runrun-benchmark.