Menรผ
Translate benchmark results into plain English for non-developer users. Read run_outcome.json and run_summary.json and produce a human-readable summary with the 2-3 worst-performing samples and one concrete next step.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fรผgen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prรผfen und installieren.
Basierend auf der SOC-Berufsklassifikation
| name | read-results |
| description | Translate benchmark results into plain English for non-developer users. Read run_outcome.json and run_summary.json and produce a human-readable summary with the 2-3 worst-performing samples and one concrete next step. |
Translates benchmark results into plain English.
This is a communication tool, not a failure-diagnosis tool.
interpret-run (developer skill) for deep failure diagnosis and debugging.outputs/benchmark_outputs/<run_id>/<model_name>/
โโโ run_outcome.json โ overall status and counts
โโโ run_summary.json โ per-task metric summary
โโโ <task>/
โโโ <subtask>/
โโโ *_metrics.json โ field-level scores
โโโ *_samples.json โ individual predictions
If no run_id is given, find the most recent run in outputs/benchmark_outputs/.
Translate run_outcome.status:
passed โ "Your benchmark completed successfully."degraded โ "Your benchmark completed with some issues โ most examples worked but some had problems."failed โ "Your benchmark encountered errors and could not complete."For extraction/structured tasks, translate field-level scores:
"Your system extracted vendor_name correctly 89% of the time. The weakest field was amount (61% correct)."
For classification:
"Your system classified correctly 74% of the time."
For freeform/QA:
"Your system matched expected answers 68% of the time (F1 score: 0.72)."
Read *_samples.json. Find the 2-3 samples with the lowest score. For each, show:
Input: [the actual input text]
Expected: [what the correct answer was]
Got: [what the AI actually returned]
Keep this concrete and readable. Truncate long inputs to ~200 characters.
Based on the results, give ONE actionable suggestion:
| Pattern | Suggestion |
|---|---|
| Score > 90% | "Excellent results. Consider testing on more diverse examples." |
| Score 70โ90% | "Good results with room to improve. The weakest field is [X] โ try adding clearer instructions in the system prompt." |
| Score 50โ70% | "Moderate results. The biggest issue is [X]. Consider improving your prompt or checking if the model understands the task." |
| Score < 50% | "The system is struggling. Check that the API is returning the right fields and that the expected outputs in your dataset are correct." |
connectivity_error | "The API wasn't reachable. Check the URL and any required authentication." |
no_samples | "No test data was found. Check the data path in your benchmark spec." |
Results: Invoice Extraction Benchmark
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโ๏ฟฝ๏ฟฝ๏ฟฝโโโโโโโโ
Overall: Completed successfully (89% average score)
Field breakdown:
โ vendor_name โ 94% correct
โ date โ 91% correct
โณ currency โ 78% correct
โ amount โ 61% correct
Weakest examples:
1. Input: "Factura Nยฐ 0001-00045678 - DISTRIBUIDORA OMEGA..."
Expected amount: 12500.00
Got: "12.500,00" โ number formatting issue
2. Input: "Invoice for services rendered in March..."
Expected amount: 875.5
Got: null โ field was missing from response
Next step:
The amount field has formatting issues (commas vs periods) and missing values.
Try adding this to your system prompt: "Always return amounts as decimal numbers
without thousand separators (e.g., 12500.00, not 12.500,00)."
diagnostic_classPlain-English summary โ no files written.
Add a new Together AI model to benchy and benchmark it on structured_extraction โ smoke first (~30-90s for iteration), then full run for the Part 3 submission. Use when a user wants to benchmark a Together AI model on extraction tasks, especially during the benchy benchmarks workshop.
Add a new benchy task with custom scoring โ copies src/tasks/_template_handler, writes a small Python handler that points at a HuggingFace dataset and declares its own metrics_config, then smoke-runs it. Use when a user wants to define their own structured-extraction benchmark from scratch, especially during the benchy benchmarks workshop.
Package a completed benchy run as a submission folder and open a PR against surus-lat/benchy. Stops at the PR โ actual leaderboard publish is handled separately by the workshop organizer because the extraction processors in src/leaderboard aren't fully modernized yet. Use during Part 3 of the benchy benchmarks workshop.
Run a benchmark with benchy and submit the results to LatamBoard via a GitHub PR. The PR merge triggers a GitHub Action that publishes scores to the HuggingFace dataset and makes them live on latamboard.surus.lat immediately. No HuggingFace token required from the contributor.
Publish new benchmark results from benchy to the LatamBoard leaderboard. Merges new model scores with the existing HuggingFace dataset and makes them live on latamboard.surus.lat immediately โ no frontend redeploy needed.
Re-run all models on the LatamBoard leaderboard from scratch after data loss. Identifies which model configs exist, runs each one with the full latam_board task suite on the cluster, and publishes results to HuggingFace after each model so progress is never lost.