| name | interpret-run |
| description | Read and interpret benchy run outputs. Covers run_outcome.json structure, status vocabulary, metric summaries, failure diagnosis, per-task/subtask breakdown, and actionable next steps. Use when asked to analyze, explain, or act on benchy evaluation results. |
Interpret Run Skill
Always parse JSON artifacts — never parse human logs. run_outcome.json is the source of truth.
Output Location
outputs/benchmark_outputs/<run_id>/<model_name_last_segment>/
├── run_outcome.json ← authoritative status
├── run_summary.json ← compact metric table
└── <task>/
├── task_status.json ← resume checkpoint
└── <subtask>/
├── *_metrics.json
├── *_samples.json
└── *_report.txt
model_name_last_segment = last path component of the model name (e.g. gpt-4o-mini).
run_outcome.json Top-Level Keys
{
"schema_version": 1,
"benchy_version": "...",
"model": "org/model-name",
"run_id": "my-run-2024",
"status": "passed",
"exit_policy": "strict",
"exit_code": 0,
"started_at": "...",
"ended_at": "...",
"duration_s": 42.3,
"counts": {
"total_tasks": 3,
"passed_tasks": 3,
"degraded_tasks": 0,
"failed_tasks": 0,
"skipped_tasks": 0,
"error_tasks": 0,
"no_samples_tasks": 0,
"pending_tasks": 0
},
"tasks": { ... },
"invocation": { "argv": [...], "cli_args": {...}, "cwd": "..." },
"git": { "repo": "...", "commit": "...", "dirty": false },
"artifacts": { ... },
"errors": []
}
Status Vocabulary
| Status | Meaning | Action |
|---|
passed | All samples processed, no errors | ✅ Proceed |
degraded | Partial errors (some samples failed) | ⚠️ Review error rate |
failed | All samples failed or no valid responses | ❌ Fix before proceeding |
skipped | Task skipped due to incompatibility | Check capability mismatch |
no_samples | Dataset empty or not found | Fix dataset path/download |
error | Pipeline-level exception | Check errors field |
pending | Task listed but never ran | Indicates interrupted run |
Overall Run Status Logic
Any task failed/error/pending → run status: failed
Any task degraded/skipped/no_samples → run status: degraded
All tasks passed → run status: passed
Agent Decision Rule (from AGENTS.md)
Proceed to full run only if ALL of these hold:
exit_code == 0
run_outcome["status"] in {"passed", "degraded"}
run_outcome["counts"]["failed_tasks"] == 0
run_outcome["counts"]["error_tasks"] == 0
run_outcome["counts"]["pending_tasks"] == 0
run_outcome["counts"]["no_samples_tasks"] == 0
run_outcome["counts"]["skipped_tasks"] == 0
Per-Task Breakdown
Each entry in run_outcome["tasks"]:
{
"status": "degraded",
"reason": "subtask_degraded",
"summary": {
"total_samples": 100,
"valid_samples": 92,
"error_count": 8,
"error_rate": 0.08,
"invalid_response_rate": 0.05,
"connectivity_error_rate": 0.03,
"response_rate": 0.92
},
"subtasks": {
"my_subtask": {
"status": "degraded",
"reason": "partial_errors",
"summary": { ... }
}
}
}
reason values
| Reason | Meaning |
|---|
subtask_failure | At least one subtask failed |
subtask_skipped | At least one subtask was skipped |
subtask_no_samples | At least one subtask had no data |
subtask_degraded | At least one subtask had partial errors |
partial_errors | Some samples failed but most succeeded |
all_connectivity_errors | 100% connectivity failures |
all_invalid_responses | 100% unparseable/wrong-format responses |
no_valid_samples | No samples produced valid output |
all_samples_failed | Error rate = 100% |
no_samples | Dataset empty |
incompatible | Task skipped due to capability mismatch |
Metric Interpretation
Key metrics in *_metrics.json
| Metric | Healthy range | Concern |
|---|
error_rate | 0.0 | > 0.10 is high |
connectivity_error_rate | 0.0 | Any > 0 means network/auth issues |
invalid_response_rate | 0.0 | > 0.05 means prompt/format issue |
response_rate | 1.0 | < 0.90 warrants investigation |
exact_match_rate | task-dependent | Compare to baseline |
partial_match_rate | task-dependent | Should be ≥ exact_match_rate |
field_f1_strict | 0.0–1.0 | Structured extraction quality |
eqs (Extraction Quality Score) | 0.0–1.0 | Composite extraction metric |
accuracy | 0.0–1.0 | MCQ classification accuracy |
Failure Diagnosis Workflow
1. Read run_outcome["counts"]
Identify which count is non-zero.
2. Look at run_outcome["tasks"]
Find the failing task(s), read their reason and summary.
3. Read run_outcome["errors"]
Structured error records with type, message, traceback.
4. Check per-subtask samples
For connectivity errors: *_samples.json → samples with error_type: "connectivity".
For invalid responses: samples with error_type: "invalid_response" show the raw model output.
5. Common fixes
reason | Fix |
|---|
all_connectivity_errors | Check API key env var, base URL, network connectivity |
all_invalid_responses | Check model name, api_endpoint mode (chat vs completions); for multimodal tasks: check image dimensions — add image_max_edge to provider config if endpoint has a px limit |
no_samples | Check dataset path, run download script |
incompatible | Check capability_requirements in metadata.yaml vs provider capabilities |
subtask_no_samples | One subtask missing data; check .data/ directory |
run_summary.json
Compact table of metrics per subtask. Useful for quick comparison across runs.
{
"model": "...",
"run_id": "...",
"tasks": {
"my_task.subtask": {
"accuracy": 0.85,
"total_samples": 100,
"error_rate": 0.02
}
}
}
Git Metadata
"git": {
"repo": "org/benchy",
"commit": "abc1234",
"dirty": false
}
dirty: true means local uncommitted changes — important for reproducibility.
Reading Samples for Debugging
When a task is degraded or failed, read the samples file.
Important: samples files are wrapped dicts {"model":..., "task":..., "samples":[...]}, not bare lists:
import json
with open("outputs/benchmark_outputs/<run_id>/<model>/<task>/<subtask>/*_samples.json") as f:
data = json.load(f)
all_samples = data["samples"] if isinstance(data, dict) else data
failures = [s for s in all_samples if s.get("error") or s.get("error_type")]
wrong = [s for s in all_samples if s.get("prediction") != s.get("expected")]
Resume a Failed Run
If the run was interrupted or you fixed an issue:
benchy eval --config <config> --run-id <same-run-id> --exit-policy strict
Tasks already marked passed or degraded in task_status.json are skipped.
