| name | diagnose-eval-failure |
| description | Explain WHY a model scored low on an eval — read the judge's reasons and the agent's trajectory, compare against the task rubric, and produce a rationale (capability gap vs. task/rubric problem). Distinct from infrastructure failures. Invoke when someone asks "why did the model fail this task?", "why is OutcomeValidity low?", "explain this low score", or "did the agent actually do the work?". |
Diagnose an eval failure
This skill answers a graded question: given a run that completed and was scored,
why did the model fall short? It reads the judge's stated reasons and the
agent's actual trajectory, lines them up against the task's rubric, and writes a
rationale. It explains; it does not fix the model.
It is not for infrastructure failures. A run that crashed, timed out, or
failed to provision never got a fair shot at scoring — those go to the recovery
router, not here.
Flow
1. Locate the run and confirm it was graded
Find the run's results.json under results/<run_…>/. Check status on the
record:
status: "success" → a real graded run. Continue.status: "failed" → infrastructure, not a model miss. The record is skipped
by scoring and carries no scores; a low/absent score here is not the model's
fault. Redirect to the recovery router in
known_issues.md and stop.
(An absent score on a success record means the metric didn't run for that
task, not that the model scored zero — see
metrics.md.)
2. Read the judge signal
In the record's scores map, read the judged entries ({score, success, reason}),
in the order that matters:
OutcomeValidity.reason — the headline "did it work" signal. Start here.ToolInvocation.reason — did it call the right tools / take a sane
trajectory (present only when MCP is on).Check: <item>.reason for each expected-output requirement, and
ChecklistScore.reason for the passed/total ratio in words.GroundingAccuracy.reason — reads "Applied X out of Y documented
constraints (Critical: a/b)"; a short critical count caps the band even when
the raw count looks fine.
Bare-number entries (DocRetrievalRate, ParameterRecallAccuracy, the chaos
performance figures) have no success flag — read the magnitude. Field shapes and
banding rules are in metrics.md.
3. Read what the agent actually did
Read the agent's trajectory / tools (the steps and tool calls it actually
made) and its output. Then read the task's expected_output rubric. Compare:
what did the agent attempt, and where did it diverge from the intended outcome?
4. Produce a rationale
Synthesize, don't just restate the reasons:
- What the agent attempted — the gist of its trajectory and final state.
- Where it diverged — the specific step or omission that lost the outcome.
- Which judge criteria it missed and why — tie each failed
reason back to
what the agent did (or skipped).
- Capability gap vs. task/rubric problem — decide which:
- Capability gap: the task is fair and single-correct-path-friendly, and the
agent genuinely couldn't do it. Say so plainly.
- Task/rubric problem: the rubric looks wrong, ambiguous, or single-path
(penalizes a valid alternative solution, demands a placeholder the task never
supplied, or grades for something the prompt didn't ask). Flag it and
recommend the
task-review skill — don't try to
fix the task here.
5. Report
Report: run id · task · model · the failed metrics with their reasons · your
rationale · the verdict (capability gap vs. rubric problem) · and the next action
(accept the result, or send the task to task-review).
Wrong tool?
- The run crashed / timed out / failed to provision (
status: "failed", or no
results.json) → infrastructure, not a model miss. Use the recovery router in
known_issues.md.
- The rubric itself is the problem → recommend
task-review;
this skill diagnoses, it does not edit tasks.
Guardrails
- Always check
status first — never read a failed record as a model miss.
- An absent score means the metric didn't run, not a zero.
- Explain only; this skill does not fix the model or edit the task.
- Quote the judge's
reason verbatim where it's load-bearing; don't paraphrase
away the signal.
