| name | judge |
| description | Have an LLM judge compare reference and generated solutions, scoring on API discovery, correctness, completeness, and functional correctness. |
| argument-hint | [project-directory] [--tests TC-001,TC-002] [--run runId] |
| disable-model-invocation | true |
| allowed-tools | Bash(agentic-usability *) Read Glob |
Judge Solutions
Run the judge stage. For each test case and target, an LLM judge compares the reference solution against the generated solution and produces scores.
echo "Arguments: $ARGUMENTS"
Options
--tests <ids>: Comma-separated test case IDs to judge--run <runId>: Target a specific run (default: latest)
Scoring Dimensions
Each test case receives scores on:
| Dimension | Range | What it measures |
|---|
apiDiscovery | 0-100 | Did the agent find the correct SDK endpoints/methods? |
callCorrectness | 0-100 | Are API calls constructed correctly (params, headers, body)? |
completeness | 0-100 | Does the solution handle all requirements? |
functionalCorrectness | 0-100 | Does the code run and produce correct output? |
overallVerdict | boolean | Does the solution actually work? |
notes | string | Brief explanation of scoring decisions |
Score Bands
- 0-20: Fundamentally wrong
- 21-40: Major issues, partially correct
- 41-60: Mostly correct with notable mistakes
- 61-80: Correct with minor issues
- 81-100: Excellent, matches reference
Judge Output
Written to results/<runId>/<target>/<testId>/judge.json:
{
"testId": "TC-001",
"target": "node-20",
"apiDiscovery": 85,
"callCorrectness": 90,
"completeness": 75,
"functionalCorrectness": 80,
"overallVerdict": true,
"notes": "Found correct APIs, minor parameter issue in error handling path"
}
DNF (Did Not Finish)
If the executor produced no solution, the judge writes an all-zero score:
{ "apiDiscovery": 0, "callCorrectness": 0, "completeness": 0, "functionalCorrectness": 0, "overallVerdict": false, "notes": "No solution produced (DNF)" }
Per-Test Judge Files
| File | Description |
|---|
judge.json | Full scoring result |
judge-cmd.log | Judge command executed |
judge-output.log | Raw judge stdout/stderr |
judge-session.jsonl | Judge conversation log (if available) |
judge-egress.log.json | Judge network traffic |
judge-error.log | Error (only on failure) |
Progress Tracking
Tracked in results/<runId>/pipeline-state.json:
completed.judge["<target>"] lists judged test IDs- State saved after each test — safe to interrupt and resume
Run agentic-usability judge -p $ARGUMENTS and report the results.
For detailed internals, see pipeline-guide.md.
