Menu
Analyze graded `trial_result` JSONL outputs from the eval pipeline, including reliability and custom slicing.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Interview the user relentlessly about a plan or design until reaching shared understanding, resolving each branch of the decision tree. Use when user wants to stress-test a plan, get grilled on their design, or mentions "grill me".
Use test-driven development for behavior-changing feature or fix work, and whenever the user mentions TDD, test-first, red-green-refactor, tracer bullets, integration tests, or public-interface behavior tests. Skip for docs-only, path-only rename, formatting-only, or purely mechanical chores unless explicitly requested.
Build command-only adapters for `eval` run mode using strict stdin/stdout JSON contracts.
End-to-end eval-suite orchestration with the `eval` command: run -> grade -> compare -> calibrate, using strict JSON contracts and schema discovery.
| name | compare-trials |
| description | Analyze graded `trial_result` JSONL outputs from the eval pipeline, including reliability and custom slicing. |
| license | ISC |
Use this skill for analysis over graded JSONL outputs, typically after:
agent-eval-harness eval '{"mode":"grade",...}' > graded.jsonl
For normal baseline-vs-challenger comparisons, use:
agent-eval-harness eval '{"mode":"compare",...}'
Use custom scripts only when suite-specific analysis is required.
When writing a custom script:
trial_result.pass and score non-null).taskId, trialIndex, runId).exactPassAtK naming for combinatorics-based pass@k metrics (do not label exact values as estimatedPassAtK).Reference helpers remain in references/ for bootstrap/statistics utilities.