메뉴
Analyze graded `trial_result` JSONL outputs from the eval pipeline, including reliability and custom slicing.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | compare-trials |
| description | Analyze graded `trial_result` JSONL outputs from the eval pipeline, including reliability and custom slicing. |
| license | ISC |
Use this skill for analysis over graded JSONL outputs, typically after:
agent-eval-harness eval '{"mode":"grade",...}' > graded.jsonl
For normal baseline-vs-challenger comparisons, use:
agent-eval-harness eval '{"mode":"compare",...}'
Use custom scripts only when suite-specific analysis is required.
When writing a custom script:
trial_result.pass and score non-null).taskId, trialIndex, runId).exactPassAtK naming for combinatorics-based pass@k metrics (do not label exact values as estimatedPassAtK).Reference helpers remain in references/ for bootstrap/statistics utilities.
Interview the user relentlessly about a plan or design until reaching shared understanding, resolving each branch of the decision tree. Use when user wants to stress-test a plan, get grilled on their design, or mentions "grill me".
Use test-driven development for behavior-changing feature or fix work, and whenever the user mentions TDD, test-first, red-green-refactor, tracer bullets, integration tests, or public-interface behavior tests. Skip for docs-only, path-only rename, formatting-only, or purely mechanical chores unless explicitly requested.
Build command-only adapters for `eval` run mode using strict stdin/stdout JSON contracts.
End-to-end eval-suite orchestration with the `eval` command: run -> grade -> compare -> calibrate, using strict JSON contracts and schema discovery.