| name | eval-report |
| description | Generate evaluation report from session metrics, checkpoint evals, and pass@k data |
| disable-model-invocation | true |
Eval Report
Generate a dashboard of session effectiveness metrics.
Arguments
--days N — look back N days (default: 7)--branch <name> — filter by branch--format table|json — output format (default: table)
Process
1. Read Metrics
Read .claude/agent-memory/session-learnings/metrics.jsonl
Guard: If metrics.jsonl does not exist, output "No session metrics recorded yet. Run sessions with evaluate-session.sh enabled to populate." and skip to step 2.
Calculate:
- Total sessions in period
- Average commits per session
- Average tasks completed per session
- Average files changed per session
- Error rate (sessions with errors / total sessions)
- Average session duration (minutes)
2. Read Checkpoint Evals
Read .claude/phases/output/eval-*.json
Calculate:
- Phase transition pass rate
- Most common failure criteria
- Average attempts per transition
3. Read pass@k Data
Read .claude/instincts/eval-tracker.jsonl
Calculate:
- pass@1: (tasks succeeded on first attempt) / (total tasks)
- pass@3: (tasks succeeded within 3 attempts) / (total tasks)
- Average attempts to success
4. Skill Effectiveness Analysis
Read .claude/instincts/eval-tracker.jsonl and analyze skill_context field:
For each pattern ID that appears in any eval-tracker entry:
- Count tasks where this pattern was active
- Calculate pass@1 rate for tasks with this pattern
- Compare to overall pass@1 rate
Display:
| Pattern | Tasks | pass@1 (with) | pass@1 (overall) | Verdict |
|---------|-------|--------------|-----------------|---------|
| inst_abc123 | 5 | 80% | 60% | effective |
| inst_def456 | 3 | 33% | 60% | harmful |
Verdicts:
- "effective": pattern pass@1 > overall + 10%
- "neutral": within 10% of overall
- "harmful": pattern pass@1 < overall - 10%
Flag harmful patterns: "Consider reducing confidence or removing: "
5. Output Report
Display as markdown table:
| Metric | Value | Trend |
|---------------------|-------|--------------|
| Sessions (7d) | N | - |
| Commits/session | N.N | up/down/flat |
| Tasks/session | N.N | up/down/flat |
| Error rate | N% | up/down/flat |
| Avg duration (min) | N | up/down/flat |
| Phase pass rate | N% | up/down/flat |
| pass@1 | N% | up/down/flat |
| pass@3 | N% | up/down/flat |
Trend: compare current period to previous equivalent period.
Data File Locations
- Metrics:
.claude/agent-memory/session-learnings/metrics.jsonl
- Checkpoint evals:
.claude/phases/output/eval-*.json
- pass@k tracker:
.claude/instincts/eval-tracker.jsonl
