| name | eval-runner |
| description | Run eval scenarios to benchmark Mycelium effectiveness. Execute tasks using reflexion loop, validate against success criteria, record metrics. |
Eval Runner
Benchmark the agent's performance against defined scenarios. Adapted from n-trax eval system.
Commands
run <category/name>
- Read YAML from
.claude/evals/scenarios/<category>/<name>.yml
- Parse fields (name, category, task_prompt, success_criteria, budget)
- Execute setup steps if defined
- Record start time
- Execute task via reflexion workflow (read corrections first)
- Record end time and iteration count
- Validate ALL success criteria
- Write result JSON to
.claude/evals/results/<timestamp>-<name>.json
- Report summary
run-all [category]
- Glob
.claude/evals/scenarios/**/*.yml
- For each: run in isolation (git stash), record result, restore
- Aggregate and report
report
- Read all results from
.claude/evals/results/
- Generate summary table:
| Category | Pass Rate | Avg Iterations | Avg Time | Notes |
|-------------|-----------|----------------|----------|-------|
| discovery | ... | ... | ... | |
| delivery | ... | ... | ... | |
| integration | ... | ... | ... | |
| **Overall** | ... | ... | ... | |
- List failure patterns and recommendations
Result Format
See .claude/evals/schema.md for YAML scenario and JSON result formats.
Creating Scenarios
Place YAML files in .claude/evals/scenarios/<category>/. Define task_prompt, success_criteria, and budget.
