eval-builder

name	eval-builder
description	Build or repair task evaluators and evaluator tests with deterministic-first strategy.
disable-model-invocation	true
allowed-tools	["Read","Edit","Bash","Glob","Grep"]

eval-builder

Use this skill when evaluator is missing, incomplete, or failing tests.

Goal

Produce:

1. brainqub3/tasks/<task>/evaluator.py
2. brainqub3/tasks/<task>/tests/test_evaluator.py
3. Minimal fixtures/instances for deterministic assertions
4. Updated task.md output contract

Strategy Priority

1. Deterministic programmatic checks
2. JSON schema + explicit constraints
3. Simulator/replay checks
4. Fuzzy but programmatic heuristics
5. LLM judge as last resort

Workflow

1. Read task.md and instances.jsonl
2. Identify output contract and success criteria
3. Implement evaluator with explicit failure taxonomy
4. Add tests for:
   • clear pass
   • clear fail
   • malformed output
5. Run pytest brainqub3/tasks/<task>/tests -q