| name | skill-selection-evals |
| description | Eval-only skill for measuring skill routing accuracy. Not invoked directly — contains selection evals that test whether the agent picks the correct skill for a given prompt. |
Skill-Selection Evals
This is not an executable skill. It contains evaluation data for measuring the accuracy of skill selection (routing) decisions.
Purpose
Crucible's 52 execution evals measure quality once a skill is invoked. Selection evals measure whether the right skill gets invoked in the first place.
Eval Types
- Direct selection: Given a prompt, does the agent pick the correct skill?
- Negative selection: Given a prompt that sounds like skill X but is not, does the agent avoid the false positive?
- Context-dependent: Same verb, different context, different correct skill.
- Cascade ordering: Multi-skill tasks requiring correct invocation order.
Boundaries Tested
- test-methodology — TDD vs test-coverage vs adversarial-tester
- review-direction — temper vs review-feedback
- adversarial-scope — red-team vs inquisitor vs audit vs siege
- completion-claims — verify vs finish
- bug-handling — debugging vs verify vs audit
- build-vs-raw-dispatch — build (full idea→PR pipeline) vs a single-skill dispatch (planning, test-driven-development, …)
- ui-implement-vs-verify — mock-to-unity (implement/fix UI from a mock) vs ui-verify (verify / drift-delta report)
- onboarding-entry — getting-started (silent conversation-start bootstrap) vs workshop (on-demand catalog tour) vs project-init (whole-repo codebase onboarding)
- explore-vs-improve — recon (task-scoped investigation) vs project-init (whole-repo + cross-repo onboarding) vs prospector (architectural-friction discovery)
Difficulty Ratings
Each eval is rated easy/medium/hard based on routing ambiguity. This enables stratified baseline measurement — distinguishing between improvements that lift hard cases (high value) vs confirming easy cases already work (low signal).
See Also
evals/evals.json — the eval data
GRADING.md — grading criteria and baseline measurement protocol