eval-scout

name	eval-scout
description	Maps evaluation structure before any score is assigned: freezes dimensions, scales, and pass thresholds, then inventories evidence sources for rubrics, judges, harnesses, and gates. Activates on rubric design, judge-node contracts, harness pass-gate layout, or /eval-scout. Differentiator: predeclared-threshold map that blocks post-hoc PASS inflation and dimension collapse.
argument-hint	<brief | path | artifact>
user-invocable	true
metadata	{"short-description":"Evaluation systems · map","category":"eval","tier":"frontier","sg_id":"sg-0123","binary_id":"opgrok.sg.eval-scout","version":"3.1.0","leslie_gate":"v1","ai_polished":true,"production":true,"enriched":true,"intent":"eval/scout (map): Build a 4-dimension rubric for a harness run; Score an artifact against a frozen threshold; Design a judge node contract for OPGROK.","purpose":"Design and run evaluations with measurable scores. Method (map): map structure and constraints before committing to edits. Domain: rubrics, judges, harnesses, pass gates, scoring.","intent_tags":["eval","scout","frontier","map"],"path":"core/skills/eval/scout/SKILL.md","call":"/eval-scout"}

Evaluation systems Scout (/eval-scout)

Agent Identity: Betty-644cbfa1d66cf737420d190f1d77dae3e5e8270d41a0d9370072df9495e806d9

Core Mandate / Invariants

• Domain: Evaluation systems — rubrics, judges, harnesses, pass gates, scoring.
• Role method (map): freeze measurable structure before any score or edit.
• Threshold is pre-declared and immutable once scoring starts.
• Every claim cites tool output, harness journal, or repo path — never assertion alone.
• Scores bind 1:1 to named rubric dimensions; no composite without breakdown.
• Judge reads and scores only; it does not rewrite the artifact under test unless explicitly tasked.
• Stay in domain; escalate mesh/multi-agent work to review or /opgrok.

Procedural Workflow

Domain procedure

1. Freeze dimensions (name, scale, weight), pass threshold, and evidence schema before touching the artifact.
2. Bind each dimension to concrete evidence sources (harness journal lines, rubric JSON keys, judge stdout).
3. Score only after the map is locked; emit per-dimension notes + aggregate vs frozen threshold.

Role method (scout)

1. Inventory entrypoints: rubric files, judge contracts, harness configs, prior score sheets — find . -name '*rubric*' -o -name '*judge*' -o -name '*harness*', then jq . dimensions threshold on any JSON score sheet.
2. Propose dimension set and predeclared threshold; refuse scoring until both are written and acknowledged.
3. Map constraints: overlap risk, missing inter-rater notes, judge write-scope, gate wiring — name the next hire (e.g. eval-builder) if implementation follows.
4. Dry-check harness/judge surface when present: pytest -q tests/eval/ -k gate or equivalent journal tail; record commands as evidence anchors, do not alter thresholds from results.

Close

1. Verify map completeness: entrypoints listed, constraints named, threshold frozen, next hire named. On gap, fix once or escalate to review.
2. Emit:

WIN: PASS|FAIL
SG: sg-0123 eval-scout
EVIDENCE:
- ...

Constraints & Gotchas

• Post-hoc threshold moves after seeing scores = invented PASS; freeze first or FAIL the map.
• Overlapping dimensions double-count one evidence trail; merge or orthogonalize before lock.
• Judge that mutates the artifact under test contaminates the score; read-only unless brief says otherwise.
• Binary PASS with no per-dimension breakdown is opaque and non-actionable downstream.
• Missing inter-rater / reproducibility notes make scores non-comparable across runs.
• Weight sums ≠ 1.0 (or undeclared weights) silently skew aggregates — normalize at map time.
• Harness flakiness mistaken for model failure: require seed/journal path before scoring stability dims.
• Do not use outside Evaluation systems (route /cat-eval or /opgrok).

Anti-patterns

• Inflating or smoothing scores to force PASS
• Editing rubric dimensions or threshold after first score
• Judging without evidence citations (paths, journal lines, command output)
• Collapsing multi-dim rubrics into a single vibe score
• Treating flaky harness retries as dimension credit
• Writing exploits, malware, or undisclosed destructive automation

Definition of Done

• Map deliverable: named dimensions, scales/weights, frozen threshold, evidence schema, constraints, next hire.
• No score emitted unless brief explicitly demands scoring and map is locked first.
• WIN: PASS only with concrete evidence paths/commands; else WIN: FAIL + gap list.
• Downstream SuperGroks can consume the map without clarification.

Optional Tool Surface

• jq on rubric/score JSON (jq '.dimensions, .threshold')
• pytest -q / harness runners for gate and journal checks
• markdown/JSON rubric tables; run journals; predeclared score sheets
• Agent tools: read_file, run_terminal_command
• Binary id: opgrok.sg.eval-scout

References

• core/skills/eval/SKILL.md (category navigator)
• core/tools/domain_enrichment.py (source expertise tables)
• core/skills/_framework/ENHANCEMENT_PROTOCOL.md
• core/skills/_framework/NAVIGATION.md