-
Define success first (before touching the harness). Split the goal into four buckets and write them
down: outcome (what must be true of the answer), process (which tools/steps must run), style
(conventions/format), efficiency (token/step budget). If any bucket is unclear, ask the user.
-
Pick the local model + make it deterministic. Use OverfitClient.LoadGguf(path, ...) (or a running
overfit serve OpenAI endpoint). Grade with greedy decoding (temperature 0) or a fixed seed so runs
are byte-reproducible — this reproducibility is the whole point vs a cloud grader.
-
Build a small prompt dataset (10–20 cases). Each case: id, prompt, should_trigger, expected_checks[]
— the case declares WHICH named checks apply (dispatch them from a CHECK_REGISTRY keyed by id; this scales
rules without combinatorial explosion). Include negative controls (should_trigger=false) so you catch
over-triggering, not just failures. The skill description is the #1 lever on trigger accuracy — write
it in user-intent language, not API/mechanism language, and make trigger cases the bulk of the set.
Trigger-selection is testable deterministically with ToolCallConstraint (constrain generation to the
skill/tool-name enum → the model's pick is a hard, gradeable choice — a cloud grader can't do this reproducibly).
-
Run each case, capture the trajectory. For a plain prompt: ChatSession.Send(prompt) → capture the
text + ChatSession.LastStats (tokens, tok/s). For an agentic skill: drive it with ReActAgent and
capture the step/tool-call trajectory (the local equivalent of OpenAI's JSONL trace). Save each run's
artifact to disk so a failing case is inspectable.
-
Write deterministic graders as small C# predicates returning (id, pass, notes) — e.g.
output.Contains("..."), trajectory.Any(s => s.Tool == "search"), JsonDocument.Parse succeeds,
stats.Tokens <= budget. Keep them model-free (they mirror RagEvaluator/CorpusLinter in
Sources/Main/LanguageModels/Retrieval/Evaluation/ — reuse that style and its fast, deterministic ethos).
-
Add the rubric grader (model-assisted, schema-locked). Feed the run's output + a rubric to the judge
model with JsonSchemaConstraint bound to this schema so the reply is always parseable:
{ "type": "object",
"properties": {
"overall_pass": { "type": "boolean" },
"score": { "type": "integer" },
"checks": { "type": "array", "items": {
"type": "object",
"properties": { "id": {"type":"string"}, "pass": {"type":"boolean"}, "notes": {"type":"string"} },
"required": ["id","pass"] } }
},
"required": ["overall_pass","score","checks"] }
Prefer a stronger local model as the judge than the one under test.
-
Grade the EVAL itself — the overfitting check. A 95% pass rate proves nothing if the eval rewards the
agent for parroting the skill's phrasing. OverfittingJudge.Analyze(judge, skillMarkdown, rubricCriteria, cases)
classifies every rubric criterion (outcome / technique / vocabulary) and every declared check
(broad / narrow) → a 0..1 score + OverfittingSeverity (Low < 0.20 ≤ Moderate < 0.50 ≤ High).
Two questions drive it: would a domain expert who never read the skill fail this item (→ overfitted), and
does it test what the model already knows — common APIs, shell escaping, general best practice
(→ overfitted)? A genuinely novel technique with no practical alternative is NOT overfitting. The
classification vocabulary is pinned by a JsonSchemaConstraint string enum, so the judge cannot invent a
label or emit unparseable JSON — no parse-retry loop. It needs only the SKILL.md + eval definition (never a
run result) → one call per skill, parallelizable with the scenarios. Informational only — never gate on it
(thresholds are uncalibrated). Act on it by rewriting flagged items toward outcomes: "identified the root
cause" beats "measured cold/warm/no-op builds".
Three traps, all found by actually running it — not theory:
- Give it token headroom. The judge emits one entry per item; a long
reasoning × N items overruns
maxNewTokens, the JSON is truncated mid-string, and the entire assessment is discarded. A 4-criteria
eval needed ~3000 maxNewTokens on a 3B (1200 was silently not enough). Budget proportional to N.
Unknown is not Low. If the judge classifies nothing (truncated/unusable reply), you get score 0.0
OverfittingSeverity.Unknown — never Low. A 0.0 that means "we never assessed it" must never render
as a clean ✅. Items the judge refuses even after the repair pass come back as unclassified and are
excluded from the score (counting them as outcome would manufacture a clean eval).
- Judge size dominates quality. On a 3B judge, measured: it reliably catches literal "use this exact
phrase" traps, but it never once emitted
technique, and it false-positived a genuine outcome
("references the package and compiles") as vocabulary. Use 7B+ and treat a small-judge score as a
hint to go read the flagged items yourself — not as a number to track.
-
Aggregate + report. Per-case pass/score/checks + an aggregate (pass rate, mean score, budget stats).
Persist the report so successive runs are comparable — that comparability across runs is the deliverable
(a single score means nothing; the delta between runs is the signal).
-
Iterate. Change the skill/prompt → re-run → compare scores. Only accept a change when the score
strictly improves on a held-out slice (this selection gate is also the core of the planned SkillOpt
self-improvement loop — see ROADMAP.md).
