| name | skill-eval-improve |
| description | Improves Agent Skills via validate → rule-based eval cases → plugin-eval → prompt evals → bounded edits with held-out gates. Use when tuning skill quality, routing, or adopting Chrome/Microsoft T-named quality gates—not for bulk validate-only or SkillOpt automation. |
| license | MIT |
| type | governance |
| metadata | {"author":"skill-steward","version":"1.1.0","category":"marketplace"} |
| paths | ["skills/**/evals/**","scripts/eval-skill.mjs","scripts/eval-tiers.mjs"] |
Skill eval & improve
Improve skills measurably: baseline → measure → bounded edit → re-validate. Combine local tooling, Codex plugin-eval (when installed), and research-backed loops (SkillOpt).
When to use
- Skill triggers wrong or never loads (description routing)
- Bloated
SKILL.md, high token cost, weak outcomes
- After adding a new procedure—need regression checks
- Porting patterns from product MCP / plugin-eval research into Skill Steward skills
When not to use
- Bulk repo validation — e.g. “validate every skill in this repo” →
pnpm run validate only (skill-authoring-lifecycle for audit); do not start benchmark or SkillOpt loops.
- Automated SkillOpt / cluster training — Skill Steward documents a manual bounded-edit loop; no overnight optimizer pipeline.
- Creating a new skill — use skill-authoring-lifecycle first; eval-improve applies after a skill exists.
Cursor scope (optional): activate when editing under skills/** or scripts/validate-skills.mjs.
Mixture of experts (evaluation stack)
| Layer | Expert | Tool / method | Cost |
|---|
| 0 — Gate | Lint | pnpm run validate, skill-authoring-lifecycle | seconds |
| 0b — Rules | Routing/docs SSOT | pnpm run eval (T1 behavior-critical YAML cases) | seconds |
| 1 — Static | Structure | Codex plugin-eval analyze (if available) | seconds |
| 2 — Human | Behavior | 3–5 prompts with/without skill | minutes |
| 3 — Measured | Usage | plugin-eval benchmark + measurement-plan | minutes–hours |
| 4 — Evolve | Text optimization | SkillOpt-style bounded edits + held-out gate | hours |
| 5 — Navigate | Telemetry / dogfood | Current steward benchmark scenarios on compact traces | seconds |
Use the cheapest layer that answers the question. Do not skip layer 0.
Layer 0 — Skill Steward validator (always)
pnpm run validate
pnpm run validate:json
Fix all error: lines. Treat warn: (missing sources.md, long SKILL.md) seriously.
| Tier | Skills | CI |
|---|
| T1 — Behavior-critical eval-gated | Routing/procedure skills where drift can change agent decisions, claims, delegation, governance, or evidence boundaries | pnpm run eval + validate |
| T2 — Structural validate-only | All others | pnpm run validate |
T1 behavior-critical currently includes harness-engineering-lifecycle, mcp-harness-repo-maintainer, mixture-of-experts, multi-agent-handoff, plugin-marketplace-setup, repo-quality-system-lifecycle, repository-governance-lifecycle, skill-authoring-lifecycle, skill-eval-improve, steward-continuity-boundary-lifecycle, and vision-alignment-foresight. Each requires evals/cases/*.yaml (≥2) + references/evals.md. Schema: eval-case-schema.md.
Layer 0b — Rule-based cases (T1 behavior-critical CI)
pnpm run eval
pnpm run eval -- --skill mcp-harness-repo-maintainer
pnpm run eval:json
Chrome eval design (failure modes, rubrics, objective vs judge): references/chrome-eval-design.md.
CI does not run LLM judges. Subjective quality stays in references/evals.md (layer 2+).
Layer 1 — Codex plugin-eval (local)
When Codex plugin-eval is installed (~/.codex/plugins/.../plugin-eval):
plugin-eval start skills/<name> --request "Evaluate this skill." --format markdown
plugin-eval analyze skills/<name> --format markdown
plugin-eval explain-budget skills/<name> --format markdown
plugin-eval init-benchmark skills/<name>
plugin-eval benchmark skills/<name> --dry-run
Hand off rewrite plans to plugin-eval’s improve-skill skill after analyze --brief-out.
Details: references/plugin-eval.md.
Layer 2 — Human prompt suite (required for material edits)
- Write 3–5 representative user prompts (should trigger + should not trigger).
- Run agent without skill → record failures.
- Run with skill → record improvements and new failures.
- Mirror prompts in
evals/cases/*.yaml (CI rules) and references/evals.md (behavior log).
Split ~60% train (edit against) / 40% held-out (gate acceptance)—mirrors SkillOpt selection gate.
Layer 3 — SkillOpt-inspired improve loop (research)
SkillOpt treats SKILL.md as trainable text with a frozen agent:
Rollout (tasks + current skill) → Reflect (failures vs successes)
→ Bounded edit (add/delete/replace under budget) → Held-out gate (keep only if better)
Skill Steward manual adaptation (no GPU cluster required):
| Step | Action |
|---|
| 1 | Baseline: held-out pass rate without skill |
| 2 | With skill: same tasks, log pass rate |
| 3 | Reflect: list 1–3 concrete failure modes |
| 4 | Bounded edit: ≤10% line churn or one new section; no wholesale rewrite |
| 5 | Re-run held-out only; keep edit only if improved |
| 6 | Record outcome in references/evals.md + sources.md |
Paper: https://arxiv.org/abs/2605.23904 · Site: https://microsoft.github.io/SkillOpt/
Related: SkillLens (model-generated skills study).
Layer 4 — Ecosystem benchmarks (2026+)
| Resource | Use |
|---|
| SkillsBench | Inspiration for paired vanilla vs skill-augmented tasks |
| skillgrade | Regression testing skill quality (mgechev) |
| Claude authoring best practices | Eval-before-write workflow |
Layer 5 — Runtime Dogfood Benchmarks
At 10,000x scale, NLP prompt evaluation fails because LLMs suffer Cognitive Overload navigating massive toolsets. Runtime dogfood should objectively assert their logical trajectory using deterministic traces.
- Capture compact traces: Store action IDs, tool counts, artifact digests, and redacted excerpts, not raw product traces.
- Define assertions: Expected action trajectory, declared surfaces used first, maximum tool calls, maximum repair/setup attempts, maximum unrelated tool calls, required
return_to_goal_step, required artifacts, and negative checks for unrelated actions.
- Run dogfood benchmarks: Use
steward benchmark --scenario <id> --json for runtime dogfood scenarios. Do not put product runtime scenarios under T1 behavior-critical skill evals.
The current steward eval --name registered-eval path is legacy/experimental. Skill quality remains pnpm run eval; runtime dogfood belongs to steward benchmark, where durability_blocked is valid blocked evidence when contract inputs are modified or untracked, not proof of runtime behavior.
Improve workflow (checklist)
- [ ] sources.md cites plugin-eval + SkillOpt if used
- [ ] pnpm run validate
- [ ] T1 behavior-critical: `pnpm run eval` + cases updated
- [ ] plugin-eval analyze (optional)
- [ ] 3+ prompt evals documented in references/evals.md
- [ ] Bounded edit applied; held-out improved
- [ ] skill-authoring-lifecycle checklist
- [ ] PR mentions eval delta
What to fix first (typical order)
name / description (routing)—must include what + when- Broken links / missing
references/sources.md
- Delete or replace duplicated rules before adding a new section or eval case
- Move bulk to
references/ (SKILL.md < 500 lines)
- Add error-handling / validation steps agents skip
- Token cost (description length, always-loaded content)
Anti-patterns
- Rewriting entire SKILL.md from one failure (destroy working rules)
- Self-editing without held-out prompts (overfit)
- Adding skill rules, evals, or tools from one observed run when a smaller FAQ, error message, native command, observed-effect check, or deletion would solve the problem
- Adding a new eval for duplicated guidance before trying to compress, delete, or replace the overlapping rule
- Claims without
references/sources.md rows
- Evaluating only with static analyze—never running real prompts
- LLM judge in CI (flake, cost) — offline only per ADR 0011
- Passing
pnpm run eval and claiming agent behavior is proven
Related skills
| Skill | Role |
|---|
skill-source-citations | Save research links |
skill-authoring-lifecycle | Scaffold |
skill-authoring-lifecycle | Pre-merge audit |
Sources
See references/sources.md.
Install
npx skills add arenukvern/skill_steward --skill skill-eval-improve
