skill-quality-reviewer

name

skill-quality-reviewer

description

Review ONE library skill's quality — the judgment layer above scripts/validate-skills.py. Runs the mechanical validator first (structure/lengths/registration/name collisions are its job — never re-checked), judges what it cannot: description trigger-oriented ("use when <situation>") vs merely descriptive; trigger collision with shipped skills, colliders NAMED (the main failure mode across the full shipped skill corpus); duplication better shipped as an EXTENSION (the LLM03/ASI04 pattern); evals that test real boundaries vs hollow filler; section substance (Stop Conditions that actually refuse); scope (one job); invocation posture (per the standard's §5 side-effect rule). Per-check PASS/CONCERN/FAIL with evidence; verdict ship/revise/reject/make-it-an-extension. Use when a skill is drafted, revised, or quality-audited. Do NOT use for a change's process compliance (agent-governance-audit), product code (code-reviewer, full-codebase-auditor), or a whole skill-adding PR (library-diff-reviewer).

Skill Quality Reviewer

Purpose

Give one skill — newly drafted or already shipped — the review the mechanical validator cannot perform. scripts/validate-skills.py proves a skill is structurally VALID; this skill judges whether it is GOOD: whether an AI matching a user request against the full shipped corpus of descriptions would pick it at the right moment and yield at the wrong one, whether it earns a place in the library instead of duplicating one, and whether its evals would actually catch it misbehaving. The deliverable is a per-check verdict with cited evidence and one overall recommendation a maintainer can act on: ship, revise, reject, or make-it-an-extension of a named existing skill.

Use When

• Use when: a new skill has been drafted and needs quality review before it enters the library.
• Use when: asked to audit an existing shipped skill's quality — trigger precision, overlap with newer skills, eval honesty, scope creep.
• Use when: a skill-adding change is in flight and the question is "is this skill good enough and distinct enough to ship?"
• Do NOT use when: the question is whether an agent CHANGE followed process — classification, approvals, merge authority, closeout — that is agent-governance-audit (different target: a change's process trail, not a skill definition's quality).
• Do NOT use when: reviewing product code — a diff is code-reviewer, whole-repo health is full-codebase-auditor. This skill's only target is a skill definition under .claude/skills/.
• Do NOT use when: auditing a skill-adding PR end-to-end (CI validator run, catalog integrity, branch/merge discipline) — that is library-diff-reviewer (D13, built D22). This skill is that review's inner loop: it judges ONE skill's quality; the PR-level mechanics belong to that skill.
• Do NOT use to re-run what the validator enforces: a validator FAIL goes back to the author against the standard — quality review never starts from a structurally invalid skill.

Inputs to Inspect

1. The target skill's full directory: SKILL.md (frontmatter and every section), references/*, evals/evals.json, and evals/trigger-evals.json when present.
2. docs/skill-generation-standard.md — the standard this review judges against (all eight sections).
3. Fresh output of python scripts/validate-skills.py — the mechanical gate this review builds on and never re-implements.
4. Every shipped skill's frontmatter description (.claude/skills/*/SKILL.md) — the corpus for the overlap sweep; the nearest neighbors read in full, not just their descriptions.
5. docs/skills-catalog.md — which overlap cluster the skill claims and which discrimination its neighbors already ship in their trigger-evals.
6. RESERVED_BUNDLED_NAMES in scripts/validate-skills.py — the validator blocks exact collisions; judgment must also catch confusably-close names.

Workflow

1. Gate on the mechanical validator. Run python scripts/validate-skills.py (or confirm a fresh pass on the exact revision under review). If it fails, stop and return the output — structure precedes judgment. Everything the validator enforces (frontmatter parse, name==directory, description present and < 1024 chars, no broad allowed-tools, < 500 lines, all nine sections present, evals JSON parse, catalog + README registration, duplicate and reserved-name collisions) is settled by that run and is NOT re-checked or re-scored below: a mechanical PASS is this review's entry ticket, never part of its verdict.
2. Check 1 — trigger quality. Is the description trigger-ORIENTED — does it state the situations that should invoke it ("use when …") and the near-misses that should not ("do NOT use …") — rather than merely descriptive ("this skill does X")? Trigger-oriented descriptions are what make auto-invocation work; a feature list with no situation is a FAIL even at 900 well-formed characters.
3. Check 2 — trigger overlap/collision (highest-value check). Sweep every shipped description and ask, per plausible user request: would an AI know which skill wins? NAME each colliding skill and quote the request that confuses them. Overlap without shipped discrimination (trigger-evals naming that exact neighbor) is a CONCERN at best, a FAIL when the loser would fire on the winner's core case.
4. Check 3 — duplication / extension. Does the skill substantially restate a shipped skill's job? When the genuinely new content would fit as a scoped additive diff to an existing skill, recommend make-it-an-extension and name the base — the library's precedent: LLM03 and ASI04 both landed as extensions of supply-chain-security-reviewer, not as clone skills.
5. Check 4 — eval integrity. Do evals/evals.json cases test the BOUNDARY: a real happy path, a genuine edge (not a happy path with an adjective), a should-not-trigger case a neighbor would actually contest, and a should-not-do/refusal case with teeth? Does trigger-evals.json discriminate against the RIGHT neighbors — the ones check 2 found, not a convenient strawman?
6. Check 5 — section substance. Are the nine sections filled or merely present to satisfy the validator? Especially: Stop Conditions are real halt/refusal conditions (who is refused, what triggers the halt), Use When is specific with counter-examples, Gotchas are earned failure modes rather than platitudes.
7. Check 6 — scope discipline. One job done well, or a catch-all that should split? Count the distinct jobs a stranger would say it performs.
8. Check 7 — invocation posture. disable-model-invocation: true iff the skill has side effects (edits files/config outside scratch, runs state-changing commands, calls networks, deploys, spends money); pure review/design/report skills stay auto-invocable — with the standard's single §5 approved-write exception: a skill that creates or appends a non-executable documentation/state file, only after exact preview and explicit, content-specific, single-use approval, stays auto-invocable. Manual-only skills must name the irreversible step in Stop Conditions.

Per-check criteria, the trigger-oriented-description rubric, the overlap sweep procedure, hollow-eval patterns, and the posture decision table: references/quality-review-checklist.md.

Output Format

SKILL QUALITY REVIEW — <skill-name>
Validator:   PASS (<run output line>) | FAIL → review stopped, output returned
Standard:    docs/skill-generation-standard.md; corpus swept: <N> shipped skills
Checks:
  1 Trigger quality        PASS | CONCERN | FAIL — <quoted description evidence>
  2 Overlap/collision      PASS | CONCERN | FAIL — <colliding skill(s) + the request that confuses them | "none found">
  3 Duplication/extension  PASS | CONCERN | FAIL — <overlapped skill + what an extension diff would contain>
  4 Eval integrity         PASS | CONCERN | FAIL — <hollow/missing case types, by case id>
  5 Section substance      PASS | CONCERN | FAIL — <present-but-empty sections, quoted>
  6 Scope discipline       PASS | CONCERN | FAIL — <the distinct jobs found, if more than one>
  7 Invocation posture     PASS | CONCERN | FAIL — <side effects found vs posture declared>
Recommendation: ship | revise | reject | make-it-an-extension of <base skill>
Colliding skills: <named list | none>
Required before ship: <specific revisions, each mapped to a check>
Not inspected: <what this review did not read, and why>

Validation Checklist

• Mechanical validator ran fresh (or a fresh pass was confirmed) BEFORE any judgment; no validator check was re-implemented or re-scored here.
• Every non-PASS verdict cites concrete evidence — quoted text, a named skill, an eval case id; no vibes-only CONCERN.
• The overlap sweep covered ALL shipped descriptions, not only the skill's own cluster or phase.
• Colliding skills are NAMED wherever check 2 or 3 is non-PASS.
• The recommendation is exactly one of the four verdicts; make-it-an-extension names its base skill.
• Nowhere does the report treat the mechanical PASS as evidence of quality.
• Not-inspected list present and honest.

Gotchas

• The rubber-stamp trap: "the validator passed" answers structure, not quality. The failure populations barely overlap — a perfectly structured skill can still be unpickable, colliding, hollow, or a duplicate.
• Descriptions written to impress rather than to trigger: capability lists with no situation. The working test: from this description alone, would a model know when this skill WINS a request and when it must YIELD?
• Overlap hides across phases: the confusable neighbor is often in a different pack (agent authority vs end-user RBAC; LLM04 vs ASI06 poisoning; evidence-policy vs manual-QA evidence). Same-cluster-only sweeps miss exactly these.
• Hollow evals look full: assertions that restate the Workflow verbatim, a should-not-trigger prompt nothing would ever route here anyway, an "edge" that is the happy path with a twist adjective, refusal cases the skill text never actually promises to refuse.
• Present-but-empty sections satisfy the validator: "Stop Conditions: ask the user when unsure" is a filled header with no refusal in it.
• A CONCERN pile-up is itself a verdict: several small concerns across trigger + evals + substance normally roll up to revise, not ship-with-nits.
• Invocation posture drifts on revision: a review skill that gains a "then apply the fix" step becomes side-effecting and must flip to manual-only — re-check posture on every re-review, not just at birth.
• Near-miss names pass the collision check: the validator blocks only EXACT reserved/duplicate names; code-review-helper next to the bundled code-review is a judgment finding, not a validator error.

Stop Conditions

• Asked to approve or ship a skill BECAUSE the mechanical validator passed → refuse the shortcut: mechanical pass is necessary, never sufficient. Run the judgment checks or return no verdict.
• Asked to soften or flip an evidence-backed FAIL, or to unname a colliding skill → refuse; the report states what the evidence states.
• The target skill's directory or SKILL.md is missing or unreadable → stop; there is no quality review of a skill from its description or from memory.
• The mechanical validator FAILS on the target revision → stop the quality review and return the validator output; structure precedes judgment.
• Asked to EDIT the skill to fix the findings → out of scope: this skill reviews and changes nothing. Hand the required-before-ship list to the author (or a build session) and re-review the revision.
• The standard and the validator disagree on a rule (one demands what the other ignores) → surface the discrepancy to a human; do not silently pick a side.

Supporting Files

• references/quality-review-checklist.md — per-check PASS/CONCERN/FAIL criteria, the trigger-oriented-description rubric with examples, the full-corpus overlap sweep procedure, the hollow-eval pattern catalog, and the invocation-posture decision table.
• evals/evals.json — behavior cases, including the descriptive-description-plus-collision → revise case and the no-rubber-stamp-on-mechanical-pass refusal.
• evals/trigger-evals.json — discrimination against agent-governance-audit, full-codebase-auditor, code-reviewer, and the library-diff-reviewer seam (built D22; owned from both sides).