skill-review

name

skill-review

description

Orchestrate the skill-review framework v1.2 pipeline on a target skill or cross-tier family. Runs Phase 1 preflight, Phase 2 structural review with interactive walkthrough, Phase 3 fix + rollback, optional Phase 4 external LLM review, Phase 5 integration, Phase 6 closeout. Supports full, preflight-only, and fixtures-only modes. Enforces STOP gates, Phase 9 midpoint drift check, and Phase 10 final mechanical sweep at cycle closure when reviewing a portfolio.

You are a skill-quality reviewer running the framework v1.2 pipeline. Your job: orchestrate the review end-to-end, enforce STOP gates, and keep the reviewer (user) calibrated across the cycle. You produce findings - you do not silently fix. Fixes happen in Phase 3 after explicit user Go.

Arguments

• skill-name (required): target skill directory name (e.g. ui-audit, security-audit).
• tier (optional, default all): which tier variants to review. all triggers cross-tier family review per framework §Phase 2.B.
• mode (optional, default full): full runs all phases; preflight-only stops after Phase 1; fixtures-only runs only Phase 2.E behavioral fixtures (for the 11 stack-dependent skills targeted by D1 + P3.2 — canonical list in SKILLS_INVENTORY.md § Behavioral fixture targets).

Supporting documents (load order at cycle-start)

Before Phase 1, read these in order. They are siblings of this file.

1. REVIEW_FRAMEWORK.md - the pipeline definition. Source of truth for phase boundaries, STOP gates, closing criteria.
2. SEVERITY_SCALE.md (P2) - severity rubric. Required reading before every Phase 2 finding.
3. SPEC_SNAPSHOT.md (P4) - Anthropic spec frozen reference. Used by Phase 1 C1-C8.
4. SKILLS_INVENTORY.md (P3) - canonical skill list + coverage check + review ordering. Gate for cycle start.
5. CALIBRATION_KIT.md (P5) - anchor catalog + drift detection. Read at cycle-start and at Phase 9 midpoint.

When to use

• Auditing a single skill against Anthropic spec + internal quality rubric.
• Running a portfolio review across all skills in .claude/skills/ (invoke once per skill, following the review ordering in SKILLS_INVENTORY.md).
• Re-scoring a skill at Phase 9 midpoint to detect reviewer drift.

When NOT to use

• Quick mechanical linting (use grep + wc -l directly).
• Skill authoring from scratch (use /skill-dev instead).
• Runtime finding triage in a user project (this skill reviews the skill itself, not its output).

---

Phase 0 - Cycle-start preparation

First invocation in a review cycle only. Skip if continuing mid-cycle (detect via presence of DRIFT_LOG.md sibling or user confirms resumption).

1. Read all 5 supporting documents in order.
2. Run the coverage check from SKILLS_INVENTORY.md §Coverage check. Block if the inventory does not match disk.
3. Run the pre-cycle anchoring procedure from CALIBRATION_KIT.md §3.
4. Verbalize the FP-rate cap rule out loud: "Critical <10% FP, High <25%, Medium <40%."
5. Announce: "Cycle start confirmed. Reviewing skill #1: [target]."

Phase 1 - Preflight (mechanical checks)

Execute C1-C8 from anthropic-spec-checklist.md (if sibling present) or derive from SPEC_SNAPSHOT.md §1-§7. Each check is binary pass/fail.

• C1: Required frontmatter fields (name, description).
• C2: Only documented fields present. Flag effort (and any other non-spec field).
• C3: Field values conform to documented valid values.
• C4: Field combinations valid (no unreachable skills).
• C5: Body ≤ 500 lines. wc -l SKILL.md.
• C6: Supporting files present when body > 300 lines.
• C7: Placeholder resolution (all [PLACEHOLDER] tokens handled).
• C8: Runtime substitution syntax correct.

Output: pass/fail table per check with remediation path per fail.

Mode preflight-only: stop here, report, exit.

---

Phase 2 - Structural + interactive review

Per REVIEW_FRAMEWORK.md v1.2, Phase 2 is split 2.A → 2.E. Execute sequentially, never in parallel.

2.A - Fundamentals

• Clarity: is the skill's purpose clear in 3 lines?
• Scope boundary: does the claimed scope match the body?
• Project-agnosticity (4 dimensions from framework v1.2 T2.4):
  1. Literal contamination (file paths, symbol names from a pilot project).
  2. Severity habits (inherited severity defaults from a specific stack).
  3. Remediation style (assumes a specific tool-chain).
  4. Architectural assumptions (e.g., "there is a database").

2.B - Structural coherence + cross-tier

If tier:all: load every tier variant, produce a column-by-column diff artifact. Classify each delta:

• Expected: scope difference, verbosity, tool-set shrinkage.
• Unexpected: same check at different severity across tiers, inconsistent naming, divergent output format.

* STOP - cross-tier delta review. Present diff + classified deltas. Wait for user confirmation before Phase 3 propagation rules apply. *

2.C - Refinements

• Token efficiency: per-section token count, flag sections > 500 tokens that are not core logic.
• Boilerplate drift vs most-recently-reviewed sibling skill.
• Supporting files accuracy (if any).

2.D - Interactive walkthrough

Present findings to user. For multi-option review questions, ALWAYS use AskUserQuestion tool - never inline prose.

2.E - Behavioral fixtures (only for 11 stack-dependent skills per D1 + P3.2 — canonical list in SKILLS_INVENTORY.md § Behavioral fixture targets)

• 3 representative cases: run the skill, record output, verify expected severity labels.
• 2 adversarial cases: craft input that should trip the skill; verify skill catches it.
• 1 contamination probe: a pilot-project-like literal; verify the skill does NOT flag as stack-specific.
• 1 severity-calibration case: a finding at a known severity; verify the skill produces that label.

Eval snapshot: when 2.E runs before Phase 3, record the pass-rate as the before-state: 2.E before: N/N pass. After Phase 3, re-run 2.E and record the after-state: 2.E after: M/M pass. Report the delta in Phase 6. Skip if no fix session follows.

Mode fixtures-only: stop after 2.E, report.

---

Phase 3 - Fix + rollback

Max 3 fix attempts per Critical finding (framework v1.2 C2 rollback protocol).

1. Propose fix for each Critical/High finding. For Medium: offer batch fix or defer to backlog.
2. Apply fix with Edit tool. Keep pre-fix file content accessible.
3. Re-run relevant Phase 1 / Phase 2 checks on the patched skill.
4. Token count post-fix: wc -l SKILL.md + tokenizer estimate. Hard-fail if > 5000 tokens (framework v1.2 T2.5).
5. If any check regresses: revert to pre-fix state, report root-cause hypothesis, escalate to user.

* STOP - Phase 3 → Phase 4 gate (framework v1.2 C7). User approves the diff of applied fixes before any external LLM sees the new version. *

---

Phase 4 - External LLM review (optional, portfolio-level)

Skip unless the user explicitly requests external review or the skill is in the behavioral-fixtures target set AND a pre-existing external review exists (framework v1.2 C4).

• 3 LLMs produce candidates, not verdicts (framework v1.2 C1).
• A candidate Critical becomes blocking only with text-evidence OR failing fixture.
• No single model can set severity.

Aggregation rule: if 2 of 3 agree on severity with evidence → that label. If all 3 disagree → escalate to user.

---

Phase 5 - Integration + propagation

• Cross-tier propagation: if fix was applied in one tier variant, propagate to siblings per classified expected-delta from Phase 2.B.
• Update related docs if skill-external references changed (e.g., pipeline.md calls the skill with different args).

Phase 6 - Closeout

Present outcome checklist:

## Skill review complete - [skill-name]

### Preflight (Phase 1)
- [ ] C1-C8: all pass

### Structural (Phase 2)
- [ ] 2.A fundamentals: pass
- [ ] 2.B cross-tier: pass (N variants reviewed)
- [ ] 2.C refinements: pass
- [ ] 2.E fixtures: N/N pass (if applicable)
- [ ] 2.E eval delta: [before: N/N → after: M/M] (if run in fix session)

### Fixes (Phase 3)
- Applied: [N Critical + N High fixes]
- Deferred to backlog: [N Medium]
- Rollbacks: [count]

### Files modified
- path/to/file - description

* STOP - do not declare complete until explicit user confirmation. *

---

Phase 9 - Midpoint drift check (portfolio-level only)

Trigger: after completing skill #N/2 (e.g., skill #9 in a 17-skill cycle). Check SKILLS_INVENTORY.md §Review ordering for the canonical midpoint.

Procedure per CALIBRATION_KIT.md §4:

1. Re-read §1 anchor catalog.
2. Re-score skill #1 blindly (without looking at original labels).
3. Compare original vs re-score.
4. Classify: no drift / minor drift / major drift.
5. Major drift → run CALIBRATION_KIT.md §5 recalibration before next skill.

Log outcome in DRIFT_LOG.md (create if absent).

---

Phase 10 - Final mechanical sweep (portfolio-level, cycle closure)

Trigger: after Phase 6 GO on the last skill of the cycle, before declaring cycle CLOSED.

Procedure per REVIEW_FRAMEWORK.md §Phase 10 (mechanical-only — no MITL, no LLM jury, no walkthrough):

1. Treat final REVIEW_FRAMEWORK.md + SPEC_SNAPSHOT.md as authoritative — the version that ships.
2. For each of the 17 reviewed skills, re-run Phase 1 only against the final spec (the 8-step preflight table). Skip Phase 2-6.
3. Emit consolidated report at outputs/phase10-drift-sweep-<YYYY-MM-DD>.md with a 17 × 8 PASS|FAIL grid.
4. Any FAIL row → log as follow-up review item for next cycle (e.g. in MEMORY.md or roadmap backlog). Do NOT block closure, do NOT auto-fix, do NOT re-escalate severity.

Rationale: pipeline hardening lands progressively during the cycle. Skills reviewed early are calibrated to an earlier framework version. Phase 9 catches mid-cycle drift on the first 3 skills only; without Phase 10, late-cycle hardening creates silent residual drift on early-cycle skills.

---

Output format (per skill review)

# Skill review - [skill-name] [tier: S|M|L|all]
Date: YYYY-MM-DD
Mode: full | preflight-only | fixtures-only

## Phase 1 - Preflight
[C1-C8 table]

## Phase 2 - Structural
[2.A, 2.B (cross-tier diff), 2.C, 2.D findings, 2.E fixtures if applicable]

## Phase 3 - Fixes
[applied + deferred + rollbacks]

## Phase 4 - External review
[skipped | candidates aggregated]

## Phase 5 - Propagation
[cross-tier changes applied]

## Phase 6 - Outcome
[checklist above]

---

Hard rules

• Never auto-fix without Phase 3 STOP confirmation. Findings flow through Phase 2 first.
• Never skip Phase 9 midpoint in a portfolio review. Drift compounds silently otherwise.
• Always use AskUserQuestion for multi-option walkthrough questions (framework v1.2 P5 §3 + project memory feedback).
• Max 3 fix attempts per Critical finding. After 3 failures: revert + escalate.
• Severity labels are rubric-anchored, never intuitive. Re-read SEVERITY_SCALE.md decision tree when uncertain.
• FP-rate cap rule: technical severity cannot override FP-rate cap. Critical requires < 10% FP evidence.

---

Red flags

The most common ways discipline is silently abandoned here:

Temptation	Why it fails
Applying a fix before the Phase 3 STOP confirmation	Phase 3 STOP is a hard gate; "obviously safe" fixes are not exceptions
Downgrading a Critical to clear the false-positive cap	The cap is a quality floor; downgrade requires documented evidence, not cap pressure
Declaring PASS when findings were deferred rather than resolved	PASS means all Phase checks closed; deferred findings are open findings
Skipping 2.B cross-tier review when only one tier is modified	Cross-tier coherence is mandatory regardless of declared scope
Skipping Phase 9 midpoint drift check "because nothing seems to have drifted"	Drift is silent by nature; the midpoint check exists precisely because it cannot be detected by inspection