validate-skillmd

# Validate SKILL.md Grade any SKILL.md file against the Intent Solutions rubric (validator v7.0 / schema 3.3.1). Four-tier validation: **Tier 0** (locate), **Tier 1** (standard or marketplace grading), **Tier 2** (static production gate), **Tier 3** (JRig behavioral eval — opt-in). Source of truth: `/skill-creator` validation workflow + `claude-code-plugins-plus-skills/000-docs/SCHEMA_CHANGELOG.md` + `j-rig-skill-binary-eval/` (Tier 3). ## Overview Schema 3.3.1 enforces the 8-field IS enterprise required-field set at marketplace tier (`name`, `description`, `allowed-tools`, `version`, `author`, `license`, `compatibility`, `tags`). Anthropic's spec floor (`name` + `description` only) sits underneath; the IS rubric sits on top. **Modes**: - **Standard** (default): Mirrors `platform.claude.com/docs/en/agents-and-tools/agent-skills/overview` exactly. Required: `name`, `description`. Everything else is silent unless invalid type/value. Fast (~10 sec). - **Marketplace** (`--marketplace`): 8-field enterprise required set + 100-point IS rubric. **Missing required fields = ERROR, not warning.** The `--enterprise` flag is a deprecated alias. Fast (~10 sec). - **Deep** (`--deep`): Intent Solutions Deep Evaluation Engine — 10 weighted dimensions, trust badges, Elo competitive ranking, optional LLM quality assessment via Groq. Fast (~30 sec). - **Thorough** (`--thorough`): Adds **Tier 3 JRig behavioral eval** on top of Tier 1+2. Runs 7-layer eval across Haiku/Sonnet/Opus. **Slow** (~10–30 min) and **costs ~$2–5 per skill in API spend** — opt-in only. Right for production-gating, not iterative authoring. > **Performance + cost note**: Tiers 0–2 run in seconds and are free. Tier 3 (JRig) is opt-in because behavioral eval across the model matrix is genuinely expensive. Default invocations stay fast; `--thorough` is for the moment a skill is being promoted to production or marketplace-verified. ## Prerequisites - Python 3 with `pyyaml` installed - Validator script: `claude-code-plugins-plus-skills/scripts/validate-skills-schema.py` (v7.0+) - For `--thorough` (Tier 3): JRig CLI on PATH (`jrig --version` returns ≥ v0.14.0). Install: `cd ~/000-projects/j-rig-binary-eval && pnpm install && pnpm build && pnpm link --global`. Tier 3 is opt-in; the rest of the skill works without JRig installed. ### Kernel schema first (canonical machine spec) Validate frontmatter structure against `references/kernel-schemas/v1/skill-frontmatter.schema.json` FIRST — the kernel-pinned canonical machine spec (`@intentsolutions/core@0.5.0`, vendored; the STRICT v2 sibling under `kernel-schemas/v2/` is not yet promoted to canonical). The prose spec references are supporting documentation only; on disagreement the kernel schema wins. `$ref`s are absolute `$id` URIs — register every file under `kernel-schemas/` to resolve them. Provenance + refresh: `references/kernel-schemas/PROVENANCE.md`. ## Schema Reminder — Frontmatter Fields Full field table: [schema-reminder-frontmatter.md](references/schema-reminder-frontmatter.md). ## Instructions ### Step 1: Locate the Skill If path provided via `$ARGUMENTS`, use it directly. Otherwise: - Check current directory for SKILL.md - Ask user with AskUserQuestion Common locations: - `~/.claude/skills/{name}/SKILL.md` (global) - `.claude/skills/{name}/SKILL.md` (project) ### Step 2: Run Validator ```bash # Standard tier (default — Anthropic spec exactly) python3 claude-code-plugins-plus-skills/scripts/validate-skills-schema.py SKILL.md # Marketplace tier (full 100-point rubric, polish recommendations as warnings) python3 claude-code-plugins-plus-skills/scripts/validate-skills-schema.py --marketplace SKILL.md # Deep evaluation (10 dimensions, badges, Elo ranking) python3 claude-code-plugins-plus-skills/scripts/validate-skills-schema.py --deep SKILL.md # Deep + LLM quality assessment via Groq (requires GROQ_API_KEY) python3 claude-code-plugins-plus-skills/scripts/validate-skills-schema.py --deep --thorough SKILL.md # Deep eval with JSON/markdown/HTML report output python3 claude-code-plugins-plus-skills/scripts/validate-skills-schema.py --deep --report-format json SKILL.md python3 claude-code-plugins-plus-skills/scripts/validate-skills-schema.py --deep --report-format html SKILL.md # Marketplace + write to compliance DB python3 claude-code-plugins-plus-skills/scripts/validate-skills-schema.py --marketplace --populate-db claude-code-plugins-plus-skills/freshie/inventory.sqlite SKILL.md # Show D/F grade skills in full scan python3 claude-code-plugins-plus-skills/scripts/validate-skills-schema.py --marketplace --show-low-grades # Minimum grade gate (exits 1 if any skill below threshold) python3 claude-code-plugins-plus-skills/scripts/validate-skills-schema.py --marketplace --min-grade B SKILL.md ``` Default to marketplace tier when the user is preparing for marketplace submission. Use `--deep` for functional quality assessment beyond structural compliance. The `--enterprise` flag still works as a deprecated alias for `--marketplace`. **v6.0 Deep Evaluation Engine:** 10 weighted dimensions (triggering accuracy 0.25, orchestration fitness 0.20, output quality 0.15, scope calibration 0.12, progressive disclosure 0.10, token efficiency 0.06, robustness 0.05, structural completeness 0.03, code template quality 0.02, ecosystem coherence 0.02). Anti-pattern detection with 5% penalty each. Elo competitive ranking. Trust badges (Flagship/Established/Emerging/Early). Optional LLM-as-judge via Groq free tier. ### Step 2.5: Tier 2 — Static Production Gate After Tier 1 grading and before any behavioral eval, run five inline static checks. These catch obvious production blockers in seconds without needing JRig. **Always run** regardless of mode (standard/marketplace/deep/thorough); each is binary pass/fail. #### 2.5.1 `allowed-tools` accuracy Every tool declared in `allowed-tools` should actually be referenced somewhere in the skill body or its `references/`/`scripts/`. Conversely, every tool the skill calls should be declared. ```bash # Extract declared tools (handles CSV string, space-separated, and YAML list forms — schema 3.3.1) declared=$(python3 -c "import yaml,sys; fm=yaml.safe_load(open('SKILL.md').read().split('---')[1]); t=fm.get('allowed-tools',''); print(t if isinstance(t,str) else ' '.join(t))") # Each declared base tool (Read, Write, Bash, etc.) must appear in the body for tool in $(echo "$declared" | grep -oE '[A-Z][a-zA-Z]+' | sort -u); do if ! grep -q "$tool" "SKILL.md"; then echo "FAIL: tool '$tool' declared but not referenced in body" fi done ``` **Fail when**: tool declared but never used (over-permissive — attack surface) OR tool used but never declared (will prompt user every invocation, defeating allowed-tools). #### 2.5.2 Auth protocol documented (when applicable) If the skill mentions an external API (any URL, `curl`, `fetch`, MCP server, OAuth flow, API key reference), an authentication method must be documented in the body or in `references/auth.md` / `references/api-surface.md`. ```bash # Heuristic: look for API indicators if grep -qE "(curl |fetch\(|mcp__|API_KEY|TOKEN|OAuth|Bearer )" "SKILL.md"; then if ! grep -qiE "(authentication|auth method|api key|bearer token|oauth flow|credentials)" "SKILL.md"; then echo "FAIL: external API referenced but no auth protocol documented" fi fi ``` **Fail when**: API surface is referenced but a future engineer reading the skill couldn't tell how authentication happens. #### 2.5.3 Dead-code / unreachable-branch sanity Conditional structures that can never fire (e.g., `if false`, mutually exclusive guards, sections that contradict an earlier hard-fail). ```bash # Conservative checks — flag for human review, don't auto-fail grep -nE "^(if false|if \[ false \]|elif false)" "SKILL.md" && echo "WARN: literal-false branch found" grep -cE "^### " "SKILL.md" # If section count grossly exceeds the table-of-contents count → drift ``` **Warn when**: a literal-false branch is found OR the body contains sections not present in the table of contents (silent drift). #### 2.5.4 Tool-safety combo flagging Dangerous combinations require explicit justification: | Combo | Why dangerous | |---|---| | `Bash` (unscoped) + `WebFetch` | Can fetch arbitrary content + execute it | | `Bash` (unscoped) + `Write` | Can write executable scripts to arbitrary locations | | `Bash(curl:*)` + `Bash(sh:*)` | Curl-pipe-shell pattern | | `Bash(rm:*)` not paired with explicit safe-paths | Unbounded delete authority | ```bash # Check for unscoped Bash + dangerous companion if grep -qE "^allowed-tools:.*\bBash\b" "SKILL.md" && \ ! grep -qE "^allowed-tools:.*Bash\(" "SKILL.md" && \ grep -qE "^allowed-tools:.*\b(Write|WebFetch)\b" "SKILL.md"; then if ! grep -qiE "(safety justification|why unscoped Bash|why Bash + )" "SKILL.md"; then echo "FAIL: unscoped Bash + Write/WebFetch without safety justification" fi fi ``` **Fail when**: a dangerous combo is declared and the body has no `## Safety Justification` section explaining why the wide scope is necessary. #### 2.5.5 Orchestration bounds Skills are NOT plugins. A skill should not spawn other skills, delegate to other agents as a primary control flow, or self-coordinate across sessions. That's `/skill-creator --forge` territory and plugin-level orchestration. Skills do one job. ```bash # Look for orchestration smells in skills if grep -qE "(spawn another skill|delegate to /|invoke .* skill|orchestrate across|self-coordinate)" "SKILL.md"; then echo "FAIL: skill appears to orchestrate other skills/agents — that belongs at the plugin layer" fi ``` **Fail when**: the skill body claims it spawns/orchestrates other skills or agents as the primary control flow. Multi-agent synthesis WITHIN one skill invocation (calling subagents to specialize) is fine and expected; cross-skill orchestration is not. #### Tier 2 verdict - **All 5 checks pass** → Tier 2 GREEN; proceed. - **Any FAIL** → Tier 2 RED; the skill is blocked from production promotion until resolved. Tier 3 (JRig) does not run if Tier 2 is RED — fail fast. - **Only WARN** → Tier 2 YELLOW; log warnings to the unified report; Tier 3 still runs. ### Step 2.7: Tier 3 — JRig 7-Layer Behavioral Eval (opt-in via `--thorough`) > **Default skipped.** Tier 3 runs only when the user passes `--thorough`. Behavioral eval across the model matrix (Haiku / Sonnet / Opus) takes 10–30 minutes per skill and costs ~$2–$5 in API spend. Right for production-gate moments, not iterative authoring. #### Prerequisites - JRig CLI on PATH: `j-rig --version` (note: bin name is `j-rig` with hyphen, not `jrig`) - Install: `cd j-rig-skill-binary-eval/packages/cli && pnpm build && ln -sf $PWD/dist/index.js ~/.local/bin/j-rig` - For Tier 3B (7-layer eval): API keys for Haiku, Sonnet, Opus configured per JRig docs If JRig isn't on PATH, the skill emits a placeholder verdict and a one-line install hint, then continues to Step 3 (grade report) without blocking. **Tier 3 absence does not mean a skill fails** — only Tier 1+2 are mandatory. #### Tier 3A: JRig package-integrity check (deterministic, fast, free) Run JRig's `check` command on the **skill directory** (not the SKILL.md path). Returns deterministic pass/warn/error verdicts on package structure: SKILL.md exists + parses, name present, description length, deprecated patterns, time-sensitive content, etc. ```bash # JSON output for parsing into the unified report j-rig check "$(dirname "SKILL.md")" --json ``` This is a separate concern from the IS spec-compliance check (Tier 1) — JRig's `check` is structural, not rubric-based. The Anthropic + AgentSkills.io spec snapshots in `000-docs/` are read by the IS validator (`scripts/validate-skills-schema.py`), not by JRig directly. The two are complementary: IS validator scores against the spec rubric; JRig verifies the package shape and surfaces structural anti-patterns. **Verdict mapping**: - All `severity: "pass"` → Tier 3A GREEN - Any `severity: "warning"` → Tier 3A YELLOW (non-blocking; surfaced in unified report) - Any `severity: "error"` → Tier 3A RED (blocks production promotion) #### Tier 3B: 7-Layer Behavioral Eval (execution-based, slow, $) ```bash # Default invocation — Sonnet only, no DB persistence j-rig eval "$(dirname "SKILL.md")" --json # Full model matrix with DB persistence j-rig eval "$(dirname "SKILL.md")" \ --models haiku,sonnet,opus \ --db claude-code-plugins-plus-skills/freshie/inventory.sqlite \ --json # Skip specific layers (when iterating) j-rig eval "$(dirname "SKILL.md")" --no-trigger --no-functional --json ``` Eval spec source: JRig reads `<skill-dir>/eval-spec.yaml` if present, or use `--spec <path>` to point at one elsewhere. A spec is currently **required** — `j-rig eval` errors if neither is found. (Auto-generating a baseline spec from the SKILL.md frontmatter — `should_trigger`/`should_not_trigger` cases derived from the `description` trigger phrases — is the planned `j-rig scaffold-spec <skill-dir>` command; until it ships, author the spec by hand or copy `skill/eval.yaml` as a template.) Layers: 1. **Trigger quality**: precision/recall on user prompts that should and should not activate the skill 2. **Functional quality**: task completion + output format match against gold cases 3. **Regression protection**: sacred-case suite cannot break (skill is locked-down on these) 4. **Baseline value**: skill output beats naked Claude on the same prompt; if not, flag for obsolescence 5. **Model variance**: independent pass/fail per Haiku / Sonnet / Opus; flags any model where the skill collapses 6. **Rollout safety**: prompt-leakage detection, unsafe-pattern surfacing, jailbreak resistance 7. **Cost / latency**: per-invocation token + wall-clock — must beat declared budget #### Tier 3 verdict - **All 7 layers pass on all 3 models** → Tier 3 GREEN; skill is JRig-Verified. - **Any layer fails on any model** → Tier 3 RED; report which layer + which model + recommended remediation. Block production promotion. - **JRig unavailable** → Tier 3 SKIPPED; emit a one-line note in the unified report; do not block. #### Persist results to Freshie JRig writes its own SQLite DB by default (`j-rig.db` in cwd, or `--db <path>`). To unify with Freshie, point `--db` at `freshie/inventory.sqlite`: ```bash j-rig eval "$(dirname "SKILL.md")" \ --models haiku,sonnet,opus \ --db claude-code-plugins-plus-skills/freshie/inventory.sqlite ``` JRig manages its own tables in that DB; cross-table joins to `skill_compliance` happen in the Freshie rebuild script. The unified-report rendering reads both: ```sql -- Inferred join (actual table names per j-rig schema)