| name | skill-audit-router |
| version | 1.0.0 |
| description | Audit installed agent skills for retrievability, then route a query to one.
skill_audit.py resolves which SKILL.md files are truly on the runtime load path
(excluding marketplace clones, cross-tool mirrors, dev checkouts, duplicates),
scores every description, and emits a rewrite brief; skill_router.py indexes the
survivors with BM25 and returns invoke / ambiguous / none. Pure stdlib.
Use when asked to "audit my skills", "why isn't my skill being picked", "which
skill should handle this", "score my skill descriptions", "find dead or duplicate
skills", or "how many skills are actually loaded". Do NOT use to install new
skills from a marketplace (that is find-skills), to codify a browser flow (that
is skillify), or to capture session learnings (that is self-improving-agent).
|
| triggers | ["audit my skills","which skill should handle this","why isn't my skill being picked","score skill descriptions","route this query to a skill"] |
| allowed-tools | ["Bash","Read","Write","Edit"] |
Skill audit and router
Two stages, in this order, because the second is worthless without the first.
Scanning for SKILL.md and calling the result your catalog produces a number that
has nothing to do with what the model can actually invoke.
Stage 1 — audit (always run this first)
python3 skill_audit.py --json catalog.json --fix-plan rewrites.md
Resolves the real load path and excludes what was never routable: cross-tool
mirrors (.cursor/, .opencode/, .kiro/), marketplace clones, uninstalled
plugins, dev checkouts outside the load path, byte duplicates, and name-shadowed
skills. Plugin liveness comes from installed_plugins.json, the only
authoritative source.
Then scores each description on static signals (length band, trigger phrasing,
negative scoping, concrete anchors, filler penalties) plus three corpus-aware
signals that are the ones that actually predict misrouting:
- Self-retrieval — query the index with a skill's own description. If it does
not rank itself first, no router will ever pick it correctly.
- Distinctiveness — mean idf of its top terms.
- Margin — lead over its nearest neighbour.
Stage 2 — route
python3 skill_router.py --catalog catalog.json "our cancel flow is leaking subscribers"
python3 skill_router.py --catalog catalog.json --eval evals.jsonl
python3 skill_router.py --catalog catalog.json --for-query "..."
BM25 over a weighted bag (name ×3, triggers ×2, keywords ×2, description ×1), with
hapax damping (a term in exactly one document is damped to 35%, so an incidental
rare word cannot rocket a skill to the top) and negative triggers (terms a
description disclaims are subtracted, not merely un-added).
The decision policy is the product. Four gates must pass before invoke:
query mass, raw floor, coverage ≥0.30, and ≥2 matched terms. The floors are
multiples of mean idf, not constants, so they scale with catalog size.
The fix loop
skill_audit.py --json catalog.json --fix-plan rewrites.md- Rewrite flagged descriptions in this shape:
what it produces. Use when
<concrete user phrasings>. Do NOT use for
<the nearest neighbour's territory>.
- Re-run the audit, confirm the score moved.
skill_router.py --eval evals.jsonl before and after.
Read the eval numbers correctly
top-1 accuracy : ranked first
auto-invoked : ranked first AND confident (rank ≠ fire)
top-3 recall : reachable via shortlist
NONE (clean) : returned NONE outright
NONE (safe) : never auto-invoked <- DEFEND THIS AT 100%
Auto-invoking on small talk hijacks the turn; returning ambiguous costs a few
hundred tokens. Those failures differ in cost by an order of magnitude. Chasing
clean-NONE to 100% pushes thresholds conservative enough to suppress real matches.
Hard-won cautions
- Descriptions may not be the problem. Check for an
AUTO-GENERATED banner and
a sibling SKILL.md.tmpl or a second copy of the tree first — a stale copy looks
exactly like bad writing. Recovering one recovered 16k chars in a real case.
- Enriching descriptions changes every other skill's score. BM25 floors are
multiples of mean idf. Re-run the eval after bulk edits; a real case saw
safe-NONE drop 100%→86% from improvements alone. Fix over-triggering at the
source with negative scoping, not by loosening constants.
- Never put a backup inside the skills tree — the audit (and the runtime) will
scan it as real skills.
- Use it as a shortlister, not a prompt card.
--card renders the whole catalog;
--for-query retrieves first and describes only survivors, at ~3% of the tokens.
Let the model make the final call — it is better at intent than any lexical scorer.
Full design notes, scoring detail and the list of traps already handled:
REFERENCE.md in this directory.
