| name | recoup-internal-eval-skill-benchmark |
| description | INTERNAL — Recoup staff tooling, gated by the recoup-internal keyword. Invoke ONLY when the request explicitly includes "recoup-internal" (e.g. "recoup-internal benchmark this skill pack"). Never use for customer-facing or artist requests. Benchmark ANY plugin, skill pack, or skills folder against the current frontier of skill/plugin design (gstack, gbrain, compound-engineering, PM OS). Use whenever someone wants to grade, audit, review, or compare a skills directory or plugin and asks things like "is this skill pack good?", "how does our plugin compare to the frontier?", "are these skills fat enough?", "should these skills be combined or split?", "what should we steal from gstack?", "benchmark this plugin", "rate my skills folder", or "audit this skill bundle". Works on any domain (engineering, music, finance, bio — anything) and any harness layout. It measures the pack deterministically (skill count, SKILL.md body sizes, tests/evals/resolver/hooks/learnings signals, catalog token budget, description-overlap risks), grades it on a 15-dimension scorecard, then produces a fat-enough / combine / split / parameterize diagnosis and a prioritized list of frontier moves to steal — each with its boundary. Do NOT use this to author a single new skill (use a skill-creator) or to run a pack's own tests. |
Skill Benchmark
Grade any plugin or skills folder against the frontier bar for what makes skills
and plugins good. The output is an evidence-backed report: where the pack
sits, what's fat vs. thin, what should be combined/split/parameterized, and the
prioritized, boundary-carrying moves worth stealing.
This skill follows the very discipline it grades: a deterministic measurement
script does the counting (so numbers are never eyeballed), and the latent
judgment — scoring, diagnosis, recommendations — is the agent's job, grounded in
the reference standard. The script and references ship alongside this SKILL.md;
invoke them with the relative paths below.
Inputs
- Target (required): a path to a plugin directory, a folder of skills, or a
whole repo. The skill auto-discovers every
SKILL.md underneath.
- Goal of the pack (infer or ask once): is it a vertical "everything-in-a-box"
bundle, a focused capability plugin, or a teaching demonstration? The bar
differs — grade against intent, not a single absolute.
If no target is given, ask for the path before doing anything else.
The procedure (five phases)
Phase 1 — Frame: what is this pack trying to be?
Before measuring, establish intent (one short paragraph). Read the pack's README
/ top-level manifest if present, and skim 2–3 skill descriptions. Decide which
archetype it is — it sets which scorecard dimensions matter most:
- Vertical "in-a-box" bundle (PM OS analog): expect tiering (system /
workflow-orchestrator / reusable-leaf) and, past ~40 skills, tested routing.
- Focused capability plugin: expect a router + a handful of real leaves;
routing tests are overkill below ~40 skills.
- Teaching / demonstration pack: thin and testless is correct — say so and
grade gently.
Phase 2 — Measure (deterministic; never guess the numbers)
Run the bundled measurement script against the target:
python3 scripts/measure.py <target-path>
python3 scripts/measure.py <target-path> --json -o /tmp/bench.json
scripts/measure.py ships alongside this skill (stdlib-only, no network). It
reports: skill count; SKILL.md body line min/median/max + size bands
(tiny/leaf/deep/fat); per-skill footprint (body + co-located refs + scripts);
the always-loaded catalog token estimate vs the ~7K budget; the pack-level
signals that the frontier treats as the defining properties of a skill
pack (deterministic scripts, unit/integration tests, LLM evals, resolver/routing
tests, reachability gate, hooks, learnings store, distribution manifests); and
description-overlap candidates (pairs most likely to mis-route).
Take the numbers as given — your job is to interpret them, not re-derive them.
Phase 3 — Grade against the 15-dimension scorecard
Read references/scorecard.md and fill one row per dimension, grading each
✅ at/ahead of frontier · ⚠️ present-but-weak · ❌ gap, with the measured
evidence and a one-line so-what. To grade the judgment-depth dimensions (1, 5,
9, 12) you must read actual files, not just the counts: open the 2–3 fattest
and 2–3 thinnest SKILL.md bodies and judge whether the lines are encoded
judgment (multi-phase procedure, interaction contract, verification loop) or
boilerplate/padding. The bar and the real frontier numbers are in
references/frontier-benchmark.md — cite it.
A signal the script can't see: when grading the deterministic substrate (dim
3), spot-read a script or two and ask the Foxconn-factory question — is this
exact work (math, parsing, validation) or code written to distrust the model?
Only the former earns its keep (see references/frontier-benchmark.md §7).
Phase 4 — Diagnose: fat enough? combine? split? parameterize?
For each plugin or skill group, classify it Hold / Fatten / Parameterize-or-
combine / Split using the framework in references/scorecard.md. Anchor every
call in evidence:
- Use the size bands to spot thin skills doing deep jobs (→ Fatten).
- Use the overlap candidates as the shortlist for Parameterize-or-combine,
but only recommend combining when the process is the same and only the
input/depth varies — otherwise keep them split (the god-skill anti-pattern).
- Flag the ~40-skill threshold explicitly if the pack is at/over it with no
resolver eval or reachability audit.
Phase 5 — Report: prioritized moves with boundaries
Produce the final report (structure below). Pull recommendations from the
prioritized "what to steal" list in references/scorecard.md (P1–P7), and
carry each boundary — when NOT to do the move. Always include a "keep what
you already lead on" section so a later refactor doesn't delete a strength.
Output format
Deliver a single report with these sections:
- Verdict in one screen — archetype, the headline gap, and the single
highest-leverage move, in ~5 sentences.
- The numbers — a compact table from
measure.py (skills, body
min/median/max, bands, catalog tokens, signals).
- Scorecard — the 15-row table, graded, with evidence.
- Diagnosis — per plugin/group: Hold / Fatten / Combine / Split, with the
reason.
- Prioritized moves — the P1–P7 steal-list filtered to what this pack
actually needs, each with its boundary.
- Keep (don't regress) — the dimensions the pack already leads on.
Lead with what's measured; present judgment calls as judgment. Never inflate
a grade the evidence doesn't support — an unbacked benchmark is worse than none.
Boundaries (when this skill does NOT apply)
- Authoring one skill → use a skill-creator, not this.
- Running a pack's own test suite → that's the pack's CI, not a benchmark.
- Grading a deliberate demonstration pack as if it were production → don't;
thin + testless is the correct shape for teaching the format.
- Cargo-culting frontier features → every recommendation must carry its
boundary. Fat ≠ verbose; tests ≠ a Foxconn factory; an explicit resolver below
~40 skills is overhead, not progress.
What to read
references/frontier-benchmark.md — the standard: the frontier packs and
their real numbers, the core patterns (thin-harness/fat-skills, latent vs.
deterministic, skill-as-method-call, the tested bundle, resolver routing, the
complexity ratchet + Foxconn counter-weight), the anatomy of fat with verified
line counts, and the size-budget discipline. Read before grading.references/scorecard.md — the 15-dimension rubric, the
Hold/Fatten/Combine/Split diagnosis framework, and the prioritized P1–P7 steal
list with boundaries.scripts/measure.py — the deterministic measurement tool (ships with this
skill; run it, don't reimplement it).
