skill-coverage-gate

스타69

포크9

업데이트2026년 7월 19일 02:22

Use when reviewing, shipping, or regression-testing a SKILL.md and you hold one or more captured trajectories where that skill was actually loaded. Translates the skill's instructions into behavior constraints, then reports which were EXERCISED (coverage) versus untested, and flags any that were in-scope but VIOLATED (compliance) — a cheap single-trajectory test-adequacy gate. It surfaces untested instructions rather than auto-blocking, and escalates fail-closed only when a violation touches shared repo state, never-surface memory, or self-modification. Complements skill-counterfactual-audit (which measures EFFECT) and skill-causal-curation (which measures contribution); this one asks ADEQUACY. Backed by the agent-skill-coverage metric (arxiv 2606.20659v2). Triggers on skill review, ship gating, and regression-testing a skill against its trajectories.

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

출처

Tibsfox

Tibsfox/gsd-skill-creator

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Skill Coverage Gate

Given a SKILL.md and one or more captured trajectories in which it was loaded, translate its instructions into discrete behavior constraints, then report — per constraint — whether the trajectory exercised it (coverage), left it untested, or violated it while in scope (compliance). This is a cheap, single-trajectory adequacy check that slots into the skill-forge ship/review loop before more expensive paired or masked evaluation is worth running.

Why

skill-forge validate + critique check a skill's frontmatter and prose; they never observe whether the skill's rules actually got tested by the run that loaded it. A skill can pass authoring review, ship, and then have half its instructions never fire in any real trajectory — untested behavior masquerading as verified behavior. Worse, an instruction can fire and be disobeyed without anyone noticing, because the task still passed. This gate makes both gaps legible: what was tested, and where the agent already broke a rule it loaded.

Data classes touched

Inputs are project-internal: the skill file plus trajectories captured by session-retro / session-observatory-live under .planning/patterns/. Those traces can quote real work that includes strongly-marked never-surface records — private origins, Fox Companies IP, credentials, Center Camp trust rules. Boundary rule: the coverage report cites constraint evidence by trace location and event index, never by pasting trace content that may carry such a record. The report is project-internal and is written under .planning/; it is never merged into published docs or any shared output.

How

Verify the input is valid. Confirm the skill was in the loaded skill set for each supplied trajectory (check the session-retro loaded-skills field). A trajectory where the skill was not loaded gives coverage undefined — discard it, do not count it as zero coverage. Require ≥1 valid trajectory.
Extract constraints. Read the skill body and decompose it into atomic behavior constraints, each as {id, precondition, required-behavior, scope-tags}. scope-tags mark whether the constraint governs shared repo state (git worktrees, refinery-merge queue, commits), sensitive memory (Grove / MEMORY.md never-surface records), or self-modification (.claude/skills, .claude/agents, .claude/hooks). This decomposition is an LLM judgment — see the failure model.
Label each constraint against the trajectory set.
- untested — no trajectory ever hit the precondition. An adequacy gap.
- exercised — precondition fired and the required behavior was observed followed. Counts toward coverage.
- violated — precondition fired and observed behavior contradicts the required behavior. A compliance failure.
Compute two numbers. Coverage = exercised / (exercised + violated + in-scope-but-ambiguous), over constraints whose precondition fired at least once. Compliance = 1 − violated / (exercised + violated).
Decide the report verdict.
- Coverage < 0.60 of extracted constraints → flag under-exercised: recommend more probe trajectories or hand off to skill-counterfactual-audit. Non-blocking — this surfaces the gap.
- Any violated constraint whose scope-tags include shared repo state, never-surface memory, or self-modification → fail-closed: block the skill-forge ship, do NOT silently proceed, and escalate to the operator via mayor-coordinator.
- A violated constraint on benign scope → report as a compliance gap, non-blocking, and recommend a skill edit or a narrower trigger.
Emit report to .planning/patterns/skill-coverage/<skill>-<timestamp>.md: the constraint table (id, label, evidence-location), the two numbers, and the verdict.

Robustness rule

Judge a constraint exercised or violated by the effect of what the agent did, not by whether the trace text echoes the skill's wording. An agent that paraphrases a rule and follows it is compliant; one that quotes the rule and then acts against it is a violation. Do not mark coverage from surface vocabulary match.

Confidence / failure model

Constraint extraction and the exercised/violated labeling are LLM judgments (semi-decidable, not a deterministic pass/fail check). Coverage from a single trajectory is a lower bound on adequacy, not proof of correctness: high coverage means the tested rules fired, not that the skill is good — that is skill-counterfactual-audit's and skill-causal-curation's question. The gate reduces the chance an untested or already-violated rule ships; it does not eliminate it. On any uncertainty about whether a scope-tagged constraint was violated, default fail-closed (escalate), never silently pass.

When to skip

No valid trajectory exists where the skill was loaded — capture a run first; there is nothing to measure adequacy against.
The skill is a pure metadata/frontmatter change — route to skill-frontmatter-doctor, which needs no trajectory.
A structural gate (skill-forge validate/critique) already failed the skill — fix that first; coverage over a broken skill is noise.

Distinctness

skill-counterfactual-audit runs a paired with/without probe and measures the skill's behavioural EFFECT (the SIP report). skill-causal-curation uses randomized masking to measure per-task-type CONTRIBUTION for keep/repair/ retire. This skill asks neither — it asks single-trajectory ADEQUACY: were the skill's own rules exercised, and were any broken. It reports gaps; it does not auto-retire or auto-block on effect.

Integration

skill-forge — this gate runs inside its ship/review loop, after validate + critique, before publish; a fail-closed verdict halts the ship.
session-retro / session-observatory-live — the trajectory source under .planning/patterns/.
skill-counterfactual-audit + skill-causal-curation — sibling eval skills answering the EFFECT and CONTRIBUTION questions; escalate an under-exercised skill to them when a single trajectory is not enough evidence.

이 저장소의 다른 Skills

같은 저장소

goal-ambiguity-gate

Tibsfox/gsd-skill-creator

Use the moment a request is underspecified and you are about to act on an ASSUMED goal. Enumerate the candidate goals the request could mean, project the action or artifact each one implies, and check whether they diverge: if two or more plausible goals lead to materially different actions or artifacts, ask exactly ONE targeted clarifying question; otherwise proceed on the most-likely goal and state the assumption in one line. Clarification is an evidence-producing action, not a delay — but one question, never an interrogation. Distinct from gsd-spec-phase (GSD-phase-bound, emits a heavy SPEC.md) and intent-router (fetch strategy, not goal disambiguation). Backed by agent goal-state inference (arxiv 2606.16813v1). Triggers on acting under an assumed goal when the request admits more than one materially different reading.

2026-07-1969

memory-hubness-gate

Tibsfox/gsd-skill-creator

Use at write time to vector/embedding memory — Grove content-addressed insertion, chroma/pgvector upserts, memory-consolidation promoting session traces to MEMORY.md, or embedding externally-ingested content. Scores each candidate record against a fixed panel of sentinel queries and quarantines any record that would become the nearest neighbor of too many unrelated queries — a hub — whether from adversarial poisoning or accidental over-generality. This is the memory-record-side sibling of skill-injection-guardian (file-side) and the write-side complement of memory-use-warrant (read-side). Quarantine, never silently drop; a human reviews. Backed by the admission-time hubness gate (arxiv 2606.19692v1). Triggers on inserting into vector memory, consolidating memory, and embedding stranger content.

2026-07-1969

memory-use-warrant

Tibsfox/gsd-skill-creator

Run this appropriateness check the moment you are about to integrate a retrieved long-term memory — a Grove content-addressed hit, a chroma/pgvector neighbour, a memory-consolidation digest, or a MEMORY.md line — into a response, especially anything touching private origins, Fox Companies IP, credentials, or Center Camp / consent-governed content. It answers a question intent-router never asks: not WHAT to fetch or HOW, but WHETHER a correctly-retrieved item should reach output. Relevance is not appropriateness — a perfect similarity match can still be a boundary violation. Default is FAIL-CLOSED: if in-context authorization is uncertain, the memory may inform behaviour but must not be surfaced. Backed by RBI-Eval (arxiv 2606.06055v1). Triggers on surfacing recalled sensitive memory into a response.

2026-07-1969

operational-anchor-preservation

Tibsfox/gsd-skill-creator

Use before you shorten, compress, or rewrite the BODY of an existing SKILL.md or agent .md — the moment you are about to cut text to reduce length. First inventory the file's operational anchors (exact code / command / API snippets, fail-closed workflow guards, and rule / threshold statements), preserve every one, then judge the rewrite by expected downstream task cost (exploration, debugging, recovery tokens) rather than by the resulting line count. A shorter skill that strips an anchor makes the agent MORE expensive per task, so length is never the objective and rewrite is not compression. Backed by Preserving Operational Anchors When Compressing Agent Instructions (arxiv 2606.09421v2). Triggers on shortening or refactoring a skill or agent body.

2026-07-1969

semantic-merge-adjudicator

Tibsfox/gsd-skill-creator

Use when parallel agents in a convoy or across git worktrees hit a write conflict on shared state — a git tree, a config, or a shared document — and you must decide whether the conflict is semantically real. Classifies each conflicting hunk as independent (keep both), redundant (pick one), or a genuine clash (escalate), instead of blind locks, abort-and-retry, or escalate-everything. Sits ABOVE refinery-merge and NEVER blind auto-merges: it wraps an LLM judgment, defaults to escalate on any uncertainty, and unconditionally escalates any hunk touching .planning/, config, credentials, or safety-critical paths. Backed by CoAgent (arxiv 2606.15376v1). Triggers on write conflicts between parallel agents on shared repo state.

2026-07-1969

trace-to-skill-inducer

Tibsfox/gsd-skill-creator

Use when you have captured session evidence — session-retro / session- observatory-live traces in .planning/patterns/, tool logs, correction records — and want to induce a reusable skill from it. Segments the traces into candidate skill units (an LLM judgment, not a deterministic parse) and decomposes each candidate into a four-part structured spec: workflow structure, execution semantics, and runtime attachments (verification, safety, rollback, state). It emits a spec object, NOT a finished SKILL.md, and hands that spec to skill-forge. It sits between skill-integration (upstream frequency detector) and skill-forge (downstream author). Backed by Agent-Trace-to-Skill Induction (arxiv 2606.06893v1). Triggers on inducing a skill from captured traces, turning a repeated pattern into a skill spec, and preparing evidence for skill-forge.

2026-07-1969

name	skill-coverage-gate
description	Use when reviewing, shipping, or regression-testing a SKILL.md and you hold one or more captured trajectories where that skill was actually loaded. Translates the skill's instructions into behavior constraints, then reports which were EXERCISED (coverage) versus untested, and flags any that were in-scope but VIOLATED (compliance) — a cheap single-trajectory test-adequacy gate. It surfaces untested instructions rather than auto-blocking, and escalates fail-closed only when a violation touches shared repo state, never-surface memory, or self-modification. Complements skill-counterfactual-audit (which measures EFFECT) and skill-causal-curation (which measures contribution); this one asks ADEQUACY. Backed by the agent-skill-coverage metric (arxiv 2606.20659v2). Triggers on skill review, ship gating, and regression-testing a skill against its trajectories.
description-frequency	on-demand
user-invocable	true
version	1.0.0
format	"2025-10-02T00:00:00.000Z"
triggers	["check this skill's instruction coverage against a trajectory","which of this skill's rules were exercised or violated","test-adequacy gate before shipping a skill"]
updated	"2026-07-18T00:00:00.000Z"
status	ACTIVE
source	arxiv 2606.20659v2 (Agent Skill Coverage Metric)

Skill Coverage Gate

Why

Data classes touched

How

Verify the input is valid. Confirm the skill was in the loaded skill set for each supplied trajectory (check the session-retro loaded-skills field). A trajectory where the skill was not loaded gives coverage undefined — discard it, do not count it as zero coverage. Require ≥1 valid trajectory.
Extract constraints. Read the skill body and decompose it into atomic behavior constraints, each as {id, precondition, required-behavior, scope-tags}. scope-tags mark whether the constraint governs shared repo state (git worktrees, refinery-merge queue, commits), sensitive memory (Grove / MEMORY.md never-surface records), or self-modification (.claude/skills, .claude/agents, .claude/hooks). This decomposition is an LLM judgment — see the failure model.
Label each constraint against the trajectory set.
- untested — no trajectory ever hit the precondition. An adequacy gap.
- exercised — precondition fired and the required behavior was observed followed. Counts toward coverage.
- violated — precondition fired and observed behavior contradicts the required behavior. A compliance failure.
Compute two numbers. Coverage = exercised / (exercised + violated + in-scope-but-ambiguous), over constraints whose precondition fired at least once. Compliance = 1 − violated / (exercised + violated).
Decide the report verdict.
- Coverage < 0.60 of extracted constraints → flag under-exercised: recommend more probe trajectories or hand off to skill-counterfactual-audit. Non-blocking — this surfaces the gap.
- Any violated constraint whose scope-tags include shared repo state, never-surface memory, or self-modification → fail-closed: block the skill-forge ship, do NOT silently proceed, and escalate to the operator via mayor-coordinator.
- A violated constraint on benign scope → report as a compliance gap, non-blocking, and recommend a skill edit or a narrower trigger.
Emit report to .planning/patterns/skill-coverage/<skill>-<timestamp>.md: the constraint table (id, label, evidence-location), the two numbers, and the verdict.

Robustness rule

Confidence / failure model

When to skip

No valid trajectory exists where the skill was loaded — capture a run first; there is nothing to measure adequacy against.
The skill is a pure metadata/frontmatter change — route to skill-frontmatter-doctor, which needs no trajectory.
A structural gate (skill-forge validate/critique) already failed the skill — fix that first; coverage over a broken skill is noise.

Distinctness

skill-counterfactual-audit runs a paired with/without probe and measures the skill's behavioural EFFECT (the SIP report). skill-causal-curation uses randomized masking to measure per-task-type CONTRIBUTION for keep/repair/ retire. This skill asks neither — it asks single-trajectory ADEQUACY: were the skill's own rules exercised, and were any broken. It reports gaps; it does not auto-retire or auto-block on effect.

Integration

skill-forge — this gate runs inside its ship/review loop, after validate + critique, before publish; a fail-closed verdict halts the ship.
session-retro / session-observatory-live — the trajectory source under .planning/patterns/.
skill-counterfactual-audit + skill-causal-curation — sibling eval skills answering the EFFECT and CONTRIBUTION questions; escalate an under-exercised skill to them when a single trajectory is not enough evidence.