skill-creator-advance

name	skill-creator-advance
description	Create new skills, redesign (phases, agents, workflow/contract), run evals, or optimize a description's triggering. Use for 'build a skill', 'make a slash command', or a redesign. For output A/B use skill-tuning; token refactor use skill-refactor.

Skill Creator Advance

A skill for creating new skills and iteratively improving them.

The core loop: draft → test → review → improve → repeat.

• Decide what the skill should do → write a draft
• Create test prompts → run claude-with-the-skill on them
• Evaluate results (qualitative review + quantitative assertions)
• Rewrite based on feedback → repeat until satisfied
• Optionally optimize the description for better triggering

Your job is to figure out where the user is in this process and help them progress — from scratch or from an existing draft. Be flexible: if they say "just vibe with me", skip the formal eval machinery.

Communicating with the user

Users range from non-technical to expert. Gauge familiarity from context cues: "evaluation" and "benchmark" are generally OK; for "JSON" and "assertion", look for serious cues the user knows the term before using it without explanation. When in doubt, briefly define terms inline — better to over-explain once than to lose someone.

---

Creating a skill

Pre-Creation Gates (recommended; skip only with stated reason)

Before intake / interview / drafting, run two lightweight gates against the user's request — one or two focused questions each; they prevent shipping a skill that should not have been built.

Gate 1 — Worth-it check — applicable when the user proposes ≥2 skills at once, or one skill with multiple supporting claims ("we need this because A, B, and C"). Triage each proposed item into KEEP / DEFER / DROP, judging it on evidence grounding (is the need real and recurring, not speculative?) and YAGNI (would you build it now?). A single skill with a single load-bearing reason keeps this gate cheap — confirm the reason holds, then move to Gate 2.

Gate 2 — Smallest-end-state check — applicable to every new skill proposal, single or multi. The three questions:

1. What's the smallest end state that solves this? (Could it be 0 functions — not really a skill, just a one-off prompt? One existing skill plus a new section instead of a new skill?)
2. Does this result in less total skill-ecosystem code than not building it? (A new skill adds surface area; default "no" unless it subtracts other artifacts or replaces ad-hoc prompts.)
3. What does this skill make obsolete? (If nothing, the rationale is purely additive — apply deletion-first skepticism as for any feature add.)

Skip explicitly if the user has already done the equivalent analysis ("we discussed this last week and concluded we need a dedicated skill"). Do NOT skip just to move faster — the cost of building the wrong skill is permanent.

If a gate verdict is DROP / REJECT / RESHAPE, surface it and ask the user how to proceed (drop, defer, or reshape smaller) before continuing to intake.

Gate 3 — User-input check (after intake, before drafting) — Does this skill have any user-input branching? If yes, plan to apply the hardened AskUserQuestion pattern from references/asking-user-questions.md when drafting the relevant STEP — without it, the skill is highly likely to inline-fallback or silently default in production. If no user input is needed, this gate is N/A.

Capture Intent

Start by understanding the user's intent. The current conversation might already contain the workflow to capture (e.g., "turn this into a skill") — if so, extract answers from the conversation history first (tools used, step sequence, corrections the user made, input/output formats observed); the user fills gaps and confirms before proceeding.

Clarify these four questions:

1. What should this skill enable Claude to do?
2. When should this skill trigger? (what user phrases/contexts)
3. What's the expected output format?
4. Should we set up test cases? Objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases; subjective outputs (writing style, art) often don't. Suggest the appropriate default, but let the user decide.

Interview and Research

Proactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out.

Check available MCPs — if useful for research (docs, similar skills, best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user.

Write the SKILL.md

Based on the user interview, fill in these components:

• name: Skill identifier
• description: The primary triggering mechanism — include what the skill does AND when to use it (see Description Best Practices below and the §House description standard)
• compatibility: Required tools, dependencies (optional, rarely needed)
• the rest of the skill :)

Description Best Practices

Before drafting the description, read references/description-design.md — it owns the design patterns: WHAT+WHEN (not WHAT+WORKFLOW), "about-to-violate" symptoms, third-person voice, natural keywords the user would type, length guidance, a validation checklist, and a worked git-memory before/after case study. Where that reference and the §House description standard (below, under Description Optimization) differ, the house standard wins.

Skill Writing Guide

Anatomy of a Skill

skill-name/
├── SKILL.md (required)
│   ├── YAML frontmatter (name, description required)
│   └── Markdown instructions
└── Bundled Resources (optional, single-level subdirectories ONLY)
    ├── scripts/    - Executable code for deterministic/repetitive tasks
    ├── references/ - Docs loaded into context as needed
    ├── assets/     - Templates / fixtures / static files
    ├── agents/     - Sub-agent definitions
    └── ... (any other single-level subdirectory you need)

CRITICAL: Folder structure must be flat — NO nested subdirectories

Anthropic skill convention forbids subdirectories inside subdirectories under a skill root. A skill may contain SKILL.md plus any number of single-level subdirectories; those subdirectories must themselves be flat.

✅ CORRECT:
skill-name/SKILL.md
skill-name/scripts/extract_lineage.py
skill-name/references/spec.md
skill-name/assets/template.md

❌ FORBIDDEN (will fail validation hooks in repos that enforce):
skill-name/assets/scripts/extract_lineage.py     ← assets/ contains scripts/

When creating a new skill or adding bundled resources to an existing one:

• Group files by subdirectory at skill root (scripts/, assets/, references/, agents/, etc.) — choose meaningful names
• Inside each subdirectory, use descriptive filenames (e.g., extract_column_lineage.py) rather than further nesting
• If you feel the need to subdivide, extract into a new top-level subdirectory at the skill root instead (e.g., scripts-redshift/ rather than scripts/redshift/)

Each skill should also have a corresponding slash command entry point in the plugin's commands/ directory — format and examples in references/plugin-conventions.md.

Progressive Disclosure

Skills use three-level loading — metadata (always in context), SKILL.md body (loaded on trigger; keep under ~6,000 tokens / ~4,500 words, extracting detail into reference files when approaching the limit), bundled resources (as needed). Details: references/plugin-conventions.md (§Progressive Disclosure, §SKILL.md Token Budget).

Key patterns:

• Reference files clearly from SKILL.md with guidance on when to read them
• For large reference files (>~8,000 tokens), include a table of contents

Domain organization: When a skill supports multiple domains/frameworks, organize by variant (e.g., references/aws.md, references/gcp.md) — Claude reads only the relevant reference file.

Working with Existing Plugin Ecosystems

When creating a skill that will live inside an existing plugin, match the plugin's conventions rather than imposing a new structure — read references/plugin-conventions.md (observing the target plugin's style, the lightweight/standard/full structure spectrum, key conventions).

Principle of Lack of Surprise

This goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a "roleplay as an XYZ" are OK though.

Empty-Prompt Onboarding

When drafting a skill that can be invoked with no prompt or a very sparse one, read references/asking-user-questions.md §Empty-Prompt Onboarding for the opt-in "surface orientation" pattern (recommended for conversational / multi-workflow skills; single-shot utility skills are exempt).

Asking the User Structured Questions (when to use AskUserQuestion)

When a skill needs user input mid-execution that's a discrete choice between 2-4 options (e.g., "which folders to exclude?"), use the AskUserQuestion tool — but apply the hardened pattern documented in references/asking-user-questions.md. The naive approach (Use AskUserQuestion to confirm... with a fenced Q&A template) fails three documented modes (inline fallback, silent default, tool unavailable); the reference closes all three and includes a copy-paste mandatory-gate template.

Always-included for skills with user-input steps. Skills with no user-input steps are exempt — don't add AskUserQuestion just because it exists.

Writing Patterns

Prefer the imperative form in instructions.

Defining output formats — for example:

## Report structure
ALWAYS use this exact template:
# [Title]
## Executive summary
## Key findings
## Recommendations

Examples pattern - Examples are useful. Format them like this (deviate a little if "Input"/"Output" already appear in them):

## Commit message format
**Example 1:**
Input: Added user authentication with JWT tokens
Output: feat(auth): implement JWT-based authentication

Writing Style

Explain to the model why things matter instead of heavy-handed MUSTs. Keep the skill general, not narrowed to specific examples. Draft first, then revisit with fresh eyes and improve.

For writing lean from the start — token economy, bloat self-review, thin-orchestrator design — read references/writing-lean.md.

Test Cases

After writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say — and share them with the user: "Here are a few test cases I'd like to try. Do these look right, or do you want to add more?" (not necessarily verbatim). Then run them.

Save test cases to evals/evals.json. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress.

{
  "skill_name": "example-skill",
  "evals": [
    {
      "id": 1,
      "prompt": "User's task prompt",
      "expected_output": "Description of expected result",
      "files": []
    }
  ]
}

See references/schemas.md for the full schema (including the assertions field, which you'll add later).

---

Improving an Existing Skill

When a user asks to "improve" or "optimize" an existing skill, first determine which kind of improvement before doing any work. Three distinct shapes — they need different tools:

Router: Identify the Improvement Type

Ask the user to clarify (or infer from their phrasing):

Improvement type	Signal	Handler
(a) Token / structure refactor with output behavior unchanged	"shorten", "reduce tokens", "tidy up", "縮減 SKILL.md", "整理結構" — and no behavior change desired	Hand off to skill-dev-toolkit:skill-refactor. Do not handle here.
(b) Output quality / variant exploration with human judgment	"test different phrasings", "improve outputs", "A/B variants", "輸出風格", "我來選哪個比較好" — taste-sensitive output dimensions	Hand off to skill-dev-toolkit:skill-tuning. Do not handle here.
(c) Structural change — add / split / merge phases, change agent decomposition, change input/output contract	"rewrite", "redesign", "add a phase", "split this skill", "重新設計", "拆 skill"	Continue with the full creation flow below, using the existing skill as the starting baseline rather than starting from scratch.

If the user's intent is unclear, ask them to clarify which of (a), (b), or (c) applies. Do not default into the creation flow without confirming — picking the wrong tool wastes time on the wrong type of work.

Case (c): Structural Rewrite Flow

For case (c), use this flow (steps 1–4). Cases (a) and (b) hand off to the dedicated sibling skills above and do not use these steps.

1. Assess the Current State

Read the existing SKILL.md and all bundled files. Understand:

• What the skill does and how it's structured
• What conventions it follows (check the parent plugin's style)
• Known issues the user reports or you observe

2. Diagnose Improvement Areas

Look at these dimensions (these apply to structural rewrites; they are not the right lens for token refactor or output A/B):

• Triggering: Is the description specific enough? Does it undertrigger or overtrigger?
• Instructions: Are they clear? Do they explain the "why"? Are there gaps?
• Structure: Is the directory organization appropriate for the skill's complexity?
• Coverage: Are edge cases handled? Are there missing workflows?
• Bundled files: Are reference files up to date? Are scripts working?

3. Propose Changes

Present a concise improvement plan to the user before making changes. Group changes by impact:

• High impact: Changes that affect the skill's core behavior or triggering
• Low impact: Cleanup, reorganization, wording improvements

4. Evaluate

Use the eval workflow (quick or full path, depending on complexity) to verify improvements. When improving an existing skill, the baseline should be the original version — snapshot it before editing (see Step 1's baseline-run instructions for the command).

---

Running and evaluating test cases

Choosing Your Eval Path

Not every skill needs the full benchmark treatment. Choose based on complexity:

	Quick eval path	Full eval path
For	Simple skills (formatters, templates, single-step workflows) whose quality is best judged by looking at the output	Complex skills (multi-step workflows, objective criteria, many moving parts) needing quantitative comparison
Method	Run 2-3 test cases manually (no subagent spawning); review outputs inline with the user; skip grading, benchmarking, and baseline comparison; iterate on direct user feedback	Spawn parallel subagent runs with baselines; grade with assertions, aggregate benchmarks; structured inline review with markdown reports

When in doubt, start with the quick path — you can always escalate if it isn't giving enough signal.

---

The full eval path follows this sequence — don't stop partway through. Do NOT use /skill-test or any other testing skill.

Put results in <skill-name>-workspace/, a sibling of the skill directory, organized by iteration (iteration-1/, iteration-2/, etc.) and, within that, a directory per test case (eval-0/, eval-1/, etc.). Create directories as you go, not upfront.

Step 1: Spawn all runs (with-skill AND baseline) in the same turn

For each test case, spawn two subagents in the same turn — one with the skill, one without. Don't spawn the with-skill runs first and come back for baselines later — launch everything at once so it all finishes around the same time.

With-skill run:

Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <eval files if any, or "none">
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
- Outputs to save: <what the user cares about — e.g., "the .docx file", "the final CSV">

Baseline run (same prompt, but the baseline depends on context):

• Creating a new skill: no skill at all. Same prompt, no skill path, save to without_skill/outputs/.
• Improving an existing skill: the old version. Before editing, snapshot the skill (cp -r <skill-path> <workspace>/skill-snapshot/), then point the baseline subagent at the snapshot. Save to old_skill/outputs/.

Write an eval_metadata.json per test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just "eval-0" — and use it for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — they don't carry over from previous iterations.

{
  "eval_id": 0,
  "eval_name": "descriptive-name-here",
  "prompt": "The user's task prompt",
  "assertions": []
}

Step 2: While runs are in progress, draft assertions

Use the wait productively: draft quantitative assertions for each test case and explain them to the user. If assertions already exist in evals/evals.json, review them and explain what they check.

Good assertions are objectively verifiable, with descriptive names that read clearly in the benchmark report — someone glancing at the results should immediately understand what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.

Update eval_metadata.json and evals/evals.json with assertions. Explain to the user what they'll see — both the qualitative outputs and the quantitative benchmark.

Step 3: As runs complete, capture timing data

When each subagent task completes, you receive a notification containing total_tokens and duration_ms. Save this data immediately to timing.json in the run directory:

{
  "total_tokens": 84852,
  "duration_ms": 23332,
  "total_duration_seconds": 23.3
}

This is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them.

Step 3.5: Self-assessment pass

Before grading and presenting results to the human, perform a quick automated check on each output. Read references/iteration-automation.md for the full protocol. In brief:

• Read each test case's output and check for obvious defects (empty output, format violations, crash artifacts)
• If a defect is clearly caused by a skill instruction issue, fix the skill and rerun that test case once
• Log results to self_assessment.json in each test case directory
• One pass only — no infinite repair loops

Step 4: Grade, aggregate, and present results

Once all runs are done:

1. Grade each run — spawn a grader subagent (or grade inline) that reads agents/grader.md and evaluates each assertion against the outputs. Save results to grading.json in each run directory. The grading.json expectations array must use the fields text, passed, and evidence (not name/met/details or other variants). Check programmatically-checkable assertions with a script rather than eyeballing — faster, more reliable, reusable across iterations.

2. Check for regressions (iteration 2+) — compare results against the previous iteration (references/iteration-automation.md has the full protocol); lead with any regressions when reporting to the user.

3. Aggregate into benchmark — run the aggregation script from the skill-creator-advance directory:

   python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name>
   

   This produces benchmark.json and benchmark.md with pass_rate, time, and tokens for each configuration, with mean +/- stddev and the delta. If generating benchmark.json manually, see references/schemas.md for the exact schema. Put each with_skill version before its baseline counterpart.

4. Do an analyst pass — read the benchmark data and surface patterns the aggregate stats might hide. See agents/analyzer.md (the "Analyzing Benchmark Results" section) for what to look for — non-discriminating assertions, high-variance (possibly flaky) evals, time/token tradeoffs.

5. Present results inline — For each test case, show the results directly in the conversation:

   ### Test Case: {eval_name}
   **Prompt:** {the task prompt}
   **Output:** {summary of key output files or inline content}
   **Grades:** {assertion pass/fail results with evidence, if graded}
   **Previous:** {what changed from last iteration, if iteration 2+}
   

   After presenting all test cases, show the benchmark summary (pass rates, timing, token usage).

6. Save a markdown report to <workspace>/iteration-N/review.md containing all test case results and benchmark data, persisting them for cross-iteration comparison.

7. Ask for feedback — ask the user for feedback on each test case; focus on specific complaints — no comment means it looked fine.

---

Improving the skill

The heart of the loop: turn the user's feedback on the test results into a better skill.

How to think about improvements

1. Generalize from the feedback. You're creating a skill to be used a million times across many prompts; you iterate on a few examples only for speed — a skill that works only for those examples is useless. Rather than fiddly overfitty changes or oppressively constrictive MUSTs, when an issue is stubborn try branching out — different metaphors, different patterns of working. It's cheap to try and you might land on something great.

2. Keep the prompt lean. Remove things that aren't pulling their weight. Read the transcripts, not just the final outputs — outputs tell you what happened, transcripts how. If the skill makes the model waste time on unproductive work, remove the parts causing it.

3. Explain the why. Today's LLMs have good theory of mind and, given a good harness, go beyond rote instructions. Even if the user's feedback is terse or frustrated, understand why they wrote what they wrote and transmit that understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, that's a yellow flag — reframe and explain the reasoning so that the model understands why it matters.

4. Look for repeated work across test cases. If the transcripts show subagents independently writing similar helper scripts or taking the same multi-step approach (all 3 test cases wrote a create_docx.py), that's a strong signal the skill should bundle that script: write it once, put it in scripts/, and tell the skill to use it.

The iteration loop

After improving the skill:

1. Apply your improvements to the skill
2. Rerun all test cases into a new iteration-<N+1>/ directory, including baseline runs. If you're creating a new skill, the baseline is always without_skill (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration.
3. Present results inline and save to review.md, noting changes from previous iteration
4. Wait for the user to review and tell you they're done
5. Read the new feedback, improve again, repeat

Keep going until:

• The user says they're happy
• The feedback is all empty (everything looks good)
• You're not making meaningful progress

---

Advanced: Blind comparison

For a more rigorous comparison between two versions of a skill (the user asks "is the new version actually better?"), read agents/comparator.md and agents/analyzer.md: give two outputs to an independent agent without telling it which is which, let it judge quality, then analyze why the winner won. Optional, requires subagents; the human review loop is usually sufficient.

Boundary note vs skill-dev-toolkit:skill-tuning: the blind comparator uses an LLM subagent as judge — fast and cheap, but inherits LLM-as-judge limitations (verbosity bias, position bias, weak signal on taste-sensitive output dimensions like voice / tone / creative quality). For taste-sensitive A/B that needs reliable preference signal, use skill-tuning instead — it uses human judgment per iteration and accumulates a preference log. Rule of thumb: blind comparator for objective / structured outputs (file transforms, code generation, fixed-format generators); skill-tuning for subjective / creative outputs (writing style, design feel, persuasive copy).

---

Description Optimization

The frontmatter description is the primary mechanism determining whether Claude invokes a skill. After creating or improving a skill, offer to optimize it for better triggering accuracy.

House description standard (the optimization target)

Any description you write or optimize MUST follow the house standard (rationale + evidence: docs/skill-mining/2026-06-19-skill-description-standard.md):

• Length: two-tier (normal vs router/CONDITIONAL) — number authority: references/description-design.md §Principle 5. Descriptions share a context-listing budget; over-long ones silently evict OTHER skills from what Claude sees.
• Content: what it does + when to use it — positive, specific triggers front-loaded (real user phrasings). Keep the step-by-step procedure / workflow / grounding citations OUT of the description; those live in the body (the body loads in full on activation, so a what+when summary does NOT cause the body to be skipped — that fear is unverified).
• Disambiguate by positive specificity, NOT by "ALWAYS invoke" directives (they over-trigger at scale and are not Anthropic-endorsed) nor "Do NOT use for X" behavioral negation (LLMs handle negation unreliably). A light positive redirect ("for X, use skill-Y") is fine.
• No CJK / multilingual keyword redundancy — cross-lingual reasoning triggers non-English prompts without translated tails (A/B-verified); appended CJK is also truncated first.
• Third person, no XML tags. The optimization loop below must keep best_description within these rules.

Step 1: Generate trigger eval queries

Create 20 eval queries — a mix of should-trigger (8-10) and should-not-trigger (8-10). Save as JSON:

[
  {"query": "the user prompt", "should_trigger": true},
  {"query": "another prompt", "should_trigger": false}
]

Before writing them, read references/description-design.md §Trigger eval query design for the query-writing craft: realistic, concrete, detailed phrasing (with a bad/good example pair); should-trigger coverage across phrasings and competing-skill cases; and should-not-trigger near-misses that are genuinely tricky, never obviously irrelevant.

Step 2: Review with user

Present the eval set to the user inline for review. For each query, show:

| # | Query | Should Trigger? |
|---|-------|-----------------|
| 1 | "ok so my boss sent me this xlsx..." | ✅ Yes |
| 2 | "can you analyze the trends in..." | ❌ No |

Ask the user to confirm, edit, add, or remove queries before proceeding. Save the final eval set to <workspace>/trigger-eval.json.

This step matters — bad eval queries lead to bad descriptions.

Step 3: Run the optimization loop

Tell the user: "This will take some time — I'll run the optimization loop in the background and check on it periodically."

Save the eval set to the workspace, then run in the background:

python -m scripts.run_loop \
  --eval-set <path-to-trigger-eval.json> \
  --skill-path <path-to-skill> \
  --model <model-id-powering-this-session> \
  --max-iterations 5 \
  --verbose

Use the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.

While it runs, periodically tail the output to update the user on which iteration it's on and the scores.

The script handles the full optimization loop automatically (train/held-out-test methodology and its rationale: references/eval-methodology.md). When done, it opens an HTML report in the browser and returns JSON with best_description — selected by test score rather than train score to avoid overfitting.

How skill triggering works

Understanding the triggering mechanism helps design better eval queries — read references/description-design.md (§How skill discovery actually works, plus §Trigger eval query design on why simple one-step queries don't trigger skills even when the description matches).

Step 4: Apply the result

Take best_description from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.

---

Packaging

After the skill is complete, package it for distribution:

python -m scripts.package_skill <path/to/skill-folder>

This creates a .skill file that can be shared and installed.

---

Platform Adaptations

The instructions above assume Claude Code. If you're running in Claude.ai or Cowork, some mechanics change (no subagents, no browser, etc.). Read references/platform-adaptations.md for the platform-specific adjustments.

---

Reference files

The agents/ directory holds instructions for specialized subagents — read when spawning the relevant one.

• agents/grader.md — How to evaluate assertions against outputs
• agents/comparator.md — How to do blind A/B comparison between two outputs
• agents/analyzer.md — How to analyze why one version beat another

The references/ directory has additional documentation:

• references/schemas.md — JSON structures for evals.json, grading.json, etc.
• references/plugin-conventions.md — Plugin ecosystem conventions, directory structures, slash command format
• references/iteration-automation.md — Self-assessment and auto-regression detection protocols
• references/eval-methodology.md — Why the eval workflow is designed this way (optional)
• references/platform-adaptations.md — Claude.ai and Cowork platform-specific adjustments
• references/mermaid-usage-guidelines.md — When to use Mermaid diagrams vs prose in skill authoring, syntax conventions, cost-benefit framework

---

Core loop reminder: Draft → Test → Evaluate (inline review + evals) → Improve → Repeat → Package. Good luck!