skill-creator

name	skill-creator
description	Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, update or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. Triggers on "create skill", "build skill", "improve skill", "test skill", "optimize description".
allowed-tools	Read, Write, Edit, Bash, Glob, Grep, Agent, AskUserQuestion, WebFetch, WebSearch
version	1.0.0
scope	public

Skill Creator

Create new skills and iteratively improve them through testing.

At a high level, the process goes like this:

• Decide what you want the skill to do and roughly how it should do it
• Write a draft of the skill
• Create a few test prompts and run claude-with-access-to-the-skill on them
• Help the user evaluate the results both qualitatively and quantitatively
• Rewrite the skill based on feedback
• Repeat until satisfied
• Expand the test set and try at larger scale
• Optimize the description for triggering accuracy
• Register to SKM (local convention)

Your job is to figure out where the user is in this process and help them progress. Maybe they want to create from scratch, or maybe they already have a draft and want to eval/iterate.

Be flexible — if the user says "I don't need evals, just vibe with me", do that instead.

Communicating with the user

Pay attention to context cues for jargon familiarity:

• "evaluation" and "benchmark" are borderline OK
• "JSON" and "assertion" need cues from the user before using without explanation
• Briefly explain terms when in doubt

---

Phase 1: Creating a Skill

Capture Intent

Start by understanding the user's intent. The current conversation might already contain a workflow to capture. If so, extract answers from history first.

1. What should this skill enable Claude to do?
2. When should this skill trigger? (what user phrases/contexts)
3. What's the expected output format?
4. Should we set up test cases? (recommend based on skill type, let user decide)

Interview and Research

Proactively ask about edge cases, input/output formats, example files, success criteria, dependencies. Check available MCPs for research. Come prepared with context.

Local convention: Ask which pack this skill belongs to (core, quality, ops, trading, or new). See references/local-conventions.md for pack details.

Optional: Ask if the user wants to integrate domain expert methodology. See references/local-conventions.md for the methodology integration process.

Write the SKILL.md

Based on the interview, fill in:

• name: kebab-case, matches folder name
• description: When to trigger + what it does. Make it slightly "pushy" to combat undertriggering. Include both Chinese and English trigger phrases for bilingual users.
• compatibility: Required tools, dependencies (optional)
• the rest of the skill

Skill Writing Guide

Anatomy of a Skill

skill-name/
├── SKILL.md (required)
│   ├── YAML frontmatter (name, description required)
│   └── Markdown instructions
└── Bundled Resources (optional)
    ├── scripts/    - Executable code for deterministic/repetitive tasks
    ├── references/ - Docs loaded into context as needed
    └── assets/     - Files used in output (templates, icons, fonts)

Progressive Disclosure

Skills use a three-level loading system:

1. Metadata (name + description) - Always in context (~100 words)
2. SKILL.md body - In context when skill triggers (<500 lines ideal)
3. Bundled resources - As needed (unlimited, scripts execute without loading)

Key patterns:

• Keep SKILL.md under 500 lines; add hierarchy with clear pointers if approaching limit
• Reference files clearly from SKILL.md with guidance on when to read them
• For large reference files (>300 lines), include a table of contents

Domain organization: When a skill supports multiple domains/frameworks, organize by variant:

cloud-deploy/
├── SKILL.md (workflow + selection)
└── references/
    ├── aws.md
    ├── gcp.md
    └── azure.md

Writing Patterns

• Use imperative form in instructions
• Define output formats with exact templates
• Include realistic examples with input/output pairs
• Explain WHY things are important (theory of mind) rather than heavy-handed MUSTs
• Make skills general, not narrow to specific examples
• Write draft, then review with fresh eyes

Principle of Lack of Surprise

Skills must not contain malware, exploit code, or security-compromising content. A skill's contents should not surprise the user in their intent.

Test Cases

After writing the draft, create 2-3 realistic test prompts. Share with the user and get confirmation. Save to evals/evals.json — just prompts, no assertions yet.

{
  "skill_name": "example-skill",
  "evals": [
    {
      "id": 1,
      "prompt": "User's task prompt",
      "expected_output": "Description of expected result",
      "files": []
    }
  ]
}

See references/schemas.md for the full schema.

---

Phase 2: Testing and Iteration

This is one continuous sequence. Put results in <skill-name>-workspace/ as a sibling to the skill directory. Organize by iteration (iteration-1/, iteration-2/) and eval (eval-0/, eval-1/).

Step 1: Spawn all runs in the same turn

For each test case, spawn two subagents simultaneously — one with the skill, one without (baseline).

With-skill run:

Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <eval files if any, or "none">
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
- Outputs to save: <what the user cares about>

Baseline run:

• Creating new skill: no skill at all -> without_skill/outputs/
• Improving existing skill: snapshot old version first (cp -r), then use snapshot -> old_skill/outputs/

Write eval_metadata.json for each test case with descriptive name.

Step 2: Draft assertions while runs are in progress

Don't wait — use this time to draft quantitative assertions. Good assertions are objectively verifiable with descriptive names. Subjective skills are better evaluated qualitatively.

Update eval_metadata.json and evals/evals.json with assertions.

Step 3: Capture timing data as runs complete

When each subagent completes, save total_tokens and duration_ms to timing.json. This data comes through task notification only — capture immediately.

Step 4: Grade, aggregate, and launch viewer

1. Grade each run — spawn grader (read agents/grader.md). Save to grading.json. Fields must use text, passed, evidence.

2. Aggregate into benchmark:

   python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name>
   

   Produces benchmark.json and benchmark.md.

3. Analyst pass — read benchmark data, surface patterns. See agents/analyzer.md (Benchmark section).

4. Launch viewer:

   nohup python <skill-creator-path>/eval-viewer/generate_review.py \
     <workspace>/iteration-N \
     --skill-name "my-skill" \
     --benchmark <workspace>/iteration-N/benchmark.json \
     > /dev/null 2>&1 &
   VIEWER_PID=$!
   

   For iteration 2+: add --previous-workspace <workspace>/iteration-<N-1>. Headless: use --static <output_path>.

5. Tell the user about the two tabs: Outputs (qualitative review + feedback) and Benchmark (quantitative stats).

Step 5: Read feedback

Read feedback.json when user is done. Empty feedback = fine. Focus on specific complaints.

kill $VIEWER_PID 2>/dev/null

Improving the Skill

1. Generalize from feedback — don't overfit to test examples. Try different metaphors if something is stubborn.
2. Keep the prompt lean — read transcripts, remove unproductive sections.
3. Explain the why — theory of mind over ALL CAPS MUSTs.
4. Look for repeated work — if all test runs wrote similar scripts, bundle in scripts/.

The iteration loop

1. Apply improvements
2. Rerun all test cases into iteration-<N+1>/ with baseline
3. Launch reviewer with --previous-workspace
4. Wait for user review
5. Read feedback, improve, repeat

Stop when: user is happy, feedback is all empty, or no meaningful progress.

Advanced: Blind comparison

For rigorous A/B testing between skill versions, read agents/comparator.md and agents/analyzer.md. Optional, requires subagents.

---

Phase 3: Description Optimization

After the skill is solid, optimize the description for triggering accuracy.

Step 1: Generate trigger eval queries

Create 20 eval queries — mix of should-trigger (8-10) and should-not-trigger (8-10).

[
  {"query": "the user prompt", "should_trigger": true},
  {"query": "another prompt", "should_trigger": false}
]

Queries must be realistic with detail (file paths, personal context, company names, typos, casual speech). For should-not-trigger, use near-misses — not obviously irrelevant queries.

Step 2: Review with user

Use assets/eval_review.html template:

1. Replace __EVAL_DATA_PLACEHOLDER__, __SKILL_NAME_PLACEHOLDER__, __SKILL_DESCRIPTION_PLACEHOLDER__
2. Write to temp file and open it
3. User edits, clicks "Export Eval Set" -> downloads eval_set.json

Step 3: Run the optimization loop

python -m scripts.run_loop \
  --eval-set <path-to-trigger-eval.json> \
  --skill-path <path-to-skill> \
  --model <model-id-powering-this-session> \
  --max-iterations 5 \
  --verbose

This splits 60/40 train/test, evaluates trigger rates (3 runs per query), uses extended thinking to propose improvements, and selects best by test score to avoid overfitting.

Step 4: Apply the result

Take best_description and update SKILL.md frontmatter. Show before/after and report scores.

How skill triggering works

Skills appear in Claude's available_skills with name + description. Claude only consults skills for tasks it can't easily handle on its own — simple one-step queries may not trigger even with perfect descriptions. Eval queries should be substantive enough that Claude would benefit from consulting a skill.

---

Phase 4: Local Registration

After the skill is complete and tested, register it in the local SKM system.

Quick path

Note: This is the new skill creation workflow (adding to the local packs registry). For installing an existing skill from a registry, use skm install <skill> instead — never use manual symlinks for that.

REPO=~/site/skills/claude-skills
PACK="[pack]"
SKILL="[name]"

# Create in packs/ (source of truth)
mkdir -p $REPO/packs/$PACK/$SKILL

# Symlink packs/ → ~/.claude/skills/ (internal registration only)
ln -s $REPO/packs/$PACK/$SKILL ~/.claude/skills/$SKILL

# Update registry.yaml, then sync
cd $REPO && ./skm sync

Security audit

After registration, run security audit automatically. See references/local-conventions.md for full details.

Validation checklist

• Frontmatter — name, description, allowed-tools complete
• Description — includes trigger phrases, under 1024 chars, slightly "pushy"
• Progressive disclosure — SKILL.md < 500 lines, details in references/
• Test cases — evals created and passing
• SKM — registry.yaml updated, skm sync run, symlink verified
• Security — audit passed or risks acknowledged

---

Packaging (if present_files tool available)

python -m scripts.package_skill <path/to/skill-folder>

Quick validation before packaging:

python -m scripts.quick_validate <path/to/SKILL.md>

---

Reference files

Agents (read when spawning subagents)

• agents/grader.md — Evaluate assertions against outputs
• agents/comparator.md — Blind A/B comparison
• agents/analyzer.md — Post-hoc analysis + benchmark pattern detection

References

• references/schemas.md — JSON schemas (evals, grading, benchmark, comparison, analysis, timing, metrics)
• references/local-conventions.md — Pack selection, SKM registration, methodology integration, security audit, evolution metadata
• references/patterns.md — Skill writing patterns (sequential workflow, multi-MCP, iterative refinement, context-aware selection, domain intelligence)

Scripts

• scripts/run_loop.py — Description optimization loop (train/test split, extended thinking)
• scripts/run_eval.py — Trigger rate testing (uses claude -p)
• scripts/improve_description.py — Description improvement with extended thinking
• scripts/aggregate_benchmark.py — Aggregate grading -> benchmark.json + .md
• scripts/generate_report.py — Description optimization HTML report
• scripts/quick_validate.py — SKILL.md frontmatter validation
• scripts/package_skill.py — Package skill as .skill (zip)

Eval Viewer

• eval-viewer/generate_review.py — Generate + serve eval review page (port 3117)
• eval-viewer/viewer.html — SPA viewer template (Outputs + Benchmark tabs)

Assets

• assets/eval_review.html — Trigger eval set editor for user review

---

Skill Categories

When creating or improving a skill, identify which category it falls into:

Category	Description	Eval approach
Capability Uplift	Base model can't do this consistently; skill teaches techniques	Focus on pass/fail — did the skill enable the capability?
Encoded Preference	Model handles components but needs specific sequencing	Focus on workflow compliance — did it follow the process?

---

Core loop reminder:

1. Figure out what the skill is about
2. Draft or edit the skill
3. Run claude-with-access-to-the-skill on test prompts
4. With the user, evaluate outputs (create benchmark.json, run eval-viewer/generate_review.py)
5. Repeat until satisfied
6. Optimize description with scripts/run_loop.py
7. Register to SKM and run security audit