| name | writing-science-skills |
| description | Use when creating new skills, editing existing skills, or verifying skills work before deployment |
Writing Science Skills
Overview
Writing a skill is an experiment, not an essay. You form a hypothesis (this guidance will change agent behavior), run a baseline (watch agents fail without it), write the skill, and test that behavior changes.
Core principle: If you didn't watch an agent fail without the skill, you don't know whether the skill teaches the right thing — the same reason you pre-register before analyzing rather than rationalizing a result after.
REQUIRED BACKGROUND: Understand science-superpowers:preregistering-analysis first. The discipline of fixing a prediction before observing the outcome is exactly the discipline here: observe baseline behavior before writing the skill, so the skill is shaped by evidence, not by what you imagine agents do.
What is a Skill?
A skill is a reference guide for a proven technique, pattern, or discipline that future agents can find and apply.
Skills are: reusable techniques, patterns, disciplines, reference guides.
Skills are NOT: narratives about how you did something once.
The Empirical Mapping
| Experiment concept | Skill creation |
|---|
| Hypothesis | "This guidance will fix behavior X" |
| Baseline / control | Agent behavior WITHOUT the skill |
| Observed failure (the signal) | The exact rationalizations agents use |
| Intervention | The skill document |
| Effect | Agent now complies |
| Replication | Re-test under different pressures until robust |
When to Create a Skill
Create when:
- The technique wasn't obvious and you'd reference it again across projects
- It applies broadly (not project-specific)
Don't create for:
- One-off solutions
- Standard practice documented elsewhere
- Project-specific conventions (put those in CLAUDE.md)
- Anything enforceable mechanically (validate it in code instead of documenting a judgment call)
Skill Types
- Technique — a concrete method with steps (e.g., investigating-anomalous-results)
- Pattern — a way of thinking (e.g., separating confirmatory from exploratory)
- Discipline — a rule that must resist rationalization (e.g., preregistering-analysis)
- Reference — API/command/method documentation (e.g., statistical-fallacies)
Directory Structure
skills/
skill-name/
SKILL.md # Main reference (required)
supporting-file.* # Only for heavy reference or reusable tools
Flat namespace. Keep principles and short patterns inline; split out only heavy reference (100+ lines) or reusable tools/templates.
SKILL.md Structure
Frontmatter (YAML): two required fields, name and description, max 1024 characters total.
name: letters, numbers, hyphens onlydescription: third person, describes ONLY WHEN to use (not what it does); start with "Use when..."
Description = When to Use, NOT What the Skill Does
This is the single most important authoring rule. If the description summarizes the workflow, the agent follows the summary and skips the skill body.
description: Use when analyzing data - pre-register, run the test, check assumptions, then report
description: Use before running any confirmatory analysis or looking at outcome data
Use concrete triggers and symptoms. Describe the problem (a convenient result, a surprising number), not just the topic. Keep it technology-agnostic unless the skill is technology-specific.
Claude Search Optimization (CSO)
Future agents must FIND your skill.
- Rich description — the triggering conditions an agent would recognize
- Keyword coverage — error messages, symptoms ("leakage", "won't converge", "p-hacked"), synonyms, tool/library names
- Descriptive, verb-first naming —
investigating-anomalous-results not anomaly-utils; gerunds work well for processes
- Token efficiency — frequently-loaded skills should be tight; move detail to references, use cross-references instead of repetition
Cross-Referencing Other Skills
Use the skill name with an explicit requirement marker:
- Good:
**REQUIRED SUB-SKILL:** Use science-superpowers:preregistering-analysis
- Bad:
See skills/.../SKILL.md (unclear if required)
- Bad:
@skills/.../SKILL.md (force-loads, burns context)
Never use @ links to other skills — they load immediately and consume context before it's needed.
Flowchart Usage
Use a small graphviz dot flowchart ONLY for a non-obvious decision point or a loop where the agent might stop too early. Never for reference material (use tables/lists), code (use code blocks), or linear steps (use numbered lists). Keep node labels semantic.
Code & Examples
One excellent, complete, runnable example beats five mediocre ones. For a computational-science framework, prefer Python for data work and shell for environment/pipeline work. Comment WHY, not what. Don't write fill-in-the-blank templates.
The Iron Law
NO SKILL WITHOUT AN OBSERVED BASELINE FAILURE FIRST
Applies to NEW skills AND EDITS. Wrote the skill before observing the baseline? You're guessing at what agents need. Delete it, run the baseline, start over.
No exceptions — not for "simple additions", not for "just a section", not for "documentation updates". Don't keep untested changes as "reference".
Testing All Skill Types
Different types need different tests. Dispatch fresh subagents (no shared context) and observe.
- Discipline skills (preregistering-analysis, verifying-results-before-claiming): pressure scenarios — time pressure, a convenient result, sunk cost, authority. Success = agent holds the line under combined pressure. Capture every rationalization and add an explicit counter.
- Technique skills (investigating-anomalous-results): application scenarios and edge cases. Success = agent applies the technique correctly to a new case.
- Pattern skills: recognition + application + counter-examples. Success = agent knows when it applies and when it doesn't.
- Reference skills (statistical-fallacies): retrieval + application. Success = agent finds and correctly uses the right entry.
RED — GREEN — REFACTOR for Skills
- RED (baseline): run the scenario WITHOUT the skill. Document the exact choices and rationalizations verbatim.
- GREEN (minimal skill): write the skill addressing those specific failures. Re-run; confirm compliance.
- REFACTOR (close loopholes): new rationalization appears? Add an explicit counter. Re-test until robust.
Bulletproofing Discipline Skills
- Close every loophole explicitly — don't just state the rule; forbid the specific workarounds ("Don't relabel the peeked analysis confirmatory").
- State the foundational principle — "Violating the letter is violating the spirit" cuts off a whole class of rationalizations.
- Build a rationalization table — every excuse from baseline testing, with its reality.
- Create a Red Flags list — the thoughts that signal the agent is about to violate the rule.
Common Rationalizations for Skipping Testing
| Excuse | Reality |
|---|
| "Skill is obviously clear" | Clear to you ≠ clear to another agent. Test it. |
| "It's just a reference" | References have gaps. Test retrieval. |
| "No time to test" | Deploying an untested skill costs more time later. |
| "Academic review is enough" | Reading ≠ using. Test application. |
Checklist (use TodoWrite for each item)
RED:
GREEN:
REFACTOR:
Quality:
STOP: Before Moving to the Next Skill
After writing ANY skill, STOP and complete its testing before starting another. Don't batch. Deploying an untested skill is deploying an untested intervention.
The Bottom Line
Creating a skill IS running an experiment on agent behavior. Same discipline as pre-registration: observe the baseline before you intervene, and let evidence — not your imagination — shape the result.
