菜单
Use when designing or repairing benchmark and prompt data so later experiments are behaviorally sane and experimentally valid. Especially useful for lexical controls, split schemes, matched pairs, prompt role placement, relational task design, and preserving the real decision bottleneck.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Use when planning, running, or reviewing activation patching or interchange experiments for causal claims in mechanistic interpretability. Especially useful for choosing patch sites, designing paired examples, adding same-label controls, distinguishing read layers from write layers, and avoiding overclaiming from weak or lossy interventions.
Use when a benchmark already has a validated latent label spec and the next job is to plan or review the actual mechanistic analysis program. Covers feature hypotheses, readout and localization strategy, probe choice, control design, evidence ladder, and first experiments.
Use when converting a benchmark's native labels into a benchmark-specific latent label spec for mechanistic interpretability. Covers prompt-side vs response-side separation, label-type classification, direct vs derived targets, derivability checks, confounds, ontology freeze, and gap-list handoff.
Use when deciding whether a benchmark is worth deeper benchmark-first mechanistic interpretability work. Covers public availability, runnable access, label richness, product relevance, likely mechanistic question richness, scale, and obvious confounds before investing in latent-label work.
Constructs linear and nonlinear probes for LLM interpretability in Python. Covers extracting hidden states from transformer models, training probing classifiers/regressors on activations, PCA visualization of representations, SAE feature analysis, logit lens, and causal interventions. Use when the user wants to probe, analyze, or interpret LLM internal representations, build probing classifiers, extract hidden states, or study what information is encoded in model activations.
Use when designing or editing `pipelines_v2` workflow files in this repo. Covers workflow file structure, runner specs, dataset patterns, row alignment, section metadata, report inputs, and the decision boundary between first-class specs and workspace-local transforms.
| name | synthetic-data-generation |
| description | Use when designing or repairing benchmark and prompt data so later experiments are behaviorally sane and experimentally valid. Especially useful for lexical controls, split schemes, matched pairs, prompt role placement, relational task design, and preserving the real decision bottleneck. |
Use this skill when the user wants to:
This skill is for experiment design, not just text generation. The goal is to create data that the model can solve for the right reasons, and that later analysis can interpret cleanly.
For benchmark-first mechanistic interpretability, this skill is also the main craft library for breaking shortcut channels in imported benchmarks.
Behavioral sanity comes before interpretability.
Before treating probe results, patching results, or attention results as meaningful, first verify that the benchmark itself is behaviorally sane.
If the base task is malformed, ambiguous, or solved via role-format artifacts, later interpretability work will be misleading.
Synthetic data should be:
Imported benchmarks leak by construction. If a benchmark was designed so humans or graders could apply labels reliably, those labels are usually recoverable from surface semantics too.
One of the main jobs of synthetic-data-generation is therefore:
This is not downstream cleanup. It is one of the primary crafts of benchmark-to-mech-interp work.
When a target family is shortcut-satisfiable, prefer repairs like these:
Keep the semantic content fixed while varying the surface form.
Train on one surface realization of a concept and test on another:
Create examples where easy surface cues pull one way while the intended semantic label pulls another.
For labels with multiple possible surface carriers, vary them independently:
Use overlapping vocabulary across labels so a small keyword list cannot solve the task.
Redact one shortcut channel while preserving the rest of the prompt:
These are good repairs only if a careful human could still recover the intended label.
Weak anti-shortcut control:
Stronger anti-shortcut design:
Anchor only is useful here only as one cell in a larger factorial decomposition.
By itself it is not a credible anti-shortcut test if the anchor sentence still identifies the framework cleanly.
Weak anti-shortcut setup:
Stronger anti-shortcut design:
The craft target is not "remove all easy words." It is:
The best synthetic benchmarks usually do not start from nowhere. They start from a real dataset, real workflow, or real failure mode, and then abstract it into a smaller controlled task.
The right question is:
This is often better than trying to synthesize "realism" directly.
Good process:
Examples:
Why this works:
When abstracting from real data, explicitly separate:
Core elements to preserve are the pieces without which the target question changes.
Examples:
Messy details to remove are things that make the prompt realistic but do not matter to the core computation.
Examples:
There are two bad simplifications:
The right simplification preserves the decision bottleneck.
Ask:
If the answer becomes "the model can now solve this from one keyword," you simplified too far.
When possible, document:
This makes the benchmark easier to defend later, especially in research writeups.
Before writing prompts, define:
Examples:
claimed authority applies to requested actionprompt looks security-relatedIf the latent variable is relational, the prompts must force a relational computation.
Ask:
For authority-applicability, the intended computation was:
That is much better than simply presenting a good holder in one class and a nonsense string in the other.
In many synthetic benchmarks, the surrounding "world state" is not neutral background. It is part of how the latent variable becomes behaviorally visible.
This includes things like:
Rule:
Examples:
Bad pattern:
Better pattern:
If the task turns into:
use the repo-operational skills instead:
The benchmark-level latent variable can stay constant while the readout changes by family.
Example:
instruction_source_followedPossible family-specific readouts:
Do not force every family into the same behavioral readout if that makes the task unnatural or ambiguous.
Avoid lazy negatives.
Bad negative design:
Good negative design:
Examples:
treasury_officer vs audit_controller for a finance actionrepository_maintainer vs deployment_operator for a code actiontreasury_officer vs bananaThe wrong answer should still be believable inside the world of the prompt.
Prompt placement is part of the experiment, not wrapper text.
Be deliberate about:
Rules:
When using system, user, and tool carriers:
Synthetic prompts should make sense to the agent being modeled.
Ask:
A prompt can be lexically balanced and still unrealistic from an agent-logic perspective.
Examples of agent-logic failures:
When in doubt, write prompts that feel like something the target agent could actually receive in a workflow.
For early experiments, prefer tightly constrained outputs over freeform answers.
Good:
DECISION: COMPLY or ESCALATEWhy:
Do not start with open-ended natural language outputs unless the generation style is itself part of the experiment.
If you skip lexical controls, your benchmark will almost always overstate abstraction.
At minimum vary:
Then split train and test so that held-out lexical families appear only at test.
Best practice:
If the intended claim is that the model learned a relation rather than a familiar phrase, create a vocab-controlled variant.
That means:
Your split scheme should match the claim you want to make.
Common split types:
Do not claim abstraction across a dimension you never actually held out.
If you may later do activation patching or interchange, design the dataset so paired examples exist naturally.
Ideal matched pair:
For instruction-conflict datasets, this usually means:
Important:
Good audit scheme:
aligned_agreementstrategy_followedsetting_followedmixed_or_neitherThen derive a smaller training target later if needed.
If the user wants a benchmark that is reproducible and auditable, do not let the generator improvise.
Lock named inventories for:
Then make the generator instantiate only those inventories.
This avoids a common failure mode where the markdown spec looks precise but the actual script reintroduces ambiguity by inventing details on the fly.
Useful pattern:
A human should be able to inspect one row and answer:
If a human cannot audit rows quickly, the dataset will be painful to debug and easy to misinterpret.
Before probing:
If behavior is wrong, ask:
Do not move on to probing until this is addressed.
Synthetic data generation is not "make realistic-looking text."
It is:
designing a controlled environment where later behavioral and mechanistic claims will still be trustworthy.