菜单
Use when a benchmark cannot support the desired latent labels cleanly and needs rewrites, matched pairs, counterbalancing, response generations, or synthetic augmentation. Covers benchmark repair for confounds, split construction, framing variants, and contrast-set design for mechanistic interpretability.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
Use when planning, running, or reviewing activation patching or interchange experiments for causal claims in mechanistic interpretability. Especially useful for choosing patch sites, designing paired examples, adding same-label controls, distinguishing read layers from write layers, and avoiding overclaiming from weak or lossy interventions.
Use when a benchmark already has a validated latent label spec and the next job is to plan or review the actual mechanistic analysis program. Covers feature hypotheses, readout and localization strategy, probe choice, control design, evidence ladder, and first experiments.
Use when converting a benchmark's native labels into a benchmark-specific latent label spec for mechanistic interpretability. Covers prompt-side vs response-side separation, label-type classification, direct vs derived targets, derivability checks, confounds, ontology freeze, and gap-list handoff.
Use when deciding whether a benchmark is worth deeper benchmark-first mechanistic interpretability work. Covers public availability, runnable access, label richness, product relevance, likely mechanistic question richness, scale, and obvious confounds before investing in latent-label work.
Constructs linear and nonlinear probes for LLM interpretability in Python. Covers extracting hidden states from transformer models, training probing classifiers/regressors on activations, PCA visualization of representations, SAE feature analysis, logit lens, and causal interventions. Use when the user wants to probe, analyze, or interpret LLM internal representations, build probing classifiers, extract hidden states, or study what information is encoded in model activations.
Use when designing or editing `pipelines_v2` workflow files in this repo. Covers workflow file structure, runner specs, dataset patterns, row alignment, section metadata, report inputs, and the decision boundary between first-class specs and workspace-local transforms.
基于 SOC 职业分类
| name | latent-label-data-augmentation |
| description | Use when a benchmark cannot support the desired latent labels cleanly and needs rewrites, matched pairs, counterbalancing, response generations, or synthetic augmentation. Covers benchmark repair for confounds, split construction, framing variants, and contrast-set design for mechanistic interpretability. |
Use this skill when the user wants to:
This skill is the data augmentation and benchmark repair phase of the benchmark-flow pipeline.
It is broader than "synthetic data generation." Often the right move is:
not whole-cloth synthesis.
Augment to break surface-label leakage so the experiment can become real.
The job is to produce data that makes the latent-label question cleaner:
Use the gap list to identify which leaks matter most. Do not treat augmentation as a last-resort cleanup step after phase 03; imported benchmark labels should be assumed leaky until phase 02 has tried to break the most plausible shortcut channels.
Do not begin augmentation from vague dissatisfaction.
This skill should usually consume a gap list from:
Typical gaps:
Every target label should get at least one explicit anti-shortcut repair design.
Do not wait until a probe fails in phase 03 to ask how the label might leak through:
The default phase-02 question is:
Prefer:
before:
Why:
If the downstream question is causal, try to create:
Matched pairs and controls must preserve everything except the target variable.
Preserve means at the level of:
A prefix-only rewrite is not a pair.
If the label lives in:
then benchmark text alone is not enough. Generate responses under the intended protocol and label those.
Simplify toward the core computation, not toward convenience.
The augmented data should still force the intended reasoning problem.
If the target label is exposed through:
then phase 02 should materialize a shortcut_stress_test repair family before calling that track clean.
Useful repair patterns include:
The goal is not just "more variants." The goal is to break one-to-one surface recoverability.
Constructing variants is not the same as having a working holdout. A variant pair earns holdout status only when a within-variant text classifier on the responses they produce lands near chance.
Required validation step:
Repair if the validation fails:
A track that ships variant pairs without running this validation has gestured at the technique rather than executing it. Mark such tracks as shortcut_stress_unvalidated in the manifest until validation passes.
This failure mode has shown up in real benchmark-repair work: a natural-prompt within-variant text classifier landed near ceiling, and the variants were not functioning as holdouts despite being labeled as such. Treat validation as a required gate, not a retrospective explanation.
Always consider:
If possible, construct data that lets these be:
For targets vulnerable to lexical or semantic leakage, phase 02 should also run a prompt-side preflight against cheap baselines before handing the family to phase 03.
Typical preflight baselines:
If these baselines still solve the target family cleanly, phase 02 is not done for that track.
Materialized artifacts in outputs/ must be fully instantiated.
Do not place literal placeholder tokens such as:
<FRAMEWORK><THEORY><LABEL>inside materialized JSON, JSONL, CSV, or parquet outputs.
If templates with holes are useful, store them under specs/ or specs/templates/.
Produce:
scaffold_onlypartial_repair_materializedrepair_complete_for_trackphase_completeAt minimum, leave behind:
workspaces/<workspace>/[<thread>/]phase_02/docs/02-augmentation-plan.md
what is being repaired and whyworkspaces/<workspace>/[<thread>/]phase_02/docs/02-gap-list-resolution.md
direct mapping from each gap-list item to the chosen repair moveworkspaces/<workspace>/[<thread>/]phase_02/docs/02-augmented-data-manifest.json
where rewritten, paired, generated, or counterbalanced data now livesworkspaces/<workspace>/[<thread>/]phase_02/docs/02-augmentation-report.md
what was changed, what improved, and what residual confounds remainworkspaces/<workspace>/[<thread>/]phase_02/docs/02-behavioral-smoke-report.md
small-slice behavioral sanity check on the augmented data before phase 03If a phase materializes prompts or responses, also leave behind:
workspaces/<workspace>/[<thread>/]phase_02/docs/02-generation-protocol.md
prompt / rewrite / generation rules used for the materialized dataIf new rows or slices were created, also leave behind:
If the smoke artifact is used to justify downstream response-side labeling or probing, it should also make explicit:
Use simple frontmatter on markdown artifacts:
benchmarkphaseversionfrozen_dateinput_artifactsSomeone inspecting this phase should be able to answer:
The augmented-data manifest should make quality visible, not just existence. At minimum, each materialized dataset entry should record:
row_countrows_with_all_placeholders_substitutedcontrols_structurally_matched_to_target when applicableknown_bugsIf a materialized family is later shown to be shortcut-dominated, keep it in the manifest and record that explicitly in known_bugs rather than silently treating it as clean.
If a confound-repair matrix marks a track as ready_now, the phase should either materialize that track or explicitly downgrade it with a reason.
This phase is done when:
ready_now repair tracks are materialized or explicitly downgraded with a reasonoutputs/ contain no unsubstituted placeholdersThe behavioral smoke report should distinguish:
tripwire checks
examples: nonempty output, parseability, no truncation, no refusal, recommendation cue presentsubstantive labelability checks
examples: whether the generated responses can actually support the proposed downstream labelsIf the smoke is intended to support response-side label work, a pure tripwire pass is not enough by itself. The report should include at least one explicit content-inspection section tied to the proposed labels.
Primary references:
Use the synthetic-data-generation skill as the main methodology library for:
Once the data is repaired enough to support the latent labels: