메뉴
Use when converting a benchmark's native labels into a benchmark-specific latent label spec for mechanistic interpretability. Covers prompt-side vs response-side separation, label-type classification, direct vs derived targets, derivability checks, confounds, ontology freeze, and gap-list handoff.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Use when planning, running, or reviewing activation patching or interchange experiments for causal claims in mechanistic interpretability. Especially useful for choosing patch sites, designing paired examples, adding same-label controls, distinguishing read layers from write layers, and avoiding overclaiming from weak or lossy interventions.
Use when a benchmark already has a validated latent label spec and the next job is to plan or review the actual mechanistic analysis program. Covers feature hypotheses, readout and localization strategy, probe choice, control design, evidence ladder, and first experiments.
Use when deciding whether a benchmark is worth deeper benchmark-first mechanistic interpretability work. Covers public availability, runnable access, label richness, product relevance, likely mechanistic question richness, scale, and obvious confounds before investing in latent-label work.
Constructs linear and nonlinear probes for LLM interpretability in Python. Covers extracting hidden states from transformer models, training probing classifiers/regressors on activations, PCA visualization of representations, SAE feature analysis, logit lens, and causal interventions. Use when the user wants to probe, analyze, or interpret LLM internal representations, build probing classifiers, extract hidden states, or study what information is encoded in model activations.
Use when designing or editing `pipelines_v2` workflow files in this repo. Covers workflow file structure, runner specs, dataset patterns, row alignment, section metadata, report inputs, and the decision boundary between first-class specs and workspace-local transforms.
Walk the documentation tree, cross-check claims against current code, and produce a pruning punch list. Use when docs feel stale, after a methodology shift, or on a regular cadence to fight drift.
| name | benchmark-to-latent-labels |
| description | Use when converting a benchmark's native labels into a benchmark-specific latent label spec for mechanistic interpretability. Covers prompt-side vs response-side separation, label-type classification, direct vs derived targets, derivability checks, confounds, ontology freeze, and gap-list handoff. |
Use this skill when the user wants to:
This skill is the ingestion and labeling phase of the benchmark-flow pipeline.
It turns:
into:
Grader-designed labels are not automatically latent-designed labels.
Benchmark existence is evidence of value, but not proof of mechanistic tractability.
Start from the benchmark as a crystallized domain artifact. Then recover the implicit mechanistic questions it seems to ask. Then audit whether the benchmark's labels really support those questions.
Before starting, record:
These choices affect which labels are even meaningful.
Use this workflow as the canonical benchmark-to-latent-labels process. Older long-form planning notes are not active dependencies for this skill.
Record:
Do not start from an external wishlist.
Ask:
Ask:
For each label, record:
Useful buckets:
Use:
Prefer a small first pass:
For each surviving label, define:
Use:
If a label cannot be assigned consistently, drop or narrow it.
Always check:
Commit the label spec before activation work.
This is the wall against post-hoc relabeling.
If some labels are still honestly blocked:
Do not silently skip the freeze because one label family still needs augmentation or generations.
Explicitly note what the benchmark still cannot support without augmentation.
At minimum, leave behind:
workspaces/<workspace>/[<thread>/]phase_01/docs/01-latent-label-spec.md
canonical human-readable spec for the benchmark's refined ontologyworkspaces/<workspace>/[<thread>/]phase_01/docs/01-label-inventory.csv or .parquet
native labels, their types, and latent-usefulness ratingsworkspaces/<workspace>/[<thread>/]phase_01/docs/01-labeling-functions.md
extracted mechanistic questions, surviving labels, and how each is assigned and validatedworkspaces/<workspace>/[<thread>/]phase_01/docs/01-confound-audit.md
target-vs-nuisance concerns and planned controlsworkspaces/<workspace>/[<thread>/]phase_01/docs/01-frozen-label-set.csv or .parquet
row-level first-pass labels with version/freeze metadata
workspaces/<workspace>/[<thread>/]phase_01/docs/01-gap-list.md
what must be repaired or augmented before deeper workworkspaces/<workspace>/[<thread>/]phase_01/docs/01-latent-label-summary.json
compact structured summary of the refined ontology, labels, and next actionOptional but useful:
workspaces/<workspace>/[<thread>/]phase_01/docs/01-derivability-report.mdworkspaces/<workspace>/[<thread>/]phase_01/docs/01-baseline-report.mdUse simple frontmatter on markdown artifacts:
benchmarkphaseversionfrozen_dateinput_artifactsSomeone reviewing this phase should be able to see:
This phase is done when:
Primary references:
Related skills:
When the ontology is frozen: