메뉴
Use when deciding whether a benchmark is worth deeper benchmark-first mechanistic interpretability work. Covers public availability, runnable access, label richness, product relevance, likely mechanistic question richness, scale, and obvious confounds before investing in latent-label work.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | benchmark-validation |
| description | Use when deciding whether a benchmark is worth deeper benchmark-first mechanistic interpretability work. Covers public availability, runnable access, label richness, product relevance, likely mechanistic question richness, scale, and obvious confounds before investing in latent-label work. |
Use this skill when the user wants to:
This skill is the front door of the benchmark-flow pipeline.
It should answer:
It should not yet do full latent-label conversion or experiment design.
A benchmark being interesting is not the same as a benchmark being ready.
Benchmark existence is evidence of value:
But validation still needs to ask:
Check:
If the benchmark is not accessible enough to inspect, run, or derive labels from, flag that early.
Look for:
Prefer benchmarks where labels are rich enough to support decomposition.
Weak starting points:
Ask:
This does not have to be the only criterion, but it should be explicit.
Ask what internal distinctions the benchmark seems to invite.
Good signs:
Useful prompt:
Check:
Small datasets can still be useful, but note when they are likely seed sets rather than end-state datasets.
Look early for:
If the confounds are severe, note whether they look repairable or fundamental.
At minimum, leave behind:
workspaces/<workspace>/[<thread>/]phase_00/docs/00-validation-memo.md
the human-readable decision memo with:
benchmark-to-latent-labelsworkspaces/<workspace>/[<thread>/]phase_00/docs/00-validation-summary.json
compact structured summary of the memoworkspaces/<workspace>/[<thread>/]phase_00/docs/00-validation-notes.md
optional working notes, edge cases, and unresolved concernsUse simple frontmatter on markdown artifacts:
benchmarkphaseversionfrozen_dateinput_artifactsThe main thing is that someone should be able to inspect:
Read these before making a validation call:
If the benchmark passes:
If the benchmark fails only because of missing controls or structure:
If the benchmark is fundamentally weak for mechanistic work:
Use when planning, running, or reviewing activation patching or interchange experiments for causal claims in mechanistic interpretability. Especially useful for choosing patch sites, designing paired examples, adding same-label controls, distinguishing read layers from write layers, and avoiding overclaiming from weak or lossy interventions.
Use when a benchmark already has a validated latent label spec and the next job is to plan or review the actual mechanistic analysis program. Covers feature hypotheses, readout and localization strategy, probe choice, control design, evidence ladder, and first experiments.
Use when converting a benchmark's native labels into a benchmark-specific latent label spec for mechanistic interpretability. Covers prompt-side vs response-side separation, label-type classification, direct vs derived targets, derivability checks, confounds, ontology freeze, and gap-list handoff.
Constructs linear and nonlinear probes for LLM interpretability in Python. Covers extracting hidden states from transformer models, training probing classifiers/regressors on activations, PCA visualization of representations, SAE feature analysis, logit lens, and causal interventions. Use when the user wants to probe, analyze, or interpret LLM internal representations, build probing classifiers, extract hidden states, or study what information is encoded in model activations.
Use when designing or editing `pipelines_v2` workflow files in this repo. Covers workflow file structure, runner specs, dataset patterns, row alignment, section metadata, report inputs, and the decision boundary between first-class specs and workspace-local transforms.
Walk the documentation tree, cross-check claims against current code, and produce a pruning punch list. Use when docs feel stale, after a methodology shift, or on a regular cadence to fight drift.