Mechanist

Representation and Parameter Analysis interprets and controls a model by directly manipulating its two kinds of internal objects — features (the hidden-state activations produced during a forward pass) and weights (the parameters of the target model). The analysis side relies on standard linear-algebra tools applied straight to these objects: techniques such as PCA, mean-difference, or simple linear classifiers are run over collected features (e.g., from contrastive prompt pairs) or over weight differences between a fine-tuned and a pre-trained checkpoint, in order to extract a small set of meaningful directions. The control side then operates directly through arithmetic on those objects: adding such a direction back into the features steers behavior at inference time, while adding, subtracting, or combining task-level weight differences edits the model in one shot — so that interpretability findings translate into behavioral control without any retraining.

Use this skill for feature-level steering of models — locating the internal feature that drives a target behavior, scoring and selecting it by its effect on the model's output, and directly amplifying or shrinking that feature's activation during generation to control behavior. Applies to features read from the model's own activations or from a Sparse Autoencoder (SAE); the bundled demo scripts happen to use an SAE, but the method does not require one.

Routing entry point for eleven families of mechanistic-interpretability methods that localize *which* internal object (layer, attention head, neuron, SAE feature, weight, or input feature) drives a model's behavior, *how influential* it is, and *what changes* when it is intervened on. Use this skill whenever the question is about a model's internal mechanism rather than its external metrics — for example, claims that a specific component is responsible for a behavior, mechanistic-evidence requests, circuit-discovery tasks, feature-attribution work (SHAP), or concept-level explanations of vision/VL models. The file lays out each family's premise, signal, cost, advantages, limitations, and how to compose them into a cheap-screen → causal-verify pipeline. Loading is hierarchical and mandatory: after picking a family from this routing file you MUST load that family's `SKILL.md`, and after picking a submethod you MUST load that submethod's `SKILL.md` — never act on the previews in this file alone.

Vocabulary Projection methods interpret internal model states by projecting them through the unembedding matrix to obtain a distribution over the vocabulary. The core idea is that the unembedding matrix, which maps the final hidden state to output logits, can serve as a universal decoder for intermediate states throughout the model. By inspecting the top-ranked tokens of the resulting distribution, researchers can directly read off the semantic content encoded in any internal object — such as a residual stream state or an attention head output — in terms of the model's output vocabulary.

Use when main results pass result-to-claim (claim_supported=yes or partial) and ablation studies are needed for paper submission. The external LLM reviewer (via llm-chat MCP) designs ablations from a reviewer's perspective, CC reviews feasibility and implements.

Search, download, and summarize academic papers from arXiv. Use when user says "search arxiv", "download paper", "fetch arxiv", "arxiv search", "get paper pdf", or wants to find and save papers from arXiv to the local paper library.

Workflow 1: Claim-stage pipeline, controlled by two orthogonal axes. BEHAVIOR_SOURCE selects the behavior stage: `given` (default; behavior taken from task.md and assumed to hold — no ideation, no novelty, no M0), `given-validation` (behavior taken from task.md but the experiment plan opens with an M0 phenomenon-validation gate), or `discovery` (mine a NEW behavior via /mechanism-behavior-discovery + full ideation: research-lit → idea-creator → novelty-check → impact-check → research-review → research-refine-pipeline, impact-first ranking; plan opens with M0). MECHANISM selects the mechanism stage: `discovery` (default; system routes — claim loads /mechanism-explore to shape direction) or `given` (user named the mechanism method/family in task.md — capture it and forward as CHOSEN_FAMILY, no routing). resource_fidelity:strict (exact models/datasets, no downscaling) is stamped iff BEHAVIOR_SOURCE=given AND MECHANISM=given (the reproduction combination). Always emits `idea-stage/IDEA_REPORT.md` + `refine-logs/{

Workflow 1.5: Bridge between idea discovery and auto review. Reads EXPERIMENT_PLAN.md, routes mechanism family inline (Phase 1.5), implements experiment code, deploys to GPU, and collects initial results. Use when user says "implement experiments", "experiment", "deploy the plan", or has an experiment plan ready to execute.

Autonomous research review loop that consumes /auto-verify's four-state output (PASS / FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS / deferred) and routes each claim to the right back-edge — brief audit, two-phase FAIL handling (variant-integrity fix then optional claim-stage re-entry), main-experiment-script fix, or variant-only fix — under a unified iteration budget. Configure the reviewer LLM via llm-chat MCP server or environment variables. Trigger with "auto review loop llm" or "llm review".

Autonomous pipeline: claim → experiment (mechanism routing folded in) → verify → iteration. Each stage is delegated to an isolated agent with its own context window and configurable model. Gates are AUTO_PROCEED-governed; defaults run end-to-end without human input. Use when user says "auto pipeline", or wants the core stages chained without confirmation.

Sub-skill of /auto-verify. Given a claim, choose one method swap, one dataset swap, and one model swap that most strongly stress-test the claim. Harvests candidates from existing research; calls /research-lit only when coverage is thin. Use when user says "pick swaps for claim", "choose alternatives", or when invoked by /auto-verify.

Workflow 1.75: stress-test claims (regardless of main-experiment verdict) by swapping method, dataset, and model, then judging whether each variant agrees with the main experiment. Three stages with two integrity gates: Stage 1 audits the main experiment's eval method for every target claim; Stage 2 runs swap variants only on the top-K admitted claims picked by importance (K = `MAX_VERIFY_CLAIMS`, default 1); Stage 3 judges (binary pass/fail per variant), audits the variants, and computes a per-claim `robustness = #pass / N_eligible ∈ [0,1]` that decides PASS / FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS (threshold default 0.5 → at least half of eligible variants must pass; e.g. at N=3 this means ≥2 of 3). INCONCLUSIVE = main-experiment integrity broken (Phase 2 FAIL — variants never ran); ZERO_ELIGIBLE_VARIANTS = variants ran but all failed integrity at Phase 9; INTEGRITY_ONLY = Stage 1 audit passed but Stage 2 was intentionally skipped (either `SWAP_VARIANTS=false` global mode or per-claim `MAX_VERIFY_CLAI

The single place for every data constraint an experiment must satisfy — dataset provenance (existing → adapted → constructed), clear train / validation / test splits, labels that reflect the target behavior, and the minimum data amount. Use whenever an experiment chooses, adapts, or constructs a dataset, defines splits, or sets a sample size — for phenomenon validation (M0), mechanism exploration, intervention, or tuning. Domain-general: no assumption about model family, modality, or task.

Audit the experimental **methodology** integrity for a specific claim (Checks A–F: GT provenance, score normalization, result-file existence, dead code, scope, eval-type). Uses cross-model review (external LLM reviewer via llm-chat MCP). The output `overall_verdict` (PASS/WARN/FAIL) is THIS claim's verdict — i.e., whether the claim's experimental process is methodologically sound. Does NOT judge whether the numbers semantically support the claim's hypothesis — that is `/result-to-claim`'s job.

Turn a refined research proposal or method idea into a detailed, claim-driven experiment roadmap. Use after `research-refine`, or when the user asks for a detailed experiment plan, ablation matrix, evaluation protocol, run order, compute budget, or paper-ready validation that supports the core problem, novelty, simplicity, and any LLM / VLM / Diffusion / RL-based contribution.

SSH job queue for multi-seed / multi-config ML experiments with OOM-aware retry, stale-screen cleanup, wave-transition race prevention, and phase-dependency enforcement. Use when user says "batch experiments", "队列实验", "run grid", "multi-seed sweep", "auto-chain experiments", or when `/run-experiment` is insufficient for ≥10 jobs that need orchestration. `/auto-experiment` Phase 4 auto-routes here when a milestone declares ≥10 jobs or has `depends_on`.

Fine-tuning hyperparameter protocol — full FT, LoRA / QLoRA / DoRA / PEFT adapter, across SFT, DPO, and GRPO / PPO / RL objectives. Fires on any fine-tune, especially when the plan hard-codes a config or copies one from a reference paper. Enforces LR-first sweep, the method × objective LR-scale table (LoRA ≈ 10× full-FT; RL ≈ 10–100× smaller than SFT), and the under-fit diagnostic signals (flat loss, base-lookalike output, dead grad-norm). Triggers: `learning_rate`, `SFTTrainer`, `DPOTrainer`, `GRPOTrainer`, `LoraConfig`, `peft_config`, `full fine-tune`, `SFT`, `DPO`, `GRPO`, `RLHF`, `QLoRA`, `DoRA`, `PEFT`, `adapter`.

Canonical ImageNet eval preprocessing — square 256×256 resize → 224 center crop → ImageNet mean/std — for CV experiments probing ImageNet-pretrained backbones (ResNet, ViT, VGG, EfficientNet). Use this skill whenever a `torchvision.transforms` / `PIL` pipeline is being written, audited, or debugged for inference-time eval: feature extraction, activation hooks, mechanistic interpretability, top-k-activating-image retrieval, neuron labeling, or reproducibility checks. Apply it even when the user does not say "preprocessing" — triggers include `Resize`, `CenterCrop`, `T.Compose`, "ImageNet eval", "my activations changed between runs", or "results differ from the paper". Covers the `Resize(256)` (int, short-side) vs `Resize((256, 256))` (tuple, square) trap.

How to grade any multiple-choice / A-B / A-D letter task when the score is read out of the model's free-form output — any eval that maps a generation to a choice letter, whatever the domain. Use whenever the plan parses the letter with a regex like `[A-D]` / `re.search(r"[AB]", ...)`. Covers why naive letter-regex is fragile (a `"Using a…"` opener parses as "A"; `"At the garden…"` parses as "A"), the LLM-judge fix (three-way `{CORRECT, INCORRECT, OTHER}` verdict against the gold letter, no silent coercion), and the two biases every multiple-choice eval must control — position bias (swap A/B orientation) and token bias (`A` gets more prior mass than `B`). Triggers: "answer A or B", `[A-D]` regex on generations, any per-choice rate (e.g. `P(chosen)`) computed from a letter parse.

Routing entry point for experiment-protocol tips that prevent silent reproducibility / overclaim failures. Use when EXPERIMENT_PLAN.md is about to become runnable code and any of these is in scope: ImageNet / torchvision preprocessing, steering coefficient (α / dose / magnitude), steering block / layer / site selection, fine-tuning hyperparameters (full FT / LoRA / QLoRA / DoRA / PEFT — LR, capacity, target modules — across SFT / DPO / GRPO / PPO / RLHF objectives), or MCQ / A-B / A-D letter-parse evaluation of a free-form model generation. The **General Rule for mechanism/Interpretability** in this file is loaded on every mechanism / interpretability experiment (locate the target neuron/feature, then intervene without damaging general ability); the numbered tips load only as their symptom triggers apply, and after matching one you MUST load its own SKILL.md — never act on the preview. Universal data constraints (provenance, splits, sample-size floors) live in `skills/data-rule/`, not here.

How to choose where (and how many sites) to intervene for any operation on internal representations — activation patching, steering, CAA, DAS, SAE feature scaling, attribution patching. Use whenever the plan declares a target block / layer / site, especially when it hard-codes a single index (`target_block: 4`, `layer: 16`) without justification or copied from another paper. Covers picking the site by screening (gradient / attribution, or an activation signal) and by heuristic (mid-to-late layers usually steer best; widen to 3–5 layers if one is inert; sweep the stack at spaced intervals), and which component to target (attention / MLP / residual stream — circuit discovery studies attention heads + MLP). Prevents two symmetric failures: too few sites → downstream norm/attention denoise it → false "no effect"; too many → off-distribution collapse and the localization claim becomes untestable. Triggers include `target_block = 4`, `sliding window = 15`, `layer ∈ [12,18]`, `early blocks (0-7)`, "steering had no e

How to set the strength of any additive intervention on internal representations — steering vector, CAA, DAS dose-response, representation engineering, SAE feature scaling, ROME-style edits. Use whenever the plan sets a steering strength (`α`, `β`, `dose`, `magnitude`, `scale`, `coefficient`, `k`) to a fixed value or small range, especially when copied from a paper. Covers a coarse `layer × β` sweep (β in σ_proj units, scored on a target metric plus a fluency / general-ability metric), why the best coefficient is layer-dependent, the mid- vs late-layer behavior (late layers break into repetition / format-spam), and the rule to use the smallest sufficient β. Prevents two symmetric failures: TOO SMALL → effect drowned in noise → false "no causal effect"; TOO LARGE → off-distribution collapse → fluency breaks / random direction matches it → false "specificity fails". Triggers include `dose ∈ {-3..3}`, `α = 3`, "random direction beat my steering vector", "steering had no effect", "model output garbage after steer

Standalone BATCH generator of research hypotheses for ONE topic — enumerates many (behavior + mechanism) candidates into a TREE library (hypothesis_library.json), each behavior scored with an IMPACT check and each mechanism with a NOVELTY check, with LLM semantic dedup. It does NOT refine, review, pilot, converge to a single idea, or build an experiment plan — but it DOES score every behavior for IMPACT and every mechanism hypothesis for NOVELTY (separate scores only — no combined ranking). Can run MANY consecutive rounds in ONE invocation via `— rounds: R` (each round adds ~N_BEHAVIORS and persists to the library before the next; later rounds rebuild their banlist from the accumulated library, so earlier behaviors are never regenerated) — e.g. `— rounds: 5, n-behaviors: 10` aims for ~50 distinct behaviors. Its only selector is BEHAVIOR_SOURCE = discover (mine new behaviors + mechanisms) | given (behavior fixed, generate mechanisms only); there is no `mode` flag. Behavior axis is enumerated from /mechanism-be

Generate and rank research ideas given a broad direction. Use when user says "brainstorm ideas", "generate research ideas", "what can we work on", or wants to explore a research area for publishable directions.

Assess whether the research problem/behavior is important — its potential value and reach. Use when user says "impact check", "check impact", or wants to judge whether an idea/behavior matters before committing.

Paper retrieval via the cloud SEARCH service. The Agent builds a decomposed query JSON from its task context (preferred) or submits a polished free-form English query; the cloud service performs multi-ranker retrieval and fusion. Use as one of an important paper sources.

Audit the **mechanistic experiment rigor** for a specific claim. Catalogue currently has six slots A–F: A (steering coefficient sweep) is implemented; B–F are reserved for future checks (direction extraction quality, site/layer selection, n_effective sufficiency, probe-vs-causal disentanglement, intervention scope). Uses cross-model review (external LLM reviewer via llm-chat MCP). Complementary to `/experiment-audit` (which audits evaluation methodology, not mechanism tuning). The output `overall_verdict` (PASS/WARN/FAIL/N/A) is THIS claim's mechanism-rigor verdict — i.e., whether the interpretability mechanism backing this claim was tuned with the necessary controls. Returns N/A when the claim's experiment does not use any mechanism intervention (e.g., pure dataset evaluation). Does NOT judge methodology honesty (that is `/experiment-audit`'s job) or semantic claim support (that is `/result-to-claim`'s job).

Mine behavioral regularities in neural-network (LLM / multimodal) models — the upstream half of the project's mission (find a behavior worth explaining, then investigate the mechanism behind it). Use this skill when the task is open-ended: surface a *new* behavioral phenomenon — a candidate claim / research direction — rather than investigate an already-named mechanism. It gives strategies for choosing which behavior to probe and how to choose the data that validates it. The output is a candidate phenomenon that hands off to `/mechanism-explore` for mechanistic investigation. Domain-general: no assumption about model family, modality, or task.

Macro-level strategic directions for investigating the *mechanism* behind a model behavior — the downstream half of the project's mission (mine LLM behaviors, then explain the mechanism behind them). Use once a phenomenon is observed in a model — whether already established/known or freshly mined by `/mechanism-behavior-discovery` — and the task is to choose *how* to investigate the internal cause. It is the strategy layer above the concrete method families in `/mechanism-skills`, organized around six parallel research directions — **Location**, **Causal Intervention**, **Tuning & Editing**, **Formation Tracing**, **Unit Interpretation**, **Decision Auditing** — plus how to combine them into strategies. Domain-general: it prescribes strategy, not any single model, modality, or method.

Analyze and dissect factual recall in auto-regressive language models using attention knockout, hidden state analysis, and intervention techniques on GPT-2 and GPT-J models

Use when analyzing neural network circuits, performing attribution patching, automated circuit discovery, or investigating model interpretability through edge attribution methods in transformer models

Use this skill when you need to edit factual knowledge in large language models like GPT-2 or GPT-J, perform causal tracing to understand model behavior, or implement Rank-One Model Editing (ROME) to modify specific factual associations without retraining

Causal Attribution methods constitute the gold standard for localization in Mechanism Interpretability. Unlike correlation-based analyses, these techniques identify which internal objects are causally responsible for a specific model behavior by systematically measuring the effect of controlled interventions

Use this skill to discover and analyze computational circuits within autoregressive transformer language models via attribution methods like Edge Attribution Patching with Integrated Gradients (EAP-IG), enabling mechanistic interpretability studies and circuit evaluation.

Activates when working on mechanistic interpretability of transformers, specifically for automated circuit discovery in transformer models using computational graph editing and analysis.

Circuit discovery methods localize the minimal subgraph of a transformer's computational graph (a set of nodes such as attention heads and MLPs, together with the edges that connect them) that is causally responsible for a specific model behavior. Rather than scoring individual objects in isolation, these techniques recover an end-to-end mechanism by jointly identifying which components, and which information pathways between them, must be preserved to reproduce the behavior on a task distribution.

Use this skill when working with sparse autoencoders (SAEs), crosscoders, dictionary learning on neural network activations, training SAEs/crosscoders from scratch, loading pretrained dictionaries, caching model activations, or comparing model internals across fine-tuned model pairs using the dictionary_learning / crosscoder_learning library.

Use this skill when applying Independent Component Analysis as a training-free interpretability lens — decomposing a target activation site (residual stream, MLP output, attention-head output, or any cached hook point) into maximally non-Gaussian directions and treating each direction as a candidate monosemantic component for sparse-probing, targeted perturbation, annotation, and cross-comparison against trained SAE / transcoder / crosscoder features.

Use this skill when working with Sparse Autoencoders (SAEs) for mechanistic interpretability of language models, including training SAEs, loading pre-trained SAEs, analyzing neural network features, or integrating SAEs with TransformerLens, HuggingFace Transformers, or other PyTorch-based models.

Feature Dictionary Learning methods address the polysemanticity of neuron-level units by decomposing a dense internal activation (e.g. a residual-stream state or MLP output) into a sparse weighted sum of directions drawn from a large over-complete dictionary. The dictionary contains far more directions than the activation's original dimensionality, and the decomposition is constrained to use only a small number of them at once. Each direction in the dictionary plays the role of an interpretable "feature": its weight measures how strongly that feature is present, turning a black-box vector into a small set of human-readable components.