| name | mech-interp-runner |
| description | Run, plan, validate, or aggregate Epistemic-Humility local mechanistic-interpretability sweeps, including hidden-state candidate inventories, causal-pilot sweep planning, explicit non-GPU/GPU gates, base-original skip handling, and offline result aggregation. Use when working on local mech-interp sweeps, causal-pilot diagnostics, activation-addition/logit-diagnostic runs, behavior-axis scans, SAE feature screens, or future reruns of the legacy full candidate inventory. |
| allowed-tools | Read, Bash, Write, Grep, Glob |
Mech-Interp Runner
This skill drives the frozen bespoke local mech-interp machinery (see
archive/experiment/phase1/probe/steering/LEGACY.md). It mints NO new cells. To author
a NEW steering / extraction / probe-fit / gate-scoring cell, use the
mechinterp-cells skill (tuner-backed mechinterp verbs), not this one.
Use the checked-in scripts and configs to plan, gate, run, and aggregate local
mech-interp work. Do not hand-roll terminal loops. This SKILL.md is a
progressive-disclosure router: it carries the invariants and routing only.
Procedural detail lives in references/, and current results live in
docs/sessions/ and experiment-local docs under experiments/<slug>/ - never
inline run history here.
Start Here
Always choose the narrowest reference needed for the task:
Do not preload all references. Read only the file needed for the current
operation, then follow any further routing inside it.
Quick Commands
Prefer the skill CLI for common local analyses instead of hand-writing long
PowerShell commands. It delegates to checked-in repo scripts, resolves paths
from the repo root, uses sys.executable, and forces UTF-8 subprocess output
(PYTHONUTF8=1, PYTHONIOENCODING=utf-8, PYTHONUNBUFFERED=1) to avoid
Windows console encoding failures.
| Task | Command |
|---|
| Discover subcommands | python .skills/mech-interp-runner/scripts/mechinterp_cli.py --help |
| Quick non-GPU validation | python .skills/mech-interp-runner/scripts/mechinterp_cli.py validate --quick |
| Behavior-axis scan | python .skills/mech-interp-runner/scripts/mechinterp_cli.py behavior-axis-scan --config <cfg> |
| Causal sweep (plan/materialize) | python .skills/mech-interp-runner/scripts/mechinterp_cli.py causal-sweep --config <cfg> --mode-filter logit_diagnostic --write-plan --materialize-configs |
| Sycophancy generation analysis | python .skills/mech-interp-runner/scripts/mechinterp_cli.py sycophancy-generation-analysis --generations <jsonl> --output-root <dir> |
| Cross-dataset panel (step 1) | python .skills/mech-interp-runner/scripts/mechinterp_cli.py xdataset-build-panel --source <jsonl> --dataset <id> --out-dir <dir> --n-known 600 --n-unknown 400 |
| Cross-dataset behavior rows (step 3) | python .skills/mech-interp-runner/scripts/mechinterp_cli.py xdataset-behavior --generation <rows.jsonl> --panel-rows <gen_rows.jsonl> --out-dir <dir> |
| Residual caution direction (read-trajectory step 1) | python .skills/mech-interp-runner/scripts/mechinterp_cli.py residual-caution-direction --extraction-dir <ext> --behavior-rows <rows.jsonl> --layer 35 --out <dir>/caution_direction_L35.json |
| Read-trajectory re-analysis (GPU-free) | python .skills/mech-interp-runner/scripts/mechinterp_cli.py residual-read-trajectory-analysis --rows <rows.jsonl> --out <analysis.json> |
Use --dry-run on any subcommand to print the delegated command before running
it. For live Docker/GPU execution the same approval rule applies: do not pass
--execute unless the user has approved that live run.
Core Invariants
- The research target is coherent epistemic-humility expression, not a raw
refusal axis. A candidate is promising only if it improves one damaged
behavior without degrading the paired desired behavior. See
references/interpretation-invariants.md.
- Generated-answer replay is the behavioral gate. Offline separability (AUC, d,
readout recall) and first-token answer-start movement are screening evidence,
never a steering claim.
- Treat outputs as Tier 2 exploratory local mechanism evidence.
- Do not edit
synaptic-tuner/.
- Do not run Docker/GPU unless the user explicitly approves that live run.
- Keep base-original
h_base adapterless work fail-closed until the live runner
explicitly supports adapterless base execution. In DPO/KTO extractions
h_base is the SFT-merged pre-adapter model, not original Qwen base.
- Keep generated outputs gitignored by default unless a governed publication
decision explicitly whitelists them.
- Legacy reruns keep their registered generation and capture backend. The
project-wide vLLM preference applies to new surfaces after the required
smokes and bridges; it never authorizes an engine swap inside a frozen cell.
- Stay local to this repository for mech-interp work. Do not use external
workflow or memory systems unless the user explicitly asks in the current turn.
Validation
Run focused non-GPU checks after runner/config/skill edits:
python -m pytest experiments/common/knowledge_probe/tests/test_mechinterp_causal_pilot_sweep.py \
experiments/common/knowledge_probe/tests/test_mechinterp_causal_pilot_runner.py \
experiments/common/knowledge_probe/tests/test_mechinterp_causal_pilot_dry_run.py -q
python -m pytest experiments/common/knowledge_probe/tests/test_mechinterp_behavior_axis_scan.py \
experiments/common/knowledge_probe/tests/test_mechinterp_behavior_axis_directions.py \
experiments/common/knowledge_probe/tests/test_mechinterp_calibrated_expression_plane.py \
experiments/common/knowledge_probe/tests/test_mechinterp_logit_cell_analysis.py -q
python -m pytest experiments/common/knowledge_probe/tests/test_mechinterp_sae_smoke.py \
experiments/common/knowledge_probe/tests/test_mechinterp_sae_train.py \
experiments/common/knowledge_probe/tests/test_mechinterp_sae_feature_analysis.py \
experiments/common/knowledge_probe/tests/test_mechinterp_sae_feature_directions.py \
experiments/common/knowledge_probe/tests/test_mechinterp_sae_behavior_feature_analysis.py -q
python -m pytest .skills/mech-interp-runner/tests/test_mechinterp_cli.py -q
python bin/sync_skills.py --check
Compact Recovery Loop
After context compaction or degraded memory:
- Re-read this skill; drill into
references/ only as needed.
- Re-read the latest relevant
docs/sessions/ note and local KG/search
results if they affect the next experiment.
- Set up or run the current local experiment with checked-in scripts/configs.
- Analyze outputs against the research target (interpretation-invariants).
- Update session notes, experiment-local runbooks/plans, and this skill when a
durable procedure or gotcha changes.
- Choose the next highest-ROI local slice from the evidence and repeat.
Skill Maintenance
Edit the canonical tree under .skills/mech-interp-runner/ only; .agents/ and
.claude/ are generated mirrors. Keep this contract intact:
- Put timeless rules, routing, commands, and validation gates in
SKILL.md.
- Put durable procedure and gotchas in one-level
references/*.md files and
link them from Start Here.
- Put current findings, numeric results, and interpretation snapshots in
docs/sessions/*.md and experiment-local docs under experiments/<slug>/ -
not in the skill.
- Put reusable analysis logic in checked-in scripts/configs, not prose.
- If a section grows because of one experiment, move the details to a session or
experiment note and leave only the general rule plus the reference path.
Before adding a paragraph here, ask: "Will this still guide a future agent six
months from now without loading today's run history?" If not, it belongs in a
reference or a note. After canonical edits, run:
python bin/sync_skills.py --write --skill mech-interp-runner
python bin/sync_skills.py --check --skill mech-interp-runner
