메뉴
Causal intervention via activation patching to identify important model components. Use when determining which layers, heads, or positions are causally responsible for model behavior.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | activation-patching |
| description | Causal intervention via activation patching to identify important model components. Use when determining which layers, heads, or positions are causally responsible for model behavior. |
Activation patching is a causal intervention technique that identifies which model components are responsible for specific behaviors by swapping activations between different inputs.
If patching a component restores the clean behavior, that component is causally important.
from nnsight import LanguageModel
import torch
model = LanguageModel("openai-community/gpt2", device_map="auto", dispatch=True)
# Indirect Object Identification (IOI) task
clean_prompt = "After John and Mary went to the store, Mary gave a bottle of milk to"
corrupted_prompt = "After John and Mary went to the store, John gave a bottle of milk to"
# Target tokens
correct_token = model.tokenizer(" John")["input_ids"][0] # Clean answer
incorrect_token = model.tokenizer(" Mary")["input_ids"][0] # Corrupted answer
def logit_diff(logits, correct_idx, incorrect_idx):
"""Measure how much model prefers correct over incorrect token."""
return (logits[0, -1, correct_idx] - logits[0, -1, incorrect_idx]).item()
n_layers = len(model.transformer.h)
results = torch.zeros(n_layers)
# Run 1: Clean - save activations
with model.trace(clean_prompt):
clean_hiddens = [layer.output[0].save() for layer in model.transformer.h]
clean_logits = model.lm_head.output.save()
# Run 2: Corrupted baseline
with model.trace(corrupted_prompt):
corrupted_logits = model.lm_head.output.save()
# Runs 3+: Patch each layer (separate forward passes)
for layer_idx in range(n_layers):
with model.trace(corrupted_prompt):
# Replace corrupted activation with clean
model.transformer.h[layer_idx].output[0][:] = clean_hiddens[layer_idx]
patched_logits = model.lm_head.output.save()
results[layer_idx] = logit_diff(patched_logits.value, correct_token, incorrect_token)
# Normalize results
clean_diff = logit_diff(clean_logits.value, correct_token, incorrect_token)
corrupted_diff = logit_diff(corrupted_logits.value, correct_token, incorrect_token)
normalized = (results - corrupted_diff) / (clean_diff - corrupted_diff)
Patch only specific token positions:
seq_len = len(model.tokenizer.encode(clean_prompt))
results = torch.zeros(n_layers, seq_len)
# Clean run - save activations
with model.trace(clean_prompt):
clean_hiddens = [layer.output[0].save() for layer in model.transformer.h]
# Patch each layer x position (separate forward passes)
for layer_idx in range(n_layers):
for pos_idx in range(seq_len):
with model.trace(corrupted_prompt):
# Patch only this position
model.transformer.h[layer_idx].output[0][:, pos_idx, :] = \
clean_hiddens[layer_idx][:, pos_idx, :]
patched_logits = model.lm_head.output.save()
results[layer_idx, pos_idx] = logit_diff(
patched_logits.value, correct_token, incorrect_token
)
Patch individual attention heads:
n_heads = model.config.n_head
head_dim = model.config.n_embd // n_heads
results = torch.zeros(n_layers, n_heads)
# Clean run - save attention outputs (before projection)
with model.trace(clean_prompt):
clean_attn = [layer.attn.c_proj.input[0][0].save()
for layer in model.transformer.h]
# Patch each layer x head (separate forward passes)
for layer_idx in range(n_layers):
for head_idx in range(n_heads):
with model.trace(corrupted_prompt):
# Patch single head's output
start = head_idx * head_dim
end = (head_idx + 1) * head_dim
model.transformer.h[layer_idx].attn.c_proj.input[0][0][:, :, start:end] = \
clean_attn[layer_idx][:, :, start:end]
patched_logits = model.lm_head.output.save()
results[layer_idx, head_idx] = logit_diff(
patched_logits.value, correct_token, incorrect_token
)
Instead of restoring clean activations, corrupt clean activations:
# Corrupted run - save activations
with model.trace(corrupted_prompt):
corrupted_hiddens = [layer.output[0].save() for layer in model.transformer.h]
# For each layer, inject corrupted activation into clean run
noising_results = torch.zeros(n_layers)
for layer_idx in range(n_layers):
with model.trace(clean_prompt):
# Inject corrupted activation into clean run
model.transformer.h[layer_idx].output[0][:] = corrupted_hiddens[layer_idx]
noised_logits = model.lm_head.output.save()
noising_results[layer_idx] = logit_diff(noised_logits.value, correct_token, incorrect_token)
import matplotlib.pyplot as plt
import seaborn as sns
plt.figure(figsize=(12, 8))
sns.heatmap(
results.numpy(),
xticklabels=[f"Pos {i}" for i in range(seq_len)],
yticklabels=[f"Layer {i}" for i in range(n_layers)],
cmap="RdBu_r",
center=0,
annot=True,
fmt=".2f"
)
plt.title("Activation Patching Results")
plt.xlabel("Token Position")
plt.ylabel("Layer")
plt.tight_layout()
plt.show()
Core nnsight concepts for neural network interpretability. Use when setting up models, tracing activations, saving values, or making basic interventions on model internals.
Optimize nnsight code for remote execution on NDIF. Use whenever the user is writing, reviewing, or refactoring nnsight code that runs against NDIF (any usage of `remote=True`, `model.session`, or models too large to run locally like Llama-70B/405B), even if they don't ask for "optimization" explicitly — naive remote code is often 100x slower than it needs to be, and almost all of that comes from two mistakes: too many requests and too much downloaded.
Gradient-based approximation to activation patching for scalable circuit analysis. Use when activation patching is too slow or when analyzing many components simultaneously.
Causal mediation analysis to identify which model components mediate specific behaviors. Use when investigating how information flows through the network and which neurons or layers are causally responsible for outputs.
Decode intermediate layer predictions using the Logit Lens technique. Use when analyzing what a model predicts at each layer, understanding information flow, or visualizing layer-wise processing.
Control model behavior through persistent edits and steering interventions. Use when modifying model outputs, applying steering vectors, or creating persistently modified model versions.