Use this skill when you need to automatically describe or interpret the functionality of individual neurons in deep neural networks (DNNs) using CLIP-based semantic analysis, perform mechanistic interpretability research on vision models, dissect convolutional or transformer-based image classifiers, identify what visual concepts activate specific neurons, or compare neuron descriptions across different probing datasets and concept sets.
Instrucciones de origen · Vista previa de solo lectura
name
clip-dissect
description
Use this skill when you need to automatically describe or interpret the functionality of individual neurons in deep neural networks (DNNs) using CLIP-based semantic analysis, perform mechanistic interpretability research on vision models, dissect convolutional or transformer-based image classifiers, identify what visual concepts activate specific neurons, or compare neuron descriptions across different probing datasets and concept sets.
CLIP-Dissect Skill
When to Use
Activate this skill when:
You need to automatically describe what individual neurons in a deep vision network respond to
You are performing mechanistic interpretability or explainable AI research on CNNs or Vision Transformers
You want to understand neuron-level representations in models like ResNet-50, ResNet-18, ViT, or custom models
You need to compare neuron descriptions against baselines like NetDissect or MILAN
You want to probe neural network layers using a concept set (e.g., 3k, 10k, 20k English words)
You are working with Broden or ImageNet as a probing dataset
You need to evaluate how well neuron descriptions predict class-level behavior in a model
Automatic Neuron Description: Assigns human-readable concept labels to individual neurons in any layer of a DNN using CLIP's vision-language embeddings.
Efficient Similarity Computation: Uses cosine similarity between neuron activation patterns and CLIP text embeddings of candidate concepts.
Multi-Layer Dissection: Dissect multiple layers of a target model in a single run (e.g., layer1, layer2, layer3, layer4, fc for ResNet-50).
Flexible Concept Sets: Bundled concept sets with 3k, 10k, and 20k English words; supports custom .txt concept files.
Activation Caching: Automatically caches computed activations in saved_activations/ to avoid recomputation on repeated runs.
Model Agnostic: Works with any PyTorch model — ResNet, ViT, custom architectures — by implementing a simple loader function.
Experiment Notebooks: Reproduces all paper figures and tables via Jupyter notebooks in experiments/.
Comparison Baselines: Includes pre-computed results from NetDissect and MILAN for direct comparison.
Device Flexibility: Runs on CUDA GPU or CPU via --device argument.
Usage Examples
Quickstart — Dissect ResNet-50 (ImageNet) with Broden
Dissects 5 layers of ResNet-50 pretrained on ImageNet using Broden as the probing dataset. Results saved in results/resnet50_{datetime}/descriptions.csv.
python describe_neurons.py
Dissect a Custom Model
Implement your model loader in data_utils.py under get_target_model:
Broden dataset — diverse visual concepts (downloaded via dlbroden.sh)
imagenet_val
ImageNet validation set (user must provide path)
Custom
User-defined torchvision Dataset
Concept Sets (bundled in data/)
File
Size
Source
data/3k.txt
3,000 words
EF English vocabulary
data/10k.txt
10,000 words
Google 10k English
data/20k.txt
20,000 words
Google 20k English
Core Functions
describe_neurons.py (main entry point)
Demo Scripts
scripts/run_clip_dissect.py
#!/usr/bin/env python3"""
CLIP-Dissect: Automated Neuron Description for Deep Vision Networks
This script demonstrates how to use the CLIP-Dissect pipeline to:
1. Load a pretrained target model (ResNet-50)
2. Load a probing dataset (Broden)
3. Compute neuron activations and CLIP text embeddings
4. Compute cosine similarity between neuron activations and concept embeddings
5. Save per-neuron descriptions to a CSV file
Requirements:
- Clone https://github.com/Trustworthy-ML-Lab/CLIP-dissect
- pip install -r requirements.txt
- bash dlbroden.sh (to download Broden dataset)
- Must be run from within the CLIP-dissect repository root directory
Usage:
cd /path/to/CLIP-dissect
python scripts/run_clip_dissect.py
"""import os
import sys
import csv
import datetime
import argparse
import torch
import torch.nn.functional as F
from torch.utils.data import DataLoader
from torchvision import transforms, models
# ── Adjust import path so we can import from the repo root ───────────────────
REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, REPO_ROOT)
try:
import clip # bundled CLIP from clip/from data_utils import get_target_model, get_data, get_resnet_imagenet_preprocess
except ImportError as e:
print(f"[ERROR] Could not import CLIP-Dissect modules: {e}")
print("Make sure you are running this script from the CLIP-dissect repository root,")
print("or that REPO_ROOT is set correctly.")
sys.exit(1)
# ── Constants ────────────────────────────────────────────────────────────────
DEFAULT_TARGET_MODEL = "resnet50"
DEFAULT_PROBE_DATASET =
DEFAULT_CONCEPT_SET = os.path.join(REPO_ROOT, , )
DEFAULT_SAVE_DIR = os.path.join(REPO_ROOT, )
DEFAULT_DEVICE = torch.cuda.is_available()
DEFAULT_BATCH_SIZE =
CLIP_MODEL_NAME =
() -> []:
os.path.exists(concept_set_path):
FileNotFoundError(
)
(concept_set_path, , encoding=) f:
concepts = [line.strip().lower() line f line.strip()]
()
concepts
() -> torch.Tensor:
clip_model.()
all_embeddings = []
()
torch.no_grad():
start (, (concepts), batch_size):
batch = concepts[start : start + batch_size]
tokens = clip.tokenize(batch, truncate=).to(device)
text_features = clip_model.encode_text(tokens)
text_features = F.normalize(text_features, dim=-)
all_embeddings.append(text_features.cpu())
(start // batch_size) % == :
()
embeddings = torch.cat(all_embeddings, dim=)
()
embeddings
() -> torch.Tensor:
target_layer =
name, module model.named_modules():
name == layer_name:
target_layer = module
target_layer :
available = [n n, _ model.named_modules() n]
ValueError(
)
activation_buffer = []
():
output.dim() == :
pool_mode == :
pooled = output.mean(dim=(, ))
:
pooled = output.amax(dim=(, ))
:
pooled = output
activation_buffer.append(pooled.detach().cpu())
hook = target_layer.register_forward_hook(hook_fn)
model.()
()
:
torch.no_grad():
batch_idx, (images, _) (dataloader):
images = images.to(device)
_ = model(images)
batch_idx % == :
()
:
hook.remove()
activations = torch.cat(activation_buffer, dim=)
()
activations
() -> torch.Tensor:
clip_model.()
()
image_embeddings_list = []
torch.no_grad():
batch_idx, (images, _) (dataloader):
images = images.to(device)
img_feats = clip_model.encode_image(images)
img_feats = F.normalize(img_feats, dim=-)
image_embeddings_list.append(img_feats.cpu())
batch_idx % == :
()
image_embeddings = torch.cat(image_embeddings_list, dim=)
()
activations_norm = F.relu(activations)
act_sum = activations_norm.(dim=, keepdim=) +
activations_normalized = activations_norm / act_sum
neuron_embeddings = activations_normalized.T @ image_embeddings
neuron_embeddings = F.normalize(neuron_embeddings, dim=-)
()
similarity = neuron_embeddings @ clip_text_embeddings.T
()
similarity
() -> :
os.makedirs(os.path.dirname(save_path), exist_ok=)
best_indices = similarity.argmax(dim=)
best_scores = similarity.(dim=).values
()
(save_path, , newline=, encoding=) csvfile:
writer = csv.writer(csvfile)
writer.writerow([, , , ])
unit_idx ((best_indices)):
concept_idx = best_indices[unit_idx].item()
score = best_scores[unit_idx].item()
writer.writerow([
layer_name,
unit_idx,
concepts[concept_idx],
,
])
()
() -> :
()
()
()
activations = compute_neuron_activations(
model=target_model,
layer_name=layer_name,
dataloader=dataloader,
device=device,
pool_mode=pool_mode,
)
similarity = compute_neuron_clip_similarity(
activations=activations,
clip_text_embeddings=clip_text_embeddings,
clip_model=clip_model,
dataloader=dataloader,
device=device,
)
save_descriptions_csv(
similarity=similarity,
concepts=concepts,
layer_name=layer_name,
save_path=save_path,
)
() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description=
)
parser.add_argument(
,
=,
default=DEFAULT_TARGET_MODEL,
=,
)
parser.add_argument(
,
=,
default=DEFAULT_PROBE_DATASET,
=,
)
parser.add_argument(
,
=,
default=DEFAULT_CONCEPT_SET,
=,
)
parser.add_argument(
,
=,
default=DEFAULT_BATCH_SIZE,
=,
)
parser.add_argument(
,
=,
default=DEFAULT_DEVICE,
=,
)
parser.add_argument(
,
=,
choices=[, ],
default=,
=,
)
parser.add_argument(
"broden"
"data"
"20k.txt"
"results"
"cuda"
if
else
"cpu"
64
"ViT-B/32"
def
load_concept_set
concept_set_path: str
list
str
"""
Load a list of concept strings from a plain-text file.
Each line in the file should contain exactly one concept (word or phrase).
Args:
concept_set_path: Path to the .txt concept set file.
Returns:
A list of concept strings (lowercased, whitespace-stripped).
Raises:
FileNotFoundError: If the concept set file does not exist.
"""
if
not
raise
f"Concept set file not found: {concept_set_path}\n"
"Make sure to run from the CLIP-dissect repository root "
"or provide the correct path."
with
open
"r"
"utf-8"
as
for
in
if
print
f"[INFO] Loaded {len(concepts)} concepts from '{concept_set_path}'"
return
def
compute_clip_text_embeddings
concepts: list[str],
clip_model,
device: str,
batch_size: int = 256,
"""
Compute normalized CLIP text embeddings for a list of concept strings.
Uses the bundled CLIP tokenizer and text encoder. Embeddings are computed
in batches to avoid memory overflow for large concept sets.
Args:
concepts: List of concept strings.
clip_model: A loaded CLIP model (from clip.load()).
device: Torch device string, e.g. 'cuda' or 'cpu'.
batch_size: Number of concepts to encode per batch.
Returns:
Tensor of shape (num_concepts, embedding_dim) — L2-normalized.
"""
eval
print
f"[INFO] Computing CLIP text embeddings for {len(concepts)} concepts ..."
"""
Compute pooled activations of a specific layer for all images in a dataloader.
Uses PyTorch forward hooks to capture intermediate layer outputs.
Spatial dimensions are pooled (avg or max) to produce a single scalar
per neuron per image.
Args:
model: PyTorch model in eval mode.
layer_name: Dot-separated layer name accessible via model.named_modules(),
e.g. 'layer4' or 'layer4.1.conv2'.
dataloader: DataLoader yielding (image_tensor, label) batches.
device: Torch device string.
pool_mode: 'avg' for average pooling or 'max' for max pooling over spatial dims.
Returns:
Tensor of shape (num_images, num_neurons) — pooled activations.
Raises:
ValueError: If the layer_name is not found in the model.
"""
# Locate the target layer by name
None
for
in
if
break
if
is
None
for
in
if
raise
f"Layer '{layer_name}' not found in model.\n"
f"Available layers: {available[:20]} ..."
# Register a forward hook to capture activations
def
hook_fn
module, input, output
# output shape: (batch, channels, H, W) for conv layers
# or (batch, features) for linear layers
if
4
# Spatial pooling
if
"avg"
2
3
else
2
3
else
eval
print
f"[INFO] Collecting activations from layer '{layer_name}' ..."
"""
Compute the similarity between each neuron and each concept.
For each neuron, we compute the weighted average of CLIP image embeddings,
weighted by the neuron's activation for each image. The resulting vector
is then compared (cosine similarity) against all concept text embeddings.
Args:
activations: Tensor (N_images, N_neurons) of pooled neuron activations.
clip_text_embeddings: Tensor (N_concepts, D) of normalized CLIP text embeddings.
clip_model: Loaded CLIP model.
dataloader: DataLoader over probing images (same order as activations).
device: Torch device string.
batch_size: Batch size for CLIP image encoding.
Returns:
Tensor of shape (N_neurons, N_concepts) — cosine similarity scores.
"""
eval
# 1. Compute CLIP image embeddings for all probe images
print
"[INFO] Computing CLIP image embeddings for all probe images ..."
"""
Save the top-1 neuron descriptions (best-matching concept) to a CSV file.
For each neuron, the concept with the highest cosine similarity score is
selected as the neuron's description.
Args:
similarity: Tensor (N_neurons, N_concepts) of cosine similarity scores.
concepts: List of concept strings (length N_concepts).
layer_name: Name of the dissected layer (written to the CSV).
save_path: Full path where the CSV file will be written.
Output CSV columns:
- layer: Layer name
- unit: Neuron index (0-based)
- description: Best-matching concept string
- similarity: Cosine similarity score (float)
"""
True
1
# (N_neurons,)
max
1
# (N_neurons,)
print
f"[INFO] Saving descriptions to '{save_path}' ..."
"""
Full CLIP-Dissect pipeline for a single model layer.
Steps:
1. Compute pooled neuron activations for all probe images.
2. Compute CLIP image embeddings and activation-weighted concept similarities.
3. Save per-neuron descriptions to CSV.
Args:
target_model: PyTorch model (eval mode).
clip_model: Loaded CLIP model.
layer_name: Name of the layer to dissect.
dataloader: DataLoader over probe images.
concepts: List of concept strings.
clip_text_embeddings: Precomputed normalized CLIP text embeddings (N_concepts, D).
device: Torch device string.
save_path: Path for the output CSV file.
batch_size: Batch size (unused here, controlled by dataloader).
pool_mode: Spatial pooling mode ('avg' or 'max').
"""
print
f"\n{'='*60}"
print
f" Dissecting layer: {layer_name}"
print
f"{'='*60}"
# Step 1: Neuron activations
# Step 2: Compute similarity
# Step 3: Save descriptions
def
parse_args
"""Parse command-line arguments for the CLIP-Dissect demo script."""
"CLIP-Dissect: Describe neuron functionalities using CLIP."
"--target_model"
type
str
help
"Name of the target model to dissect (default: resnet50)."
"--d_probe"
type
str
help
"Probing dataset name: 'broden' or 'imagenet_val' (default: broden)."
"--concept_set"
type
str
help
"Path to concept set .txt file (default: data/20k.txt)."
