Use this skill when working with convolutional neural feature ansatz (CNFA), deep convolutional recursive feature machines (Deep ConvRFM), or tasks involving kernel-based feature learning with convolutional architectures, VGG networks, patch-based Jacobian computation, and EGOP (expected gradient outer product) analysis on image datasets such as ImageNet and CIFAR.
Use this skill when working with convolutional neural feature ansatz (CNFA), deep convolutional recursive feature machines (Deep ConvRFM), or tasks involving kernel-based feature learning with convolutional architectures, VGG networks, patch-based Jacobian computation, and EGOP (expected gradient outer product) analysis on image datasets such as ImageNet and CIFAR.
Convrfm skill
When to use
Activate this skill when:
Implementing or experimenting with the convolutional neural feature ansatz (CNFA)
Training or evaluating deep convolutional recursive feature machines (Deep ConvRFM)
Computing expected gradient outer products (EGOP) for patch-based convolutional layers
Verifying neural feature ansatz properties in pretrained or VGG-style networks
Extracting feature embeddings from convolutional neural networks for kernel analysis
Running binary or multiclass classification experiments on image datasets using RFM-based methods
Generating toy datasets for testing convolutional feature learning pipelines
Analyzing hyperparameter sensitivity of CNFA-based models
Visualizing VGG kernel eigenvectors or passing eigenvectors through network layers
Keywords that trigger this skill: CNFA, convolutional neural feature ansatz, ConvRFM, deep RFM, recursive feature machine, EGOP, patch Jacobian, VGG features, kernel learning, convolutional kernel, imagenet features, patchify, conv embedding.
Description: Code for convolutional neural feature ansatz and deep ConvRFM
Related paper concepts: Neural Feature Ansatz (NFA), Recursive Feature Machines (RFM), Expected Gradient Outer Product (EGOP), convolutional patch kernels
No official documentation site or demo URL is provided by the repository
Installation/setup
Prerequisites
Python 3.7+
PyTorch (with CUDA support recommended for large-scale experiments)
torchvision
NumPy
SciPy (for kernel/eigenvalue computations)
h5py (for dataset loading)
Install dependencies
pip install torch torchvision numpy scipy h5py
Clone the repository
git clone https://github.com/aradha/convrfm.git
cd convrfm
Dataset setup
ImageNet: Provide a local path to the ImageNet dataset directory. The loader expects the standard ImageNet folder structure.
CIFAR / toy data: Use the provided gen_toy_data.py script to generate synthetic datasets for quick experimentation.
python gen_toy_data.py
Core features
Convolutional neural feature ansatz (CNFA) verification: Verify that the EGOP of a trained convolutional network matches its learned kernel features, using pretrained and VGG-based networks (cnfa_verification/).
Deep ConvRFM training: Iteratively train convolutional recursive feature machines that refine feature representations through gradient-based kernel updates (deep_conv_rfm/).
Convolutional network training: Train standard and binary classification convolutional networks on image datasets with utilities for embedding extraction and model evaluation (conv_nets/).
Patch-based Jacobian computation: Decompose convolutional network outputs into patch-level Jacobians, enabling per-patch feature analysis and EGOP estimation (cnfa_verification/pretrained_conv_nfa.py, cnfa_verification/vgg_conv_nfa.py).
EGOP computation: Compute the expected gradient outer product over a dataset to derive the effective feature matrix for kernel comparisons.
VGG kernel visualization: Compute and visualize eigenvectors of VGG-derived kernels, and pass eigenvectors through network layers for interpretability (vgg_vis/).
Hyperparameter CNFA verification: Study the effect of hyperparameters on CNFA properties, including correlation analysis across training regimes (hyperparam_cnfa_verification/).
Binary classification support: Specialized pipelines for binary classification tasks with filter extraction utilities (conv_nets/binary_main.py, deep_conv_rfm/binary_main.py).
Flexible dataset utilities: Custom dataset classes for ImageNet and other image datasets with configurable batch sizes and preprocessing (cnfa_verification/dataset.py, cnfa_verification/loader.py).
Toy data generation: Generate synthetic datasets to test and validate pipeline components without requiring large real-world datasets (gen_toy_data.py).
Usage examples
Note: The repository README does not contain inline code examples. The following examples are derived directly from analysis of the repository source files.
Generate toy data
python gen_toy_data.py
Train a convolutional network (multiclass)
cd conv_nets
python main.py
Train a convolutional network (binary classification)
cd conv_nets
python binary_main.py
Run deep ConvRFM (multiclass)
cd deep_conv_rfm
python main.py
Run deep ConvRFM (binary classification)
cd deep_conv_rfm
python binary_main.py
Run CNFA verification with a pretrained network
cd cnfa_verification
python pretrained_conv_nfa.py
Run CNFA verification with VGG
cd cnfa_verification
python vgg_conv_nfa.py
Run VGG kernel visualization
cd vgg_vis
python main.py
Run hyperparameter CNFA verification
cd hyperparam_cnfa_verification
python main.py
Key APIs/models
Classes
Class
Module
Description
PatchConvLayer
cnfa_verification/pretrained_conv_nfa.py
Wraps a convolutional layer to operate on patchified inputs for Jacobian/EGOP computation
PatchBasicBlock
cnfa_verification/pretrained_conv_nfa.py
Patch-based wrapper for ResNet BasicBlock layers
PatchBottleneck
cnfa_verification/pretrained_conv_nfa.py
Patch-based wrapper for ResNet Bottleneck layers
PatchConvLayer
cnfa_verification/vgg_conv_nfa.py
Patch-based convolutional layer wrapper for VGG-style networks
ImageNet
cnfa_verification/loader.py
Custom PyTorch Dataset class for loading ImageNet with transforms
MyDataset
conv_nets/binary_main.py
Custom dataset class for binary classification experiments
Key functions
Function
Module
Description
patchify(x, patch_size, stride_size)
cnfa_verification/pretrained_conv_nfa.py
Extracts overlapping patches from input tensor x
get_jacobian(net, data, c_idx)
cnfa_verification/pretrained_conv_nfa.py
Computes the Jacobian of network output w.r.t. input patches for class index c_idx
egop(model, X)
cnfa_verification/pretrained_conv_nfa.py
Computes the expected gradient outer product (EGOP) over dataset X using model
patchify(x, patch_size, stride_size)
cnfa_verification/vgg_conv_nfa.py
Patch extraction for VGG-style feature maps
get_jacobian(net, data, c_idx)
cnfa_verification/vgg_conv_nfa.py
Jacobian computation for VGG-based networks
egop(model, z)
cnfa_verification/vgg_conv_nfa.py
EGOP computation for VGG-based models
get_imagenet(batch_size, path)
cnfa_verification/dataset.py
Returns a DataLoader for ImageNet given batch size and dataset path
get_filter(net, layer)
conv_nets/binary_main.py
Extracts filter weights from a specified layer of network net
get_classes(X_full, y_full, c1)
conv_nets/binary_main.py
Filters dataset to return samples belonging to class c1 for binary tasks
Sub-modules
Module
Purpose
cnfa_verification/
CNFA verification using pretrained ResNet and VGG networks
conv_nets/
Standard CNN training, embedding extraction, and evaluation utilities
deep_conv_rfm/
Deep ConvRFM training loop, gradient computation, and model definitions
hyperparam_cnfa_verification/
Hyperparameter sensitivity analysis for CNFA
vgg_vis/
VGG kernel eigenvector computation and visualization
Common patterns and best practices
Patchify before Jacobian computation: Always apply patchify(x, patch_size, stride_size) to input tensors before passing them to get_jacobian. Patch size and stride should match the receptive field of the target convolutional layer.
EGOP over batches: For large datasets like ImageNet, compute EGOP in mini-batches and accumulate to avoid memory overflow. The egop function takes a model and a data tensor X; ensure X is moved to the appropriate device before calling.
Binary vs multiclass pipelines: Use binary_main.py for two-class problems and main.py for multiclass. The binary pipeline uses get_classes to filter datasets and get_filter to inspect learned filters.
Dataset paths: When using get_imagenet(batch_size, path), ensure path points to the root ImageNet directory containing train/ and val/ subdirectories in the standard ImageNet folder format.
Device management: The codebase is designed for GPU use. Always set tensors and models to .cuda() or the appropriate device before running Jacobian or EGOP computations.
Gradient computation: When calling get_jacobian, ensure the model is in evaluation mode (model.eval()) and that torch.no_grad() is not active, since Jacobian computation requires gradient tracking.
VGG visualization workflow: Use vgg_vis/kernel.py to compute kernels, then vgg_vis/pass_eigvs.py to propagate eigenvectors through the network, and finally vgg_vis/main.py to orchestrate the full visualization pipeline.
Toy data for debugging: Use gen_toy_data.py to generate small synthetic datasets when debugging pipeline components before scaling to ImageNet or CIFAR.
Demo Scripts
scripts/egop_and_patchify_demo.py
#!/usr/bin/env python3"""
Demo: EGOP computation and patch extraction using ConvRFM utilities
This script demonstrates how to use the core components of the convrfm
repository:
- patchify(): extract overlapping patches from image tensors
- get_jacobian(): compute per-patch Jacobians of a network output
- egop(): compute the Expected Gradient Outer Product (EGOP) over a dataset
- get_imagenet(): construct an ImageNet DataLoader
- PatchConvLayer: wrap a convolutional layer for patch-based processing
Requirements:
pip install torch torchvision numpy scipy
NOTE: This script is structured to run standalone with synthetic data
for demonstration. Replace IMAGENET_PATH with your actual ImageNet
root directory to use real data.
"""import sys
import os
import numpy as np
import torch
import torch.nn as nn
import torchvision.models as tv_models
import torchvision.transforms as transforms
from torch.utils.data import DataLoader, TensorDataset
# ---------------------------------------------------------------------------# Path setup: add repository root to sys.path so local modules are importable# when running from inside the cloned convrfm directory.# Adjust REPO_ROOT to point to the cloned convrfm repository on your system.# ---------------------------------------------------------------------------
REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
sys.path.insert(0, REPO_ROOT)
# ---------------------------------------------------------------------------# Configuration# ---------------------------------------------------------------------------
IMAGENET_PATH = "/path/to/imagenet"# Replace with your ImageNet root path
BATCH_SIZE = 16
PATCH_SIZE = 3
STRIDE_SIZE = 1
NUM_CLASSES =
DEVICE = torch.cuda.is_available()
()
():
( + * )
()
( * )
x = torch.randn(, , , )
()
:
cnfa_verification.pretrained_conv_nfa patchify
patches = patchify(x, patch_size=PATCH_SIZE, stride_size=STRIDE_SIZE)
()
()
()
(
)
ImportError e:
(
)
()
patches = standalone_patchify(x, patch_size=PATCH_SIZE, stride_size=STRIDE_SIZE)
()
patches
() -> torch.Tensor:
N, C, H, W = x.shape
patches = x.unfold(, patch_size, stride_size).unfold(, patch_size, stride_size)
n_h = patches.shape[]
n_w = patches.shape[]
patches = patches.contiguous().view(N, C, n_h * n_w, patch_size * patch_size)
patches = patches.permute(, , , ).contiguous()
patches = patches.view(N, n_h * n_w, C * patch_size * patch_size)
patches
():
( + * )
()
( * )
(nn.Module):
():
().__init__()
.fc = nn.Linear( * PATCH_SIZE * PATCH_SIZE, NUM_CLASSES)
():
N, P, D = x.shape
out = .fc(x.view(N * P, D))
out.view(N, P, NUM_CLASSES)
net = ToyConvNet().to(DEVICE)
net.()
data = torch.randn(, , * PATCH_SIZE * PATCH_SIZE, requires_grad=).to(DEVICE)
c_idx =
:
cnfa_verification.pretrained_conv_nfa get_jacobian
jac = get_jacobian(net, data, c_idx)
()
()
()
ImportError e:
()
()
jac = standalone_get_jacobian(net, data, c_idx)
()
()
()
jac
() -> torch.Tensor:
data = data.detach().requires_grad_()
output = net(data)
scalar = output[:, :, c_idx].()
scalar.backward()
jac = data.grad.clone()
jac
():
( + * )
()
( * )
patch_dim = * PATCH_SIZE * PATCH_SIZE
num_patches =
N =
(nn.Module):
():
().__init__()
.proj = nn.Linear(in_dim, n_classes)
() -> torch.Tensor:
N, P, D = x.shape
.proj(x.view(N * P, D)).view(N, P, -)
model = ToyNet(in_dim=patch_dim, n_classes=NUM_CLASSES).to(DEVICE)
model.()
X = torch.randn(N, num_patches, patch_dim).to(DEVICE)
:
cnfa_verification.pretrained_conv_nfa egop
egop_matrix = egop(model, X)
()
()
()
ImportError e:
()
()
egop_matrix = standalone_egop(model, X)
()
()
egop_matrix
() -> np.ndarray:
N, P, D = X.shape
egop_accum = np.zeros((D, D), dtype=np.float64)
i (N):
x_i = X[i:i+].detach().requires_grad_()
output = model(x_i)
num_classes = output.shape[-]
c (num_classes):
scalar = output[, :, c].()
scalar.backward(retain_graph=(c < num_classes - ))
x_i.grad :
g = x_i.grad[].detach().cpu().numpy()
egop_accum += g.T @ g
x_i.grad.zero_()
egop_accum /= N
egop_accum
():
( + * )
()
( * )
os.path.exists(IMAGENET_PATH):
:
cnfa_verification.dataset get_imagenet
loader = get_imagenet(batch_size=BATCH_SIZE, path=IMAGENET_PATH)
batch = ((loader))
images, labels = batch
()
()
()
loader
ImportError e:
()
:
()
()
fake_images = torch.randn(, , , )
fake_labels = torch.randint(, , (,))
fake_dataset = TensorDataset(fake_images, fake_labels)
fake_loader = DataLoader(fake_dataset, batch_size=BATCH_SIZE, shuffle=)
batch = ((fake_loader))
images, labels = batch
(
)
()
(
)
fake_loader
():
( + * )
()
( * )
(nn.Module):
():
().__init__()
.conv1 = nn.Conv2d(, , kernel_size=, padding=)
.conv2 = nn.Conv2d(, , kernel_size=, padding=)
.fc = nn.Linear( * * , )
() -> torch.Tensor:
x = torch.relu(.conv1(x))
x = torch.relu(.conv2(x))
x = x.view(x.size(), -)
.fc(x)
net = SimpleConvNet()
:
conv_nets.binary_main get_filter
filters = get_filter(net, layer=)
()
ImportError e:
()
()
filters = standalone_get_filter(net, layer=)
()
np.random.seed()
X_full = np.random.randn(, ).astype(np.float32)
y_full = np.random.randint(, , size=)
c1 =
:
conv_nets.binary_main get_classes
X_c1, y_c1 = get_classes(X_full, y_full, c1)
()
()
()
()
ImportError e:
()
()
X_c1, y_c1 = standalone_get_classes(X_full, y_full, c1)
()
()
()
()
() -> torch.Tensor:
(net, layer).weight.data
() -> :
mask = y_full == c1
X_full[mask], y_full[mask]
():
( + * )
()
( * )
N, C, H, W = , , ,
n_classes =
patch_size =
stride =
patch_dim = C * patch_size * patch_size
images = torch.randn(N, C, H, W).to
"""
Demonstrate the patchify function from cnfa_verification/pretrained_conv_nfa.py.
patchify(x, patch_size, stride_size) extracts overlapping spatial patches
from a 4D tensor of shape (N, C, H, W), returning a tensor of shape
(N, num_patches, C * patch_size * patch_size).
"""
x: torch.Tensor, patch_size: int, stride_size: int
"""
Standalone reimplementation of patchify() for demonstration purposes.
Mirrors the logic in cnfa_verification/pretrained_conv_nfa.py.
Args:
x (torch.Tensor): Input tensor of shape (N, C, H, W).
patch_size (int): Height and width of each square patch.
stride_size (int): Stride between consecutive patches.
Returns:
torch.Tensor: Tensor of shape (N, num_patches, C * patch_size * patch_size).
"""
"""
Demonstrate get_jacobian() from cnfa_verification/pretrained_conv_nfa.py.
get_jacobian(net, data, c_idx) computes the Jacobian of the network output
for class index c_idx with respect to the input data (patches).
The Jacobian shape is (num_patches, input_dim) for a single sample or
batched as (N, num_patches, input_dim).
"""
print
"\n"
"="
60
print
"Section 2: get_jacobian() demonstration"
print
"="
60
# Build a minimal network for illustration
# In practice, this would be a pretrained ResNet or VGG layer wrapper
class
ToyConvNet
def
__init__
self
super
self
3
def
forward
self, x
# x: (N, num_patches, patch_dim) -- flatten for demo
"""
Standalone Jacobian computation mirroring the logic in
cnfa_verification/pretrained_conv_nfa.py.
Args:
net (nn.Module): Network to differentiate through.
data (torch.Tensor): Patchified input of shape (N, num_patches, patch_dim).
c_idx (int): Class index to compute gradients for.
Returns:
torch.Tensor: Jacobian of shape (N, num_patches, patch_dim).
"""
"""
Demonstrate egop() from cnfa_verification/pretrained_conv_nfa.py.
egop(model, X) computes the Expected Gradient Outer Product over dataset X.
The EGOP is the average of J^T J over all samples, where J is the Jacobian
of the network output with respect to the input features. This forms the
basis of the Neural Feature Ansatz.
Returns:
np.ndarray: EGOP matrix of shape (patch_dim, patch_dim).
"""
print
"\n"
"="
60
print
"Section 3: egop() demonstration"
print
"="
60
3
16
8
# Small synthetic dataset
class
ToyNet
"""Minimal network for EGOP demonstration."""
def
__init__
self, in_dim: int, n_classes: int
super
self
def
forward
self, x: torch.Tensor
# x: (N, num_patches, patch_dim)
return
self
1
eval
# Synthetic dataset: (N, num_patches, patch_dim)
try
from
import
print
f"Input X shape: {X.shape}"
print
f"EGOP matrix shape: {egop_matrix.shape}"
print
f"EGOP matrix dtype: {egop_matrix.dtype}"
except
as
print
f"[INFO] Could not import from repository: {e}"
print
"[FALLBACK] Running standalone EGOP computation:"
print
f"Input X shape: {X.shape}"
print
f"EGOP matrix shape: {egop_matrix.shape}"
return
def
standalone_egop
model: nn.Module, X: torch.Tensor
"""
Standalone EGOP computation mirroring egop() in
cnfa_verification/pretrained_conv_nfa.py.
Computes E[J^T J] where J is the Jacobian of all class outputs
w.r.t. each patch, averaged over the dataset.
Args:
model (nn.Module): Trained neural network.
X (torch.Tensor): Patchified dataset of shape (N, num_patches, patch_dim).
Returns:
np.ndarray: EGOP matrix of shape (patch_dim, patch_dim).
"""
"""
Demonstrate get_imagenet() from cnfa_verification/dataset.py.
get_imagenet(batch_size, path) returns a PyTorch DataLoader for the
ImageNet validation set. Requires a local copy of ImageNet at `path`.
This demo shows the expected call signature and what the loader returns.
It uses a synthetic TensorDataset as fallback when the real path is absent.
"""
"""
Demonstrate get_filter() and get_classes() from conv_nets/binary_main.py.
get_filter(net, layer): extracts the weight tensor from a named layer
of network `net`.
get_classes(X_full, y_full, c1): filters dataset arrays to return only
samples belonging to class c1, used for binary classification setup.
"""
print
"\n"
"="
60
print
"Section 5: get_filter() and get_classes() demonstration"
print
"="
60
# --- get_filter() ---
class
SimpleConvNet
"""Minimal CNN with named layers for filter extraction demo."""
"""
Standalone implementation of get_filter() from conv_nets/binary_main.py.
Args:
net (nn.Module): Trained convolutional network.
layer (str): Name of the layer to extract filters from.
Returns:
torch.Tensor: Weight tensor of the specified layer.
"""
return
getattr
def
standalone_get_classes
X_full: np.ndarray,
y_full: np.ndarray,
c1: int
tuple
"""
Standalone implementation of get_classes() from conv_nets/binary_main.py.
Args:
X_full (np.ndarray): Full feature matrix of shape (N, D).
y_full (np.ndarray): Full label array of shape (N,).
c1 (int): Target class index to filter.
Returns:
tuple: (X_c1, y_c1) -- filtered features and labels for class c1.
"""
