Skip to main content Accueil Créateurs zjunlp mechanist tp4-feature-learning
tp4-feature-learning Use this skill when working with infinite-width neural networks for feature learning, replicating Word2Vec or MAML experiments from the Tensor Programs series (TP4), or implementing infinite-width limits (GP, NTK, muP) for meta-learning and word embedding tasks.
Aller à l'installation Skills Marketplace Découvrez et explorez les compétences IA créées par la communauté.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Afficher les détails du prompt Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
npx skills add https://github.com/zjunlp/Mechanist --skill tp4-feature-learningLa commande reste sur une seule ligne. Faites défiler horizontalement pour la vérifier avant de la copier.
Vous préférez une copie locale ? Téléchargez les fichiers actuellement disponibles dans SkillsMP.
Télécharger Zip Téléchargement... The single place for every data constraint an experiment must satisfy — dataset provenance (existing → adapted → constructed), clear train / validation / test splits, labels that reflect the target behavior, and the minimum data amount. Use whenever an experiment chooses, adapts, or constructs a dataset, defines splits, or sets a sample size — for phenomenon validation (M0), mechanism exploration, intervention, or tuning. Domain-general: no assumption about model family, modality, or task.
Métiers associés SOC
Basé sur la classification professionnelle SOC
Explorateur de fichiers
5 fichiers article_references.md 129 B name tp4-feature-learning description Use this skill when working with infinite-width neural networks for feature learning, replicating Word2Vec or MAML experiments from the Tensor Programs series (TP4), or implementing infinite-width limits (GP, NTK, muP) for meta-learning and word embedding tasks.
TP4: Feature Learning in Infinite-Width Neural Networks
When to Use
Implementing infinite-width neural network experiments (GP, NTK, muP/feature learning limits)
Replicating Word2Vec experiments with infinite-width models
Running MAML (Model-Agnostic Meta-Learning) with infinite-width networks on Omniglot
Studying the Tensor Programs series of papers empirically
Comparing finite vs. infinite-width neural network behavior
Keywords: infinite-width, feature learning, NTK, GP, muP, MAML, Word2Vec, Tensor Programs, meta-learning
Quick Reference
Installation / Setup
Prerequisites
Python 3.x
C compiler (for Word2Vec C source)
PyTorch
MAML Experiment Setup
cd TP4MAML
pip install -r requirements.txt
cd meta
pip install -r requirements.txt
Word2Vec Experiment Setup
cd Word2Vec
make
bash scripts/create-text8-data.sh
bash scripts/create-fil9-data.sh
Core Features
InfGP1LP: Infinite-width Gaussian Process limit for a 1-hidden-layer perceptronFinGP1LP: Finite-width GP baseline for comparisonInfNTK1LP: Infinite-width NTK limit for a 1-hidden-layer perceptronInfSGD: Custom SGD optimizer for infinite-width networks with proper scalingInfMultiStepLR: Learning rate scheduler compatible with InfSGDInfMAML: Infinite-width MAML metalearner for Omniglot few-shot classificationCachedOmniglot: Efficient cached Omniglot dataset loaderWord2Vec C implementation: Modified word2vec supporting infinite-width training modes
Usage Examples
Running All MAML Experiments
cd TP4MAML
bash train_all.sh
Training MAML (finite width)
cd TP4MAML/meta
python train.py --dataset omniglot --num-ways 5 --num-shots 1
Training Infinite-Width MAML
cd TP4MAML/meta
python train.py --dataset omniglot --num-ways 5 --num-shots 1 --inf
Word2Vec Training (text8, standard)
cd Word2Vec
bash scripts/train-text8.sh
Word2Vec Training (text8, infinite-width)
cd Word2Vec
bash scripts/train-text8-inf.sh
Word2Vec Evaluation
cd Word2Vec
bash scripts/evaluate.sh
Key APIs / Models
TP4MAML/inf/inf1lp.py
InfGP1LP — Infinite GP limit 1-layer perceptronFinGP1LP — Finite GP baselineInfNTK1LP — Infinite NTK limit 1-layer perceptron
TP4MAML/inf/optim.py
InfSGD(params, lr, ...) — SGD optimizer scaled for infinite-width networksInfMultiStepLR(optimizer, milestones, gamma) — LR scheduler for InfSGD
TP4MAML/inf/utils.py
safe_sqrt(arr, eps) — Numerically stable square rootsafe_acos(arr, eps) — Numerically stable arccosF00ReLU(c, v, v2) — ReLU kernel function used in GP/NTK computationsMyLinear — Custom linear layer with infinite-width scaling
TP4MAML/meta/maml/metalearners/infmaml.py
InfMAML — Meta-learner implementing MAML for infinite-width networks
TP4MAML/meta/maml/metalearners/maml.py
MAML — Standard MAML meta-learner
TP4MAML/meta/cached_omniglot.py
CachedOmniglot — Omniglot dataset with cachingOmniglotClassDataset — Per-class Omniglot datasetOmniglot — Base Omniglot loader
TP4MAML/inf/dynamicarray.py
DynArr — Dynamic array for storing activations during infinite-width forward passesCycArr — Cyclic array variant
Common Patterns & Best Practices
Use --inf flag in train.py to switch between finite and infinite-width MAML
The infinite-width models do not store explicit weights; instead they accumulate kernel computations
For Word2Vec, the train-*-inf.sh scripts set hyperparameters appropriate for infinite-width training
Always build the C binaries before running Word2Vec experiments (make in Word2Vec/)
The train_all.sh script in TP4MAML/ runs all configurations sequentially for full replication
Demo Scripts
scripts/inf_network_demo.py
"""
TP4 Feature Learning - Infinite-Width Network Demo
Demonstrates usage of the TP4MAML inf module:
- InfGP1LP, InfNTK1LP for infinite-width 1-hidden-layer perceptrons
- InfSGD optimizer
- Utility functions (safe_sqrt, safe_acos, F00ReLU)
Requires: torch, numpy
Run from the repo root: python scripts/inf_network_demo.py
"""
import sys
import os
sys.path.insert(0 , os.path.join(os.path.dirname(__file__), '..' , 'TP4MAML' ))
import torch
import torch.nn as nn
import numpy as np
def demo_utils ():
"""Demonstrate utility functions from TP4MAML/inf/utils.py"""
try :
from inf.utils import safe_sqrt, safe_acos, F00ReLU, MyLinear
print ("=== Utility Functions ===" )
arr = torch.tensor([-1e-10 , 0.0 , 1.0 , 4.0 ])
result = safe_sqrt(arr, eps=1e-6 )
print (f"safe_sqrt({arr.tolist()} ) = {result.tolist()} " )
arr2 = torch.tensor([-1.0 - 1e-10 , -1.0 , 0.0 , 1.0 , 1.0 + 1e-10 ])
result2 = safe_acos(arr2, eps=1e-6 )
print (f"safe_acos(clipped) = )
scripts/maml_training_demo.py
"""
TP4 MAML Training Demo
Shows how to invoke MAML training programmatically (mirroring train.py usage).
Demonstrates the dataset loading and metalearner API.
Requires: torch, torchmeta (see TP4MAML/meta/requirements.txt)
Run from TP4MAML/meta/: python ../../scripts/maml_training_demo.py
"""
import sys
import os
import argparse
MAML_META_PATH = os.path.join(os.path.dirname(__file__), '..' , 'TP4MAML' , 'meta' )
MAML_INF_PATH = os.path.join(os.path.dirname(__file__), '..' , 'TP4MAML' )
sys.path.insert(0 , MAML_META_PATH)
sys.path.insert(0 , MAML_INF_PATH)
def build_omniglot_dataset (data_folder: str , num_ways: int = 5 , num_shots: int = 1 ,
num_shots_test: int = 15 ):
"""
Build Omniglot meta-learning dataset using CachedOmniglot.
Args:
data_folder: Path to Omniglot data directory.
num_ways: Number of classes per episode (N-way).
num_shots: Number of support examples per class (K-shot).
num_shots_test: Number of query examples per class.
Returns:
Tuple of (meta_train_dataset, meta_val_dataset, meta_test_dataset)
"""
try :
from cached_omniglot import CachedOmniglot
import torchmeta
from torchmeta.transforms import ClassSplitter, Categorical
from torchvision.transforms import Compose, Resize, ToTensor
transform = Compose([Resize(28 ), ToTensor()])
meta_train = CachedOmniglot(
data_folder,
num_classes_per_task=num_ways,
transform=transform,
target_transform=Categorical(num_ways),
class_augmentations=[torchmeta.transforms.Rotation([{result2.tolist()}
0.5
1.0
1.0
print
f"F00ReLU(c=0.5, v=1, v2=1) = {k.item():.4 f} "
10
5
3
10
print
f"MyLinear(10->5) output shape: {out.shape} "
except
as
print
f"Could not import inf.utils (run from repo root with TP4MAML in path): {e} "
def
demo_dynamic_arrays
"""Demonstrate DynArr and CycArr from TP4MAML/inf/dynamicarray.py"""
try
from
import
print
"\n=== Dynamic Arrays ==="
for
in
range
5
3
4
print
f"DynArr length after 5 appends: {len (darr)} "
3
for
in
range
6
float
print
f"CycArr (capacity=3) last 3 values: {[carr[i].item() for i in range (len (carr))]} "
except
as
print
f"Could not import inf.dynamicarray: {e} "
def
demo_inf_models
"""Demonstrate InfGP1LP, FinGP1LP, InfNTK1LP from TP4MAML/inf/inf1lp.py"""
try
from
import
print
"\n=== Infinite-Width 1-Layer Perceptron Models ==="
16
5
8
0
4
print
"Building InfGP1LP..."
print
f" InfGP1LP created: {type (gp_model).__name__} "
print
"Building InfNTK1LP..."
print
f" InfNTK1LP created: {type (ntk_model).__name__} "
print
"Building FinGP1LP..."
256
print
f" FinGP1LP created: {type (fin_model).__name__} "
print
f" FinGP1LP forward output shape: {logits.shape} "
except
as
print
f"Could not import inf.inf1lp: {e} "
except
as
print
f"Error in inf model demo: {e} "
def
demo_inf_sgd
"""Demonstrate InfSGD optimizer from TP4MAML/inf/optim.py"""
try
from
import
print
"\n=== InfSGD Optimizer ==="
10
5
0.01
0.9
print
f"InfSGD created with lr=0.01, momentum=0.9"
10
20
0.1
print
f"InfMultiStepLR created with milestones=[10, 20], gamma=0.1"
4
10
0
5
4
for
in
range
3
print
f" Step {step+1 } : loss={loss.item():.4 f} , lr={scheduler.get_last_lr()} "
except
as
print
f"Could not import inf.optim: {e} "
except
as
print
f"Error in InfSGD demo: {e} "
def
demo_maml_structure
"""Show the MAML metalearner interface (TP4MAML/meta/maml/metalearners/)"""
try
0
'..'
'TP4MAML'
'meta'
from
import
print
"\n=== MAML Metalearner ==="
print
f"MAML class imported: {MAML} "
print
" MAML implements standard Model-Agnostic Meta-Learning."
print
" Use train.py --inf flag to switch to InfMAML."
except
as
print
f"\nCould not import MAML metalearner: {e} "
if
"__main__"
print
"TP4 Feature Learning in Infinite-Width Neural Networks - Demo\n"
print
"\nDemo complete."
90
180
270
True
True
print
f"CachedOmniglot meta-train: {len (meta_train)} tasks"
return
except
as
print
f"Could not build Omniglot dataset (missing torchmeta or data): {e} "
return
None
except
as
print
f"Dataset error: {e} "
return
None
def
demonstrate_metalearner_api
"""
Show the API surface of MAML and InfMAML metalearners.
"""
print
"=== MAML / InfMAML API ===\n"
try
from
import
print
f"MAML class: {MAML.__module__} .{MAML.__name__} "
print
f" __init__ signature: see references/api_reference.md"
print
f" Key methods: train(), evaluate(), get_outer_loss()"
except
as
print
f"MAML import failed: {e} "
try
from
import
print
f"\nInfMAML class: {InfMAML.__module__} .{InfMAML.__name__} "
print
f" Extends MAML for infinite-width networks using InfGP1LP/InfNTK1LP"
except
as
print
f"InfMAML import failed: {e} "
try
from
import
print
f"\nMetaSGD class: {MetaSGD.__module__} .{MetaSGD.__name__} "
print
f" Meta-SGD variant with per-parameter learned learning rates"
except
as
print
f"MetaSGD import failed: {e} "
def
show_training_command_equivalents
"""
Print the equivalent train.py CLI commands for common configurations.
"""
print
"\n=== Equivalent train.py Commands ===\n"
"description"
"5-way 1-shot Omniglot, finite MAML"
"cmd"
"python train.py --dataset omniglot --num-ways 5 --num-shots 1 --num-steps 5"
"description"
"5-way 1-shot Omniglot, infinite-width MAML (GP limit)"
"cmd"
"python train.py --dataset omniglot --num-ways 5 --num-shots 1 --num-steps 5 --inf --use-gp"
"description"
"5-way 5-shot Omniglot, infinite-width MAML (NTK limit)"
"cmd"
"python train.py --dataset omniglot --num-ways 5 --num-shots 5 --num-steps 5 --inf"
"description"
"Run all experiments (uses train_all.sh)"
"cmd"
"cd TP4MAML && bash train_all.sh"
for
in
print
f"# {cfg['description' ]} "
print
f" {cfg['cmd' ]} \n"
def
show_word2vec_commands
"""
Show Word2Vec experiment commands.
"""
print
"=== Word2Vec Experiment Commands ===\n"
"Build C binaries"
"cd Word2Vec && make"
"Prepare text8 data"
"bash Word2Vec/scripts/create-text8-data.sh"
"Prepare fil9 data"
"bash Word2Vec/scripts/create-fil9-data.sh"
"Train text8 (finite)"
"bash Word2Vec/scripts/train-text8.sh"
"Train text8 (infinite)"
"bash Word2Vec/scripts/train-text8-inf.sh"
"Train fil9 (finite)"
"bash Word2Vec/scripts/train-fil9.sh"
"Train fil9 (infinite)"
"bash Word2Vec/scripts/train-fil9-inf.sh"
"Evaluate embeddings"
"bash Word2Vec/scripts/evaluate.sh"
for
in
print
f"# {name} "
print
f" {cmd} \n"
if
"__main__"
print
"TP4 MAML Training Demo\n"
if
len
1
1
print
f"\n=== Building Omniglot Dataset from {data_folder} ==="
5
1
if
is
not
None
print
"Dataset built successfully."
else
print
"\nTip: Pass a data folder path as argument to test dataset loading."
print
" python maml_training_demo.py /path/to/omniglot/data"
