Linear contrastive decoders outperform complex nonlinear models for fMRI-based brain decoding. Key insight: fMRI averaging linearizes representations, so training objective (contrastive alignment) matters more than architectural complexity. Validated across vision, language, audio. Activation: brain decoding, fMRI, contrastive learning, linear decoder, alignment, foundation models.
Linear contrastive decoders outperform complex nonlinear models for fMRI-based brain decoding. Key insight: fMRI averaging linearizes representations, so training objective (contrastive alignment) matters more than architectural complexity. Validated across vision, language, audio. Activation: brain decoding, fMRI, contrastive learning, linear decoder, alignment, foundation models.
Training objective > architectural complexity for brain decoding from fMRI. Linear contrastive decoders consistently outperform ridge regression and nonlinear alternatives because fMRI measurements effectively linearize neural representations through spatial/temporal averaging and noise.
Theoretical Foundation
Why Linear Works for fMRI
Spatial Averaging: Each voxel aggregates ~10^5 neurons
Temporal Averaging: HRF smooths neural activity over ~6s
Measurement Noise: Further linearizes observable representations
Result: High-dimensional nonlinear neural computations appear linear at fMRI resolution
Contrastive Learning for Brain Decoding
Objective: Align fMRI patterns with foundation model embeddings
Biological Plausibility: Matches theory that concepts organized as high-dimensional vectors
Semantic Structure: Directions and angles in embedding space capture meaning
Methodology
Linear Contrastive Decoder
# Simplified workflow1. Extract fMRI features for stimulus X
2. Extract foundation model embedding for X
3. Learn linear map: W such that fMRI @ W ≈ embedding
4. For retrieval: find nearest embedding to decoded fMRI pattern
Training Objective
Contrastive Loss: Maximize similarity between matched fMRI-embedding pairs
Negative Sampling: Push apart mismatched pairs
Alignment: Maps fMRI space into foundation model embedding space
Foundation Models Used
Vision: CLIP, DINO
Language: Sentence-BERT, GPT embeddings
Audio: AudioCLIP, wav2vec
Validation Results
Across Modalities
Modality
Linear Contrastive vs. Ridge
vs. Nonlinear
Vision
+15-25% accuracy
+10-20% accuracy
Language
+12-18% accuracy
+8-15% accuracy
Audio
+10-15% accuracy
+5-12% accuracy
Key Findings
Linear > Nonlinear: Simpler models work better
Contrastive > Ridge: Training objective is critical
Generalization: Results hold across multiple datasets and modalities
Interpretability: Linear maps are more interpretable than deep networks
Practical Implementation
Data Preprocessing
# Standard fMRI preprocessing1. Motion correction
2. Spatial normalization to MNI space
3. Temporal filtering (0.01-0.1 Hz)
4. ROI extraction or voxel selection
5. Z-scoring within run
Contrastive Training
# Pseudocodefor fMRI_batch, embedding_batch in dataloader:
# Positive pairs: (fMRI_i, embedding_i)# Negative pairs: (fMRI_i, embedding_j) for i≠j
decoded = linear_model(fMRI_batch) # Linear projection
pos_sim = cosine_similarity(decoded, embedding_batch)
neg_sim = compute_all_pairs(decoded, embedding_batch)
loss = contrastive_loss(pos_sim, neg_sim)
loss.backward()
Hyperparameters
Learning Rate: 1e-3 to 1e-4
Batch Size: 32-128 (depends on data size)
Temperature: 0.07-0.1 for contrastive loss
Regularization: L2 weight decay 1e-5
Advantages Over Complex Models
Data Efficiency: Linear models need less data to avoid overfitting
Training Speed: Orders of magnitude faster than deep networks
Interpretability: Weights directly show which fMRI features map to which embeddings
Robustness: Less sensitive to hyperparameter choices
Transferability: Can reuse pretrained foundation models without fine-tuning
Limitations & Considerations
Resolution Limit: Linearization is fMRI-specific; may not apply to single-unit or ECoG
Foundation Model Bias: Decoder quality depends on foundation model quality
Alignment Assumption: Assumes fMRI and embeddings share representational geometry
