brainsimsiam-self-supervised-fmri

name	brainsimsiam-self-supervised-fmri
description	Lightweight self-supervised representation learning for fMRI using positive-only data pairs, achieving strong cross-task generalization without large-scale pretraining

BrainSimSiam: Siamese Self-Supervised Learning for Robust fMRI Representations

arXiv: 2605.28990
Authors: Jiyao Wang, Peiyu Duan, Nicha C. Dvornek, Lawrence H. Staib, Denis Sukhodolsky, Pamela Ventola, James S. Duncan
Published: 2026-05-27
Categories: cs.LG

Background

fMRI analysis faces fundamental challenges:

• Small sample sizes: High data acquisition costs → limited datasets
• Variable label quality: Subjective psychiatric rating scales
• High dimensionality: 100K+ voxels → overfitting risk
• Foundation model cost: Large-scale pretraining computationally prohibitive

Supervised models struggle with generalization across tasks, subjects, and clinical populations.

Methodology

BrainSimSiam Framework

Positive-only contrastive learning (no negative samples):

• Based on SimSiam architecture (Chen & He, 2021)
• Avoids large batch sizes for negative sampling
• Data-efficient: works with small fMRI datasets

Architecture

Input fMRI (4D volume)
  ↓
3D CNN Encoder (voxel-wise features)
  ↓
Projector MLP (representation space)
  ↓
Predictor MLP (asymmetric branch)
  ↓
Stop-gradient (no negative samples)
  ↓
Cosine similarity loss

Key Components

1. Augmentation Pipeline

Augmentation(fMRI):
  - Spatial jittering (±2 voxels)
  - Temporal windowing (random time segments)
  - Gaussian noise injection
  - Voxel intensity scaling
  → Two augmented views: v1, v2

2. Positive Pair Formation

• Same fMRI scan → two augmented views
• No cross-subject negative pairs (simplifies training)

3. Stop-gradient Mechanism

L = -cos(p1, z2) / 2 - cos(p2, z1) / 2
  where z1, z2 = encoder outputs (stop-gradient)
        p1, p2 = predictor outputs (learnable)

4. Representation Extraction

• Use encoder output for downstream tasks
• Linear evaluation protocol (frozen encoder)

Key Findings

Cross-Task Generalization

BrainSimSiam representations generalize across diverse downstream tasks:

Task	Performance vs Supervised	vs Foundation Models
ADHD classification	+15% accuracy	~90% of large models
Age regression (cognitive)	-8% MSE	Comparable
Autism detection	+12% F1	85% of foundation
Emotion recognition	+10% accuracy	88% of pretrained
Disease severity (clinical)	+18% correlation	92% of large-scale

Advantages

1. Data-efficient: Works with small datasets (50-200 subjects)
2. Task-invariant: One representation → multiple downstream tasks
3. Robust: Outperforms supervised baselines in low-data regimes
4. Lightweight: No large-scale pretraining required
5. Computationally feasible: Hours vs weeks for foundation models

Comparison

• Outperforms fully supervised baselines (task-specific models)
• Approaches large-scale foundation models (e.g., BrainLM, NeuroBERT)
• More efficient than contrastive learning with negatives (requires large batches)
• Better than random initialization for linear probing
• Comparable to supervised pretraining but more generalizable

Applications

Use Cases

1. Clinical neuroimaging: Small patient datasets (50-100 subjects)
2. Multi-task learning: Single representation for diverse tasks
3. Data-limited domains: Rare neurological conditions, pediatric populations
4. Transfer learning: Pretrain on healthy → fine-tune on clinical
5. Cost-effective analysis: Avoid large-scale foundation model training

Trigger Conditions

• Keywords: self-supervised fMRI, Siamese learning, cross-task generalization, BrainSimSiam, positive-only contrastive
• Context: Small fMRI datasets, multi-task analysis, clinical populations
• Problem: Overfitting, high dimensionality, label scarcity, computational constraints

Pitfalls

Limitations

1. Augmentation sensitivity: Spatial/temporal augmentations must preserve semantic content
2. Encoder architecture: 3D CNN choice matters → may need customization
3. Domain mismatch: Healthy brain pretraining → clinical fine-tuning may fail
4. Task-specific bias: Representations may favor certain tasks (e.g., classification > regression)
5. No explicit negatives: May collapse to trivial solutions (stop-gradient prevents)

Edge Cases

• Very small datasets: < 50 subjects → representations may be unstable
• Highly heterogeneous subjects: Age/condition extremes → augmentation may break semantics
• Non-structural tasks: Functional connectivity analyses → voxel-wise features insufficient
• Multi-site data: Scanner variability → augmentation may not address site effects
• 4D temporal dynamics: Long time-series → temporal windowing may miss global patterns

Implementation

Pseudocode

class BrainSimSiam:
    def __init__(encoder, projector, predictor):
        self.encoder = encoder    # 3D CNN (voxel → features)
        self.projector = projector  # MLP (features → z)
        self.predictor = predictor  # MLP (z → p)
    
    def forward(fMRI_volume):
        # Augmentation
        v1 = augment(fMRI_volume)  # Spatial jitter + temporal window
        v2 = augment(fMRI_volume)  # Different augment
        
        # Encoder
        h1 = self.encoder(v1)
        h2 = self.encoder(v2)
        
        # Projector
        z1 = self.projector(h1)
        z2 = self.projector(h2)
        
        # Predictor (asymmetric)
        p1 = self.predictor(z1)
        p2 = self.predictor(z2)
        
        # Stop-gradient + cosine similarity
        loss = -cos(p1, stop_grad(z2)) / 2 - cos(p2, stop_grad(z1)) / 2
        
        return loss
    
    def extract_representation(fMRI_volume):
        return self.encoder(fMRI_volume)  # Frozen encoder for downstream

# Downstream evaluation
for task in [ADHD, Autism, Age, Emotion, Severity]:
    representation = brain_simsiam.extract_representation(fMRI)
    prediction = linear_head(representation)  

Computational Complexity

• Encoder: O(V × K) where V=voxels (100K), K=kernel operations
• Training: ~2-4 hours for 100 subjects (single GPU)
• Inference: O(V × K) for representation extraction
• Downstream linear probe: O(D × T) where D=representation dim, T=task outputs

References

• brain-dit-fmri-foundation-model - Brain-DiT foundation model
• eeg-foundation-model-adapters - EEG foundation model adapters
• meta-learning-in-context-brain-decoding - Meta-learning for brain decoding
• layer-wise-interactive-dual-stream-network-for-e - LI-DSN EEG decoding

---

See also: Self-supervised learning, SimSiam, contrastive learning, fMRI foundation models, cross-task generalization, data-efficient representation learning