TRIBE v2 tri-modal foundation model for in-situ fMRI brain-to-image decoding with synthetic data augmentation. Addresses low-data regime challenges in brain decoding.
TRIBE v2 tri-modal foundation model for in-situ fMRI brain-to-image decoding with synthetic data augmentation. Addresses low-data regime challenges in brain decoding.
TRIBE v2: Tri-modal Foundation Model for Brain Decoding
Overview
TRIBE v2 is a large encoding model pretrained on more than 1000 hours of fMRI responses to video, audio, and language stimuli. It enables synthetic data augmentation for boosting brain-to-image decoding in low-data regimes.
Key Contribution: Addresses the fundamental limitation in brain decoding — availability of labeled neural data — by using pretrained foundation models to generate synthetic fMRI data for augmentation.
Core Methodology
1. Tri-modal Foundation Model Architecture
Pretraining: 1000+ hours of fMRI responses to:
Video stimuli
Audio stimuli
Language stimuli
Encoding model: Maps stimuli → fMRI responses
Synthetic generation: Given new stimuli, generate predicted fMRI responses
2. Data Augmentation Strategy
Grid-based evaluation:
Systematic grids showing how augmentation effectiveness varies with:
Amount of synthetic data added
Quality threshold for synthetic samples
Domain mismatch between pretrained model and target dataset
Training strategy (augment vs. pretrain)
3. Low-Data Regime Applications
When augmentation helps:
Small fMRI datasets (< 50 subjects)
Limited labeled stimuli
Novel stimulus types not in pretraining
Cross-subject generalization
When augmentation is less effective:
Large datasets (> 500 subjects)
Domain mismatch without adaptation
Very different stimulus modalities
Technical Details
Synthetic Data Generation Pipeline
Input Stimuli → TRIBE v2 Encoder → Predicted fMRI → Quality Filter → Augmented Training Set
Quality filtering criteria:
Prediction confidence threshold
Activation pattern similarity to real data
Region-of-interest consistency
Training Strategies
Option A: Direct Augmentation
# Mix real and synthetic fMRI data
augmented_dataset = real_fMRI + synthetic_fMRI[quality > threshold]
model.train(augmented_dataset)
Option B: Pretrain-then-Finetune
# Use TRIBE v2 as pretrained backbone
pretrained_encoder = TRIBE_v2.load()
finetuned_decoder = pretrained_encoder.adapt(target_dataset)
Brain-to-Image Decoding
Traditional limitation:
Requires large fMRI-to-image paired datasets
Subject-specific training expensive
Limited stimulus diversity
TRIBE v2 solution:
Generate synthetic fMRI for any image/video stimulus
Expand training set without additional scanning
Enable zero-shot or few-shot decoding for novel stimuli
Use Cases
1. Visual Reconstruction from fMRI
Problem: Reconstruct viewed images from brain activity
TRIBE v2 approach: Augment with synthetic fMRI generated from image dataset
2. Cross-subject Transfer
Problem: Train decoder on few subjects, generalize to new subjects
Solution: Use TRIBE v2 pretrained representations as shared basis
3. Novel Stimulus Modalities
Problem: Decode brain responses to stimuli not in training set
Approach: Generate synthetic fMRI for new stimuli using TRIBE v2 encoder
Related: Brain-DiT, NeuroSTORM, fMRI foundation models
Activation: Use this skill when working on brain decoding, fMRI data augmentation, foundation models for neuroscience, or low-data regime brain-to-image reconstruction.
