tribe-v2-foundation-model

# TRIBE v2: Tri-Modal Foundation Model for In-Silico Neuroscience ## Overview TRIBE v2 is a foundation model that predicts high-resolution fMRI brain activity from video, audio, and text stimuli. Built on a unified dataset of 1000+ hours of fMRI across 720 subjects, it enables zero-shot generalization to novel stimuli, tasks, and subjects. **arXiv**: 2605.04326 (May 2026) **Authors**: d'Ascoli, Rapin, Benchetrit, Brooks, Begany, Raugel, Banville, King (FAIR at Meta) **Code**: https://github.com/facebookresearch/tribev2 **Weights**: https://huggingface.co/facebook/tribev2 ## Core Methodology ### Architecture - **Transformer encoder** with modality dropout and subject-specific blocks - **Tri-modal input**: video (ImageBind/Vision), audio (Wav2Vec), language (LLaMA) embeddings - **Unseen subject prediction**: learns population-level priors for zero-shot subject generalization ### Training Pipeline 1. **Feature Extraction**: Pre-trained AI models extract embeddings per modality 2. **Modality Dropout**: Randomly drop modalities during training for robustness 3. **Subject Blocks**: Per-subject linear adapters capture individual variability 4. **High-Resolution fMRI**: Predicts at vertex-level resolution (not just ROI averages) ### Four Essential Criteria 1. **Integration**: Whole-brain responses across diverse experimental conditions 2. **Performance**: Exceeds traditional linear encoding models 3. **Generalization**: Zero-shot to novel stimuli, tasks, and subjects 4. **Interpretability**: Decomposes cognitive function organization ## Key Findings ### Encoding Performance - Accurately predicts cortical AND subcortical responses across naturalistic and experimental conditions - Several-fold improvements over classic linear baselines - Scaling laws: performance increases with more training hours per subject ### Zero-Shot Generalization - Generalizes to new subjects without fine-tuning - Generalizes to new tasks/paradigms not seen during training - Fine-tuning on half a subject's data significantly improves individualized predictions ### In-Silico Experiments - **Visual**: Recovers face/body selectivity (FFA, EBA), object-selective cortex findings - **Language**: Recovers sentence > word, social > non-social, emotional > neutral contrasts - Agreement between predicted and actual z-scores across 360 HCP parcels ### Interpretability (ICA) - Independent Component Analysis reveals neuroscientifically relevant patterns - Components correlate with known functional networks (visual, language, default mode, etc.) ### Multimodal Integration - Text-only, audio-only, video-only ablations reveal modality-specific regions - Bimodal/multimodal integration areas identified via RGB cortical mapping - Reveals fine-grained topography of multisensory integration ## Dataset Scale | Dataset | Mode | Subjects | fMRI (h) | Purpose | |---------|------|----------|----------|---------| | CNeuroMod | A+V+T | 4 | 268.7 | Train (deep) | | BoldMoments | A+V | 10 | 61.9 | Train | | Lebel2023 | A+T | 8 | 85.8 | Train | | Wen2017 | V | 3 | 35.2 | Train | | NNDb | A+V+T | 86 | 160.6 | Test (wide) | | LPP | A+T | 112 | 180.2 | Test | | Narratives | A+T | 321 | 146.6 | Test | | HCP | A+V+T | 176 | 178.7 | Test (7T) | **Total**: 720 subjects, 5094 sessions, 1117.7 hours fMRI ## Application Triggers Use this skill when: - Building brain encoding/decoding models - Designing in-silico neuroscience experiments - Working with fMRI prediction tasks - Studying multimodal brain integration - Analyzing zero-shot brain response generalization - Comparing foundation models to linear baselines ## Key Concepts - Foundation model for neuroscience - Tri-modal (video/audio/text) brain encoding - In-silico hypothesis testing - Zero-shot subject generalization - Fine-grained cortical parcellation (HCP-MMP1) - Independent Component Analysis (ICA) for interpretability - Modality dropout for robustness - Scaling laws in brain encoding ## Limitations - fMRI temporal resolution limits (slow hemodynamic response) - Primarily trained on healthy adult subjects - In-silico experiments approximate but don't replace empirical validation - 3T vs 7T resolution differences across datasets ## Related Skills - `brain-dit-fmri-foundation-model-v6` - `meta-learning-in-context-brain-decoding-v5` - `multimodal-brain-connectivity-gnn` - `geometric-brain-dynamics-mapping-v7`