tribev2-brain-foundation-model

name	tribev2-brain-foundation-model
description	TRIBE v2 tri-modal foundation model for in-situ fMRI brain-to-image decoding with synthetic data augmentation. Addresses low-data regime challenges in brain decoding.
category	neuroscience
tags	["brain decoding","fMRI","foundation model","data augmentation","multimodal","encoding model","video","audio","language"]
version	1
arxiv_id	2606.06345v1
authors	Yohann Benchetrit, Marlène Careil, Simon Dahan, Hubert Banville, Stéphane d'Ascoli
published	2026-06-04T00:00:00.000Z
activation_keywords	["brain decoding","fMRI","TRIBE","foundation model","data augmentation","brain-to-image","encoding model"]

TRIBE v2: Tri-modal Foundation Model for Brain Decoding

Overview

TRIBE v2 is a large encoding model pretrained on more than 1000 hours of fMRI responses to video, audio, and language stimuli. It enables synthetic data augmentation for boosting brain-to-image decoding in low-data regimes.

arXiv: 2606.06345v1

Key Contribution: Addresses the fundamental limitation in brain decoding — availability of labeled neural data — by using pretrained foundation models to generate synthetic fMRI data for augmentation.

Core Methodology

1. Tri-modal Foundation Model Architecture

• Pretraining: 1000+ hours of fMRI responses to:
  • Video stimuli
  • Audio stimuli
  • Language stimuli
• Encoding model: Maps stimuli → fMRI responses
• Synthetic generation: Given new stimuli, generate predicted fMRI responses

2. Data Augmentation Strategy

Grid-based evaluation:

• Systematic grids showing how augmentation effectiveness varies with:
  • Amount of synthetic data added
  • Quality threshold for synthetic samples
  • Domain mismatch between pretrained model and target dataset
  • Training strategy (augment vs. pretrain)

3. Low-Data Regime Applications

When augmentation helps:

• Small fMRI datasets (< 50 subjects)
• Limited labeled stimuli
• Novel stimulus types not in pretraining
• Cross-subject generalization

When augmentation is less effective:

• Large datasets (> 500 subjects)
• Domain mismatch without adaptation
• Very different stimulus modalities

Technical Details

Synthetic Data Generation Pipeline

Input Stimuli → TRIBE v2 Encoder → Predicted fMRI → Quality Filter → Augmented Training Set

Quality filtering criteria:

• Prediction confidence threshold
• Activation pattern similarity to real data
• Region-of-interest consistency

Training Strategies

Option A: Direct Augmentation

# Mix real and synthetic fMRI data
augmented_dataset = real_fMRI + synthetic_fMRI[quality > threshold]
model.train(augmented_dataset)