tribe-v2-multimodal-brain-foundation

name	tribe-v2-multimodal-brain-foundation
description	TRIBE v2 tri-modal foundation model methodology for in-situ fMRI brain-to-image decoding with synthetic data augmentation. Pretrained on 1000+ hours of video/audio/language fMRI responses, enables 68% Top-10 image-retrieval improvement and zero-shot decoding in low-data regimes.

Context

Brain decoding is fundamentally limited by labeled neural data scarcity. TRIBE v2 addresses this by augmenting small fMRI datasets with synthetic data from a pretrained multimodal encoding model — achieving up to 68% improvement in image-retrieval accuracy compared to real-data-only decoders.

Key Innovation: Zero-shot brain-to-image decoding works in some settings when trained exclusively on synthetic fMRI, suggesting the pretrained encoder captures sufficiently general neural response patterns.

arXiv: 2606.06345v1 (2026-06-04) Categories: cs.AI, cs.LG, q-bio.NC

Core Methodology

1. Pretrained Multimodal Encoder: TRIBE v2 is pretrained on >1000 hours of fMRI responses to video, audio, and language stimuli across multiple subjects and datasets.

2. Synthetic Data Generation Pipeline:

   • Input: Stimulus images (natural scenes dataset)
   • Process: Pass through pretrained encoder → generate synthetic fMRI responses
   • Output: Augmented fMRI dataset combining real + synthetic responses

3. Grid Search for Optimal Augmentation Ratio:

   • Systematic evaluation: vary synthetic data proportion (e.g., 0%, 50%, 100%, 200% of real data size)
   • Find optimal ratio that maximizes Top-10 image-retrieval accuracy
   • Different datasets require different augmentation ratios (data source dependency)

4. Decoder Training:

   • Train image decoders on augmented dataset (real + synthetic fMRI)
   • Evaluation: Top-10 image-retrieval accuracy on held-out real fMRI
   • Comparison: real-only vs. augmented vs. pure-synthetic training

5. Zero-Shot Decoding Validation:

   • Train decoder exclusively on synthetic fMRI (no real subject data)
   • Test on real fMRI from unseen subjects
   • Above-chance performance indicates encoder captures generalizable neural patterns

Implementation Steps

Step 1: Load Pretrained Encoder

# TRIBE v2 pretrained on video/audio/language fMRI
encoder = load_tribe_v2_encoder(checkpoint_path)
encoder.eval()  # Freeze for synthetic generation

Step 2: Generate Synthetic fMRI Responses

def generate_synthetic_fmri(encoder, stimuli_images, target_subject_roi):
    """
    Generate synthetic fMRI responses for given stimuli.
    
    Args:
        encoder: Pretrained TRIBE v2 model
        stimuli_images: List of image stimuli
        target_subject_roi: ROI mask for target subject
    
    Returns:
        synthetic_fmri: Generated fMRI responses
    """
    with torch.no_grad():
        synthetic_fmri = encoder.encode_stimuli(stimuli_images, roi=target_subject_roi)
    return synthetic_fmri

Step 3: Augment Real Dataset

def augment_dataset(real_fmri, synthetic_fmri, augmentation_ratio=1.0):
    """
    Combine real and synthetic fMRI data.
    
    Args:
        real_fmri: Measured fMRI responses (N samples)
        synthetic_fmri: Generated responses
        augmentation_ratio: Proportion of synthetic relative to real
    
    Returns:
        augmented: Combined dataset
    """
    n_synthetic = int(len(real_fmri) * augmentation_ratio)
    augmented = torch.cat([real_fmri, synthetic_fmri[:n_synthetic]], dim=0)
    return augmented

Step 4: Train Image Decoder

# Image decoder: fMRI → image retrieval
decoder = BrainToImageDecoder(fmri_dim=augmented.shape[1], image_feature_dim=512)
optimizer = torch.optim.Adam(decoder.parameters(), lr=1e-4)

for epoch in range(num_epochs):
    predictions = decoder(augmented_fmri)
    loss = retrieval_loss(predictions, ground_truth_images)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Step 5: Evaluate Retrieval Performance

def evaluate_retrieval(decoder, test_fmri, image_database, k=10):
    """
    Compute Top-k image-retrieval accuracy.
    
    Args:
        decoder: Trained decoder
        test_fmri: Real fMRI test set
        image_database: All candidate images
        k: Number of top candidates to retrieve
    
    Returns:
        accuracy: Top-k retrieval accuracy
    """
    predicted_features = decoder(test_fmri)
    retrieved_indices = retrieve_top_k(predicted_features, image_database, k)
    accuracy = compute_top_k_accuracy(retrieved_indices, ground_truth_indices)
    return accuracy

Key Results

• Natural Scenes Dataset (7T fMRI): 68% improvement in Top-10 accuracy with optimal augmentation
• BOLD5000 (3T fMRI): Similar gains with dataset-specific augmentation ratios
• Zero-shot: Pure-synthetic training achieves above-chance performance in some settings
• Data Efficiency: Small real datasets (few subjects) benefit most from augmentation

Pitfalls

1. Augmentation Ratio Dataset Dependency: Different fMRI datasets (7T vs 3T, different ROIs) require different optimal synthetic/real ratios. Must grid-search for each new dataset.

2. Subject-Specific ROI Alignment: Synthetic generation assumes encoder's ROI representation aligns with target subject's ROI. Misalignment can degrade augmentation quality.

3. Encoder Pretraining Domain: TRIBE v2 pretrained on video/audio/language — image decoding may be suboptimal if visual domain wasn't sufficiently covered during pretraining.

4. Synthetic Distribution Shift: Generated fMRI may not match real fMRI distribution exactly, leading to domain shift in decoder training. Consider distribution alignment techniques.

5. Zero-Shot Limits: Above-chance zero-shot doesn't mean competitive performance — expect significant gap from fully-supervised decoders.

Verification

• Top-10 retrieval accuracy on NSD and BOLD5000 test sets
• Compare augmented vs. real-only training
• Grid-search curves showing accuracy vs. augmentation ratio
• Zero-shot retrieval baseline (pure synthetic training)

Activation

brain-to-image decoding, TRIBE v2, multimodal foundation model, fMRI augmentation, synthetic neural data, zero-shot decoding, image retrieval, natural scenes dataset, BOLD5000, data efficiency, brain foundation model, neural encoding model, video/audio/language fMRI