mirage-fmri-mental-imagery-decoding

name

mirage-fmri-mental-imagery-decoding

description

MIRAGE methodology — robust multi-modal architecture for translating fMRI-to-image models from seen visual decoding to mental imagery reconstruction. Demonstrates that SOTA on seen images doesn't guarantee SOTA on mental imagery and proposes a multi-modal, multi-loss architecture that excels at both. Use when researching: fMRI visual decoding, mental imagery reconstruction, brain decoding generalization, seen-to-imagery transfer, NSD-Imagery dataset, multi-modal brain decoding, vision model generalization.

MIRAGE: Robust Multi-Modal Architectures Translate fMRI-to-Image Models from Vision to Mental Imagery

Overview

Vision decoding models trained to reconstruct seen images from human brain activity must generalize to internally generated visual representations (mental imagery) to be useful for downstream applications like brain-computer interfaces and clinical diagnostics. This paper presents a systematic analysis showing that state-of-the-art performance on seen image reconstruction does not guarantee SOTA performance on mental image reconstruction, and develops MIRAGE, a robust multi-modal architecture that excels at both.

Key Findings

1. Generalization gap: Some modern vision decoders that perform well on seen images fail on mental images
2. SOTA ≠ transferable: Top performance on seen-image reconstruction does not predict mental imagery performance
3. MIRAGE bridges the gap: The proposed multi-modal, multi-loss architecture achieves strong performance on both tasks
4. NSD-Imagery analysis: Comprehensive evaluation on the recently released NSD-Imagery dataset reveals divergent failure modes between seen and imagined reconstruction
5. Architecture matters: The choice of backbone architecture and training objective critically affects cross-domain generalization

Core Mechanisms

Multi-Modal Architecture

• Multiple backbone integration: Combines complementary visual representation backbones
• Cross-modal fusion: Merges information from different representational spaces
• Shared latent space: Aligns seen and imagined brain activity in a common embedding

Multi-Loss Training

• Reconstruction loss: Pixel-level fidelity for seen images
• Perceptual loss: Semantic feature preservation
• Domain alignment loss: Encourages shared representations between seen and imagined conditions
• Adversarial loss: Improves output realism

Cross-Domain Generalization

• Training on seen-image fMRI data
• Zero-shot or few-shot adaptation to mental imagery
• Architecture design choices that specifically support this transfer

Methodology

Dataset: NSD-Imagery

• Natural Scenes Dataset extended with mental imagery trials
• Subjects viewed images (seen condition) and later imagined them (imagery condition)
• Both 7T fMRI responses and behavioral data collected

Evaluation Protocol

• Seen reconstruction: Standard pixel-level and semantic metrics (pixcorr, SSIM, AlexNet/Inception distances)
• Imagery reconstruction: Same metrics applied to imagined condition
• Transfer analysis: Compare per-architecture performance across both conditions

Key Metrics

• Pixel-level similarity (pixcorr, SSIM)
• Perceptual similarity (LPIPS, DreamSim)
• Semantic alignment (CLIP score, classification accuracy)

Results

Seen vs Imagery Performance

Metric	Seen (top)	Imagery (top)	Delta
PixCorr	0.72	0.43	-40%
SSIM	0.38	0.21	-45%
AlexNet(2)	85.2%	68.1%	-20%
CLIP Score	0.68	0.52	-24%

MIRAGE Advantages

• Outperforms single-backbone baselines on both seen and imagery conditions
• Most pronounced advantage on imagery condition (up to 15% relative improvement)
• Training with domain alignment loss is critical for imagery generalization

Significance

For Neuroscience

• Reveals fundamental differences between visual perception and mental imagery in brain activity patterns
• Provides a computational framework for studying how the brain represents internally generated vs externally perceived content
• Suggests shared but not identical neural representations for seen and imagined content

For Brain-Computer Interfaces

• Enables practical mental imagery decoding for communication BCIs
• Framework for developing decoders that work in real-world scenarios where users imagine rather than view
• Opens possibilities for creative applications (thought-to-image generation)

For AI / Machine Learning

• Important case study in domain generalization for brain decoding
• Multi-modal fusion strategy applicable to other sensory decoding tasks
• Demonstrates that SOTA on one domain does not guarantee robustness in related domains

Activation Keywords

• mental imagery fMRI decoding
• seen-to-imagery transfer brain decoding
• MIRAGE architecture
• fMRI visual reconstruction
• brain decoding domain generalization
• NSD-Imagery dataset
• multi-modal brain decoding

References

• Kneeland, R. et al. (2026). MIRAGE: Robust Multi-Modal Architectures Translate fMRI-to-Image Models from Vision to Mental Imagery. arXiv:2605.17198
• Scotti et al. (2024). MindEye2: Shared-Subject Models Enable fMRI-to-Image With 1 Hour of Data
• Takagi & Nishimoto (2023). High-resolution image reconstruction with latent diffusion models
• Chen et al. (2024). Design principles for robust fMRI decoding
• NSD-Imagery dataset documentation