brainjanus-unified-brain-vision-language

name	brainjanus-unified-brain-vision-language
description	BrainJanus: first unified brain model integrating brain, vision, and language via autoregressive Transformer. Unified Brain Tokenizer quantizes neural dynamics into discrete tokens aligned with visual/linguistic representations. Supports any-to-any generation (image-to-brain, text-to-brain, brain-to-image, brain-to-text). Use when researching unified brain encoding/decoding, brain-vision-language integration, discrete brain tokenization, or autoregressive brain modeling.
metadata	{"arxiv_id":"2606.30319","published":"2026-06-29","venue":"ICML 2026","authors":["Haitao Wu","Qirui Zhang","Zhouheng Yao","Shangquan Sun","Qihao Zheng","Mianxin Liu","Chi Zhang","Wanli Ouyang","Chunfeng Song","Changqing Zhang","Jiamin Wu"],"affiliations":["Tianjin University","Shanghai AI Lab","CUHK"],"tags":["brain-encoding","brain-decoding","unified-model","autoregressive","brain-tokenizer","fMRI","EEG","multimodal"],"github":"https://github.com/HaitaoWuTJU/BrainJanus"}

BrainJanus: Unified Brain-Vision-Language Autoregressive Model

First unified brain model integrating brain, vision, and language modalities within a single autoregressive framework. Replaces task-specific pipelines with any-to-any generation.

Core Innovation

Unlike prior methods treating encoding/decoding as isolated tasks with separate adapters, BrainJanus uses a single Transformer backbone for all four directions: image→brain, text→brain, brain→image, brain→text.

Key Components

1. Unified Brain Tokenizer (UBT)

• Quantizes continuous neural dynamics (fMRI/EEG) into discrete tokens
• Aligns brain tokens with vision and language tokens in shared Omni space
• Uses residual VQ-VAE style quantization with hierarchical codebooks
• Preserves biological topography (cortical spatial structure)

2. All-in-One Autoregressive Model

• Single Transformer backbone for next-token prediction across all modalities
• Task switching via prefix tokens (no separate heads needed)
• Supports zero-shot generalization to unseen tasks
• Joint multi-task learning promotes cross-modal knowledge transfer

Training Strategy

1. Stage 1: Pre-train brain tokenizer (reconstruction objective)
2. Stage 2: Joint training on all 4 tasks with unified vocabulary
3. Cross-modal alignment: brain tokens share embedding space with visual+text tokens

Key Results

• Surpasses task-specific models on encoding AND decoding benchmarks
• Zero-shot generalization: model trained on all tasks outperforms single-task models on held-out tasks
• Preserves interpretable cortical topography in generated fMRI
• Biological variability captured (inter-subject differences preserved)

Biological Motivation

Human brain is intrinsically multimodal:

• Visual stimuli elicit both visual AND linguistic/semantic responses
• Semantic representations tile the entire cortex (Huth et al., 2016)
• Prior methods only use unimodal CLIP alignment → insufficient exploitation of brain's multimodal semantics

Comparison with Prior Work

Method	Paradigm	Modalities	Unified?
MindEye2	brain→image (task-specific)	brain→vision only	No
BrainCLIP	brain↔CLIP alignment	unimodal	No
BrainFLORA	brain representation pretraining	brain only	No
BrainJanus	any-to-any	brain+vision+language	Yes

Pitfalls

• Discrete tokenization may lose fine-grained continuous information (trade-off for unification)
• Requires large paired brain-image-text datasets (NSD, GOD for training)
• Biological interpretability depends on tokenizer quality
• Zero-shot claims depend on task overlap in training distribution