| name | mind-omni-brain-vision-language-unified |
| description | Mind-Omni unified multi-task framework for Brain-Vision-Language modeling via discrete diffusion |
| version | 1.0.0 |
| author | Yizhuo Lu et al. (arXiv:2605.29591) |
| created | 2026-06-01T00:00:00.000Z |
| arxiv_id | 2605.29591 |
| paper_title | Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion |
| categories | ["neuroscience","brain-computer-interface","multimodal-learning","foundation-model"] |
| tags | ["brain-vision-language","discrete-diffusion","multi-task","BCI","neural-encoding","neural-decoding"] |
| activation_keywords | ["mind-omni","brain vision language","unified framework","discrete diffusion BCI","multi-task brain model"] |
Mind-Omni: Unified Brain-Vision-Language Framework
Overview
Mind-Omni is the first versatile framework that unifies seven distinct encoding and decoding tasks through a discrete diffusion paradigm. It addresses the limitation of specialized single-task models in Brain-Computer Interfaces (BCIs) by providing a unified approach that captures inter-task synergies.
Key Innovation: Uses discrete diffusion to bridge brain signals (fMRI/EEG), visual content, and language in a single coherent model.
Core Architecture
1. Novel Brain Tokenizer
- Converts neural signals into discrete tokens
- Enables seamless integration with vision-language models
- Preserves spatial-temporal brain activity patterns
2. Discrete Diffusion Model
- Unified generative framework for all tasks
- Joint modeling of brain-vision-language representations
- Enables bidirectional transformations
3. Seven Unified Tasks
| Task Type | Description |
|---|
| Brain Encoding | Encode visual stimuli → brain activity |
| Brain Decoding | Decode brain signals → visual reconstruction |
| VQA from Brain | Answer questions directly from fMRI |
| Brain Captioning | Generate natural language descriptions |
| Visual Retrieval | Retrieve images matching brain patterns |
| Cross-Modal Generation | Generate images from brain signals |
| Neural Representation Analysis | Understand brain encoding structure |
Technical Details
Brain Tokenizer Design
Input: fMRI voxel patterns / EEG time series
Processing:
1. Spatial-temporal feature extraction
2. Vector quantization (VQ-VAE style)
3. Discrete token assignment
Output: Brain token sequence [b₁, b₂, ..., bₙ]
Discrete Diffusion Process
Forward diffusion: Add noise to brain tokens
Reverse diffusion: Generate brain/visual/language content
Joint training: Learn bidirectional mappings
Key Advantages
- Versatility: Single model for 7+ tasks
- : Shared representations improve all tasks
