| name | neuralbench-unified-neuroai-benchmark |
| description | NeuralBench unified benchmarking framework for NeuroAI models. Standardized evaluation across EEG/MEG/fMRI tasks with 36 tasks, 14 architectures, 94 datasets. Covers foundation model evaluation, task-specific baselines, cross-modal extension. Activation: neuralbench, neuroai benchmark, brain model evaluation, EEG benchmark, fMRI benchmark, MEG benchmark, NeuroAI evaluation. |
NeuralBench: Unified Framework to Benchmark NeuroAI Models
A standardized, extensible benchmarking framework for AI models processing brain recordings, with EEG v1.0 covering 36 tasks, 14 architectures, and 94 datasets.
Metadata
- Source: arXiv:2605.08495
- Authors: Hubert Banville, Stéphane d'Ascoli, Simon Dahan, Jérémy Rapin, Marlène Careil, Yohann Benchetrit, Jarod Lévy, Saarang Panchavati, Antoine Ratouchniak, Mingfang (Lucy) Zhang, Elisa Cascardi, Katelyn Begany, Teon Brooks, Jean-Rémi King
- Published: 2026-05-08
- Pages: 31 pages, 9 figures
Core Methodology
Key Innovation
NeuralBench addresses the fragmentation in NeuroAI model evaluation by providing:
- Standardized interface for accessing brain recording datasets
- Unified task definitions across preprocessing, training, and evaluation
- Cross-modal extensibility designed from the start for EEG, MEG, fMRI
NeuralBench-EEG v1.0
- 36 electroencephalography (EEG) tasks spanning cognitive decoding, clinical predictions, and more
- 14 deep learning architectures evaluated systematically
- 94 datasets accessed through a standardized interface
- Open-source framework inviting community expansion
Key Findings
- Foundation models only marginally outperform task-specific models in current benchmarks
- Many tasks remain highly challenging (cognitive decoding, clinical predictions) even for the best models
- Preliminary extensions to MEG and fMRI demonstrate cross-modal capability
Technical Framework
Architecture
NeuralBench Framework
├── Dataset Interface (standardized access)
├── Task Definitions (preprocessing + training + evaluation)
├── Model Registry (architecture implementations)
├── Evaluation Pipeline (metrics, benchmarks)
└── Extension API (new tasks/datasets/modalities)
Benchmarking Dimensions
- Task diversity: Motor imagery, cognitive state decoding, clinical prediction, etc.
- Model diversity: CNNs, Transformers, foundation models, task-specific models
- Dataset scale: 94 datasets for EEG alone
- Metric standardization: Unified evaluation across all task-dataset-model combinations
Applications
- NeuroAI model comparison: Systematic benchmarking of brain-AI models
- Foundation model evaluation: Testing generalization across brain tasks
- Clinical AI validation: Assessing model performance on clinical prediction tasks
- Cross-modal research: Extending from EEG to MEG/fMRI within same framework
- Research direction identification: Finding gaps where models underperform
Implementation Guide
Prerequisites
- Python environment with PyTorch
- Access to EEG/MEG/fMRI datasets (framework provides standardized interface)
Steps
- Install NeuralBench framework (open-source)
- Select target modality (EEG, MEG, fMRI)
- Choose tasks from available task registry
- Register or implement model architecture
- Run standardized evaluation pipeline
- Compare results across models and tasks
Extending the Framework
- Add new tasks via task definition API
- Add new datasets via standardized dataset interface
- Add new models via model registry
- Add new modalities (MEG, fMRI) following existing pattern
Pitfalls
- Foundation models show only marginal gains over task-specific models currently
- Many clinically-relevant tasks remain unsolved even for best models
- Benchmark coverage is EEG-first; MEG/fMRI extensions are preliminary
- Preprocessing variations across studies can affect comparability
Related Skills
- ai-science-benchmarking
- open-ended-science-benchmark
- eeg-foundation-model-adapters
- tta-eeg-foundation-models
- brain-dit-fmri-foundation-model
