neuralbench-unified-neuroai-benchmark

name	neuralbench-unified-neuroai-benchmark
description	NeuralBench unified benchmarking framework for NeuroAI models. Standardized evaluation across EEG/MEG/fMRI tasks with 36 tasks, 14 architectures, 94 datasets. Covers foundation model evaluation, task-specific baselines, cross-modal extension. Activation: neuralbench, neuroai benchmark, brain model evaluation, EEG benchmark, fMRI benchmark, MEG benchmark, NeuroAI evaluation.

NeuralBench: Unified Framework to Benchmark NeuroAI Models

A standardized, extensible benchmarking framework for AI models processing brain recordings, with EEG v1.0 covering 36 tasks, 14 architectures, and 94 datasets.

Metadata

• Source: arXiv:2605.08495
• Authors: Hubert Banville, Stéphane d'Ascoli, Simon Dahan, Jérémy Rapin, Marlène Careil, Yohann Benchetrit, Jarod Lévy, Saarang Panchavati, Antoine Ratouchniak, Mingfang (Lucy) Zhang, Elisa Cascardi, Katelyn Begany, Teon Brooks, Jean-Rémi King
• Published: 2026-05-08
• Pages: 31 pages, 9 figures

Core Methodology

Key Innovation

NeuralBench addresses the fragmentation in NeuroAI model evaluation by providing:

1. Standardized interface for accessing brain recording datasets
2. Unified task definitions across preprocessing, training, and evaluation
3. Cross-modal extensibility designed from the start for EEG, MEG, fMRI

NeuralBench-EEG v1.0

• 36 electroencephalography (EEG) tasks spanning cognitive decoding, clinical predictions, and more
• 14 deep learning architectures evaluated systematically
• 94 datasets accessed through a standardized interface
• Open-source framework inviting community expansion

Key Findings

1. Foundation models only marginally outperform task-specific models in current benchmarks
2. Many tasks remain highly challenging (cognitive decoding, clinical predictions) even for the best models
3. Preliminary extensions to MEG and fMRI demonstrate cross-modal capability

Technical Framework

Architecture

NeuralBench Framework
├── Dataset Interface (standardized access)
├── Task Definitions (preprocessing + training + evaluation)
├── Model Registry (architecture implementations)
├── Evaluation Pipeline (metrics, benchmarks)
└── Extension API (new tasks/datasets/modalities)

Benchmarking Dimensions

• Task diversity: Motor imagery, cognitive state decoding, clinical prediction, etc.
• Model diversity: CNNs, Transformers, foundation models, task-specific models
• Dataset scale: 94 datasets for EEG alone
• Metric standardization: Unified evaluation across all task-dataset-model combinations

Applications

• NeuroAI model comparison: Systematic benchmarking of brain-AI models
• Foundation model evaluation: Testing generalization across brain tasks
• Clinical AI validation: Assessing model performance on clinical prediction tasks
• Cross-modal research: Extending from EEG to MEG/fMRI within same framework
• Research direction identification: Finding gaps where models underperform

Implementation Guide

Prerequisites

• Python environment with PyTorch
• Access to EEG/MEG/fMRI datasets (framework provides standardized interface)

Steps

1. Install NeuralBench framework (open-source)
2. Select target modality (EEG, MEG, fMRI)
3. Choose tasks from available task registry
4. Register or implement model architecture
5. Run standardized evaluation pipeline
6. Compare results across models and tasks

Extending the Framework

• Add new tasks via task definition API
• Add new datasets via standardized dataset interface
• Add new models via model registry
• Add new modalities (MEG, fMRI) following existing pattern

Pitfalls

• Foundation models show only marginal gains over task-specific models currently
• Many clinically-relevant tasks remain unsolved even for best models
• Benchmark coverage is EEG-first; MEG/fMRI extensions are preliminary
• Preprocessing variations across studies can affect comparability

Related Skills

• ai-science-benchmarking
• open-ended-science-benchmark
• eeg-foundation-model-adapters
• tta-eeg-foundation-models
• brain-dit-fmri-foundation-model