visual-semantic-decoding-ecog-video-stimuli

name	visual-semantic-decoding-ecog-video-stimuli
title	Visual Semantic Decoding of Electrocorticography from Video Stimuli using End-to-End Deep Learning
description	End-to-end deep learning framework for visual semantic decoding from ECoG, demonstrating promising performance without handcrafted features while maintaining interpretability.
arxiv_id	2607.18923
date	2026-07-21T00:00:00.000Z
authors	["Stella Ho","Joel Villalobos","Joseph West","Jingyang Liu","Weijie Qi","Haruhiko Kishima","Ryohei Fukuma","Takufumi Yanagisawa","Sam E. John","David B. Grayden"]
categories	["cs.LG","q-bio.NC"]
trigger_words	["visual semantic decoding","electrocorticography","ECoG","end-to-end deep learning","video stimuli","Transformer encoder","high-gamma"]

Visual Semantic Decoding of Electrocorticography from Video Stimuli using End-to-End Deep Learning

Overview

This study examines the feasibility of visual semantic decoding using an end-to-end deep learning framework with electrocorticography (ECoG) to predict visual categories from video stimuli using time-series neural inputs.

Key Contributions

1. End-to-End Framework Design

• No Handcrafted Features: Demonstrates that end-to-end deep learning can yield promising decoding performance without manual feature engineering
• Data Efficiency: Achieves good performance with fewer than 50 training samples per visual category
• Multiple Architecture Evaluation: Evaluates various deep learning approaches and neural network architectures

2. Optimal Configuration

• Mixup Augmentation: Uses mixup data augmentation to improve generalization
• Transformer Encoder: Employs a Transformer-based encoder for sequence modeling
• High-Gamma Band: Focuses on high-gamma frequency band (80-150 Hz) inputs
• Temporal Window: Uses 900 ms post-stimulus temporal window

3. Interpretability Analysis

• Spectral Dimension: Analyzes discriminative information across frequency bands
• Temporal Dimension: Examines temporal dynamics of decoding performance
• Cortical Dimension: Identifies key brain regions contributing to decoding

4. Key Brain Regions Identified

• Early Visual Cortex: V2-V4 areas contribute substantially
• Ventral Stream: Ventral stream visual cortex involvement
• MT+ Complex: MT+ complex with neighboring visual areas
• Lateral Temporal Cortex: Significant contribution from lateral temporal cortex

Applications

Brain-Computer Interfaces (BCIs)

• Visual Category Decoding: Enables real-time decoding of perceived visual categories
• Dynamic Stimuli Processing: Handles complex, dynamic video stimuli rather than static images
• Clinical Applications: Potential applications for patients with communication disorders

Neuroscience Research

• Neural Representation Mapping: Maps neural activity to semantic visual categories