visual-semantic-decoding-ecog

name	visual-semantic-decoding-ecog
description	Visual Semantic Decoding of Electrocorticography from Video Stimuli using End-to-End Deep Learning. Extract visual semantic categories from ECoG brain signals during video viewing using Transformer-based deep learning models. Use when working with ECoG neural decoding, brain-computer interfaces, or visual semantic classification from neural data.

Visual Semantic Decoding of Electrocorticography from Video Stimuli

End-to-end deep learning framework for decoding visual semantic categories from electrocorticography (ECoG) brain signals recorded during video stimulus presentation.

Key Features

• Transformer-based encoder for temporal sequence modeling
• High-gamma band (80-150 Hz) inputs as primary neural features
• 900ms post-stimulus window for optimal decoding performance
• Mixup augmentation for limited training data scenarios (<50 samples per category)
• Interpretable model analysis across spectral, temporal, and cortical dimensions

Brain Regions Contributing to Decoding

The framework identifies key cortical regions that contribute substantially to visual semantic decoding performance:

• Early visual cortex (V2-V4)
• Ventral stream visual cortex
• MT+ complex with neighboring visual areas
• Lateral temporal cortex

Implementation Guidelines

Data Preprocessing

1. Extract high-gamma band (80-150 Hz) neural activity from ECoG recordings
2. Apply 900ms temporal window starting from stimulus onset
3. Use mixup data augmentation when training samples are limited (<50 per category)

Model Architecture

• Use Transformer-based encoder architecture
• Input: Time-series neural activity from multiple electrodes
• Output: Visual semantic category probabilities
• No handcrafted features required - end-to-end learning

Training Considerations

• Works effectively with fewer than 50 training samples per visual category
• Mixup augmentation improves generalization with limited data
• High-gamma band provides most discriminative information

Evaluation Metrics

• Decoding accuracy across visual semantic categories
• Spectral analysis to identify frequency bands contributing to performance
• Temporal analysis to understand timing of neural responses
• Cortical contribution analysis to map brain regions involved in decoding

Applications

• Brain-Computer Interfaces (BCIs) for visual perception decoding
• Neural prosthetics for communication systems
• Cognitive neuroscience research on visual semantic processing
• Clinical applications for patients with communication disorders

Activation Keywords

• visual semantic decoding
• ECoG decoding
• brain-computer interface
• neural decoding
• electrocorticography
• visual category decoding
• Transformer neural decoding
• high-gamma decoding

References

• arXiv:2607.18923v1 - "Visual Semantic Decoding of Electrocorticography from Video Stimuli using End-to-End Deep Learning"
• Natural Scenes Dataset (NSD) for fMRI validation
• VEDB (Visual Experience Dataset) for egocentric vision studies

Best Practices

1. Start with high-gamma band: Focus on 80-150 Hz frequency range for optimal results
2. Use appropriate temporal window: 900ms post-stimulus provides best decoding performance
3. Apply data augmentation: Use mixup when training data is limited
4. Validate across dimensions: Analyze spectral, temporal, and cortical contributions for interpretability
5. Compare with established neuroscience: Ensure results align with known visual processing pathways