stsbench-dorsal-stream-visual-cortex

name	stsbench-dorsal-stream-visual-cortex
description	STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex - Skill for understanding and applying the methods from arXiv:2607.15631

STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex

Overview

This skill provides guidance on working with STSBench, a large-scale dataset of single neuron recordings from the superior temporal sulcus (STS) in primates. The dataset enables modeling of neuronal activity in the dorsal stream, which is responsible for spatial relations and motion processing, addressing a key gap in visual neuroscience where ventral stream models are well-developed but dorsal stream models lack sufficient data.

Key Contributions

• Large-scale neural recordings: Over 2,000 neurons from STS, ~50x larger than previous dorsal stream datasets
• Naturalistic stimuli: Thousands of unique natural videos viewed by Rhesus macaques
• Benchmarking framework: Enables encoding and decoding models of dorsal stream neuronal responses
• Reconstruction capability: Supports visual input reconstruction from neural activity

Core Concepts

Visual Stream Organization

• Ventral stream: Object recognition ("what" pathway)
• Dorsal stream: Spatial relations and motion ("where/how" pathway) - Focus of STSBench

Dataset Characteristics

• Recording site: Superior Temporal Sulcus (STS) - key dorsal stream area
• Subject: Rhesus macaques
• Stimuli: Thousands of unique natural videos
• Scale: ~2,000+ single neurons
• Data type: Electrophysiological spike trains

Analytical Framework

1. Encoding models: Predict neural responses from visual stimuli
2. Decoding models: Reconstruct stimuli from neural activity
3. Benchmarking: Compare model performance against noise ceiling
4. Comparison to ventral stream: Leverage insights from CNN models of ventral stream

Technical Approach

Data Structure

• Spike times or binned spike counts for each neuron
• Time-aligned to video stimulus presentation
• Metadata including stimulus identity, timing, behavioral variables

Encoding Modeling Steps

1. Stimulus feature extraction: Extract features from video frames (e.g., using CNNs, hand-crafted features, or raw pixels)
2. Feature-neural mapping: Learn mapping from stimulus features to neural responses
3. Model validation: Predict responses on held-out stimuli
4. Performance evaluation: Compute explained variance, correlation with actual responses

Decoding/Reconstruction

1. Neural feature extraction: Extract features from neural population activity
2. Neural-stimulus mapping: Learn mapping from neural features to stimulus representation
3. Stimulus reconstruction: Generate or select stimuli that best match neural activity
4. Reconstruction quality: Measure similarity between original and reconstructed stimuli

Implementation Guidelines

Data Access

The dataset is available through the associated paper's supplementary materials or repositories mentioned in the arXiv paper.

Preprocessing Steps

1. Spike sorting validation: Ensure quality of single-unit isolation
2. Response alignment: Align neural responses to stimulus onset
3. Baseline subtraction: Remove spontaneous activity if needed
4. Binning: Convert spike trains to binned activity (e.g., 10-50ms bins)
5. Z-scoring: Normalize responses across trials or time windows

Feature Extraction for Encoding

Visual features (from video frames):

• Raw pixels (downsampled)
• Hand-crafted features (edges, motion energy, Fourier spectrum)
• CNN features (from networks pretrained on ImageNet or similar)
• Biological vision model outputs (V1-like filters, HMAX)
• Temporal features (optical flow, motion energy)

Temporal modeling:

• Time-lagged features to account for neural response delays
• Basis function expansion (e.g., raised cosine bases)
• Temporal convolutional filters
• Recurrent connections for history dependence

Model Types

Linear models:

• Linear regression (ridge/Lasso regularization)
• Generalized Linear Models (GLMs) with appropriate nonlinearity (e.g., exponential, softplus)
• Sparse coding approaches

Nonlinear models:

• Kernel methods (RBF, polynomial)
• Neural networks (shallow/deep)
• Random forests, gradient boosting
• Hybrid approaches combining linear-nonlinear cascades

Evaluation Framework

1. Cross-validation: Use appropriate scheme for time series data
2. Metrics:
   • Pearson correlation (predicted vs actual)
   • Explained variance (R²)
   • Mean squared error
   • Reliability-adjusted metrics (split-half reliability)
3. Statistical significance: Compare against shuffled data distributions
4. Model comparison: Use information criteria (AIC/BIC) or cross-validated likelihood

Advanced Applications

• Stimulus reconstruction: Use decoding models to generate/identify stimuli
• Population decoding: Analyze information content in neural ensembles
• Dimensionality reduction: Identify shared variance across neurons
• Casual analysis: Perturbation modeling (if intervention data available)
• Cross-area comparisons: Compare STS with other visual areas

Best Practices

Experimental Considerations

• Account for eye movements and fixation patterns
• Consider behavioral state and attention effects
• Control for low-level stimulus confounds
• Verify stimulus presentation timing accuracy

Modeling Best Practices

• Always compare performance to noise ceiling
• Use cross-validation to prevent overfitting
• Regularize appropriately given high dimensionality
• Test multiple feature sets and model types
• Validate findings with multiple random seeds/splits

Interpretation Guidelines

• Relate model properties to known neural properties
• Consider alternative explanations for model performance
• Link findings to dorsal stream functions (space, motion, navigation)
• Relate to human neuroimaging and lesion literature

Reproducibility

• Share code and preprocessing steps
• Document random seeds and train/test splits
• Report confidence intervals or error bars
• Compare against multiple baseline models

Validation Strategies

1. Known response properties: Check if model reproduces known tuning (e.g., direction selectivity)
2. Generalization to novel stimuli: Test on completely unseen natural videos
3. Cross-area generalization: Test if models trained on STS predict other areas
4. Manipulation predictions: If available, test predictions under stimulus manipulations
5. Comparison to ventral stream models: Leverage insights from ventral stream CNN success

Extensions and Variations

• Multi-area modeling: Joint modeling of STS with other visual areas
• State-dependent models: Separate models for different behavioral states
• Plasticity models: How responses change with stimulus statistics
• Developmental trajectories: Changes across age or experience
• Cross-species comparisons: Human neuroimaging homologs

Common Pitfalls and Solutions

• Overfitting to noise: Use stringent regularization and validation
• Stimulus confounds: Control for low-level features that co-vary with target variables
• Non-stationarities: Account for drift in neural responses over time
• Limited stimulus diversity: Ensure sufficient variation in natural video statistics
• Evaluation bias: Use independent test sets, avoid peeking at test data

Connection to Broader Fields

• Computational neuroscience: Bridging neural data with quantitative models
• Machine learning: Applying advanced ML techniques to neural data
• Systems neuroscience: Understanding hierarchical visual processing
• Brain-inspired AI: Developing vision models informed by dorsal stream processing
• Neural engineering: Potential applications in BMI or neural prosthetics

Activation Keywords

stsbench, dorsal stream, visual cortex, neuronal recording, primate vision, STS, superior temporal sulcus, encoding model, neural decoding, natural vision, computational neuroscience, neural encoding, visual decoding, population neuroscience, systems neuroscience

Related Skills

• neuroscience-spiking-afe-20260714
• transient-synapse-activity-regen
• visual-imagery-decoding-fmri
• visual-place-recognition-rate-encoded-snn-stdp
• brain-inspired-attention-mechanisms
• sensory-modulation-network

References

• Primary paper: arXiv:2607.15631 [q-bio.NC] - STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex
• Venue: Advances in Neural Information Processing Systems 38 (NeurIPS 2025) Datasets and Benchmarks Track