spikeprophecy-benchmark

name	spikeprophecy-benchmark
description	SpikeProphecy: First large-scale benchmark for causal, autoregressive neural population spike-count forecasting. Introduces population metric decomposition (temporal fidelity, spatial pattern accuracy, magnitude-invariant alignment) on 105 Neuropixels sessions (~89,800 neurons). arXiv:2605.12992
tags	["neural-population","forecasting","benchmark","neuropixels","spike-count","evaluation","ssm","transformer"]
related_skills	["realm-lfp-retrospective-decoding","neural-population-dynamics"]

SpikeProphecy: Large-Scale Benchmark for Autoregressive Neural Population Forecasting

Paper: arXiv:2605.12992v1 (May 13, 2026) Authors: John R. Minnick, Jinghui Geng, Kamran Hussain, Jesus Gonzalez-Ferrer, Ash Robbins, Mohammed A. Mostajo-Radji, David Haussler, Jason K. Eshraghian, Mircea Teodorescu (UC Santa Cruz)

Problem

Neural population models (predicting joint firing of many simultaneously recorded neurons) are evaluated by a single aggregate Pearson correlation r, which:

• Masks critical structure (brain region differences, neuron subpopulation failures)
• Collapses temporal dynamics capture vs. spatial pattern fidelity
• Hides the distinction between population-level vs. individual-neuron accuracy

No established benchmark existed for spike-count forecasting at scale on real electrophysiology data.

SpikeProphecy Benchmark

Scale

• 105 Neuropixels sessions from two public datasets:
  • Steinmetz 2019: 75 sessions, multiple brain regions
  • IBL Repeated Site: 30 sessions, repeated recording sites
• ~89,800 neurons total
• First large-scale autoregressive spike-count forecasting benchmark

Population Metric Decomposition (Core Contribution)

Instead of a single aggregate Pearson r, SpikeProphecy decomposes evaluation into three orthogonal axes:

1. pop_rate_r (Temporal Fidelity)

   • How well does the model capture population-level firing rate dynamics over time?
   • Measures temporal pattern matching across the entire population
   • Example: r_pop = 0.76 (good temporal capture)

2. spatial_r (Spatial Pattern Accuracy)

   • How well does the model capture which specific neurons are firing?
   • Measures individual neuron identity preservation
   • Example: r_spatial = 0.55 (moderate spatial capture)

3. cosine_sim (Magnitude-Invariant Alignment)

   • Directional alignment of population activity vectors, independent of magnitude
   • Captures whether the model gets the "shape" of population activity right

Why Decomposition Matters

An aggregate r = 0.50 sounds mediocre, but decomposition reveals:

• Temporal population dynamics: r_pop = 0.76 (well captured)
• Individual neuron identity: r_spatial = 0.55 (moderately captured)

This guides targeted model improvement.

Architecture Baselines Tested

Seven models across four structural families:

State Space Models (SSMs) - 4 variants

1. S4 (Structured State Space)
2. Mamba (Selective SSM)
3. Griffin (Gated SSM)
4. RWKV (Receptance Weighted Key Value, non-diagonal SSM)

Other architectures

5. Transformer (standard attention-based)
6. LSTM (classic recurrent)
7. Spiking Network (event-driven SNN)

Key Findings

1. Brain-Region Predictability Ranking

• A consistent hierarchy of brain region predictability emerges across ALL seven baselines
• Survives ANCOVA correction for firing-statistics covariates (region ΔR² = 0.018)
• Some regions are inherently more predictable than others, independent of model choice

2. Sub-Poisson Evaluation Floor

• Rigorous metrics combined with genuine biophysical constraints reveal a "floor"
• Regular spike trains have inherent unpredictability below Poisson level
• This is a fundamental biophysical limit, not a model limitation

3. KL-on-Output-Rates Distillation (Negative Result)

• ANN→SNN transfer via KL divergence on output rates does NOT work well
• In this Poisson count domain, distillation fails to preserve distributional properties
• Important negative result for the community

4. Linear vs. Deep Model Hierarchy

• Decomposition exposes distinct failure modes between linear and deep models
• Single-scalar reporting misses these failure mode differences entirely

Why This Matters

For BCI Development

• 50-100ms look-ahead predictions compensate for sensing/processing delays
• Essential for closed-loop BCIs
• Enables "digital twin" neural simulators for algorithm development without animal experiments

For Neural Science

• Provides standardized evaluation protocol for neural population models
• Enables fair comparison across architectures
• Reveals fundamental structure in neural population predictability

Application Protocol

When to Use SpikeProphecy

• Evaluating neural population forecasting models
• Comparing architectures for spike-count prediction
• Building closed-loop BCI systems requiring look-ahead predictions
• Developing in silico neural population simulators
• Studying brain-region-specific neural dynamics predictability

Metric Selection Guide

For temporal pattern analysis:     → pop_rate_r
For neuron-specific prediction:    → spatial_r
For population geometry:           → cosine_sim
For comprehensive evaluation:      → all three metrics
Avoid:                             → single aggregate Pearson r alone

Architecture Selection Guide

For best overall performance:      → SSM family (Mamba, Griffin)
For interpretability:              → S4 (structured state space)
For event-driven efficiency:       → Spiking network (with caveats)
For baseline comparison:           → LSTM, Transformer

Implementation Pattern

class SpikeProphecyEvaluator:
    """Population metric decomposition for spike-count forecasting."""
    
    def __init__(self, n_neurons):
        self.n_neurons = n_neurons
    
    def pop_rate_r(self, predicted, actual):
        """Temporal fidelity: population-level firing rate dynamics."""
        # Sum across neurons at each timestep
        # Pearson r between predicted and actual population rates
        pred_rates = predicted.sum(axis=1)  # sum across neurons
        actual_rates = actual.sum(axis=1)
        return pearsonr(pred_rates, actual_rates)
    
    def spatial_r(self, predicted, actual):
        """Spatial pattern accuracy: individual neuron identity."""
        # Pearson r per neuron, then average
        neuron_rs = [pearsonr(predicted[:, i], actual[:, i]) 
                     for i in range(self.n_neurons)]
        return np.mean(neuron_rs)
    
    def cosine_sim(self, predicted, actual):
        """Magnitude-invariant alignment: population activity shape."""
        # Cosine similarity between population activity vectors
        return cosine_similarity(predicted, actual)

Data Access

• Steinmetz 2019: 75 sessions, publicly available
• IBL Repeated Site: 30 sessions, International Brain Laboratory
• Total: ~89,800 neurons across 105 sessions

Activation Keywords

• spike forecasting, neural population model, Neuropixels
• population metric decomposition, spike-count prediction
• autoregressive neural forecasting, closed-loop BCI
• brain region predictability, neural digital twin
• SSM for neural data, Mamba neural population
• benchmark neural population, spike prophecy

Pitfalls

• Don't use aggregate Pearson r alone — it masks critical structure
• Account for sub-Poisson floor — some unpredictability is biophysical, not model failure
• KL distillation fails on output rates in Poisson count domain
• Match temporal context when comparing models (same look-ahead window)
• Fire-rate covariates matter — region predictability differences persist after ANCOVA correction

Related Work

• LFADS: Latent Factor Analysis via Dynamical Systems
• NDT/NDT2/NDT3: Neural Decoding Transformers
• CEBRA: Contrastive Embedding for Brain Activity
• S4/Mamba/Griffin: State space model families
• REALM (arXiv:2605.14867): LFP-based decoding (complementary modality)

Open Questions

• What determines brain-region predictability hierarchy?
• Can models surpass the sub-Poisson evaluation floor?
• How does forecast quality scale with session count and neuron count?
• Can the metric decomposition guide architecture search?
• What is the minimum look-ahead needed for effective closed-loop BCI?