eeg-fmri-spatiotemporal-neural-frames

title	EEG-conditioned fMRI Reconstruction for High-Resolution Brain Dynamics
name	eeg-fmri-spatiotemporal-neural-frames
category	ai_collection
description	EEG-conditioned framework for reconstructing dynamic fMRI as continuous neural sequences with high spatial fidelity and temporal coherence at cortical-vertex level. Incorporates null-space intermediate-frame reconstruction for handling sampling irregularities.
arXiv_id	2603.24176
author	Wanying Qu, Jianxiong Gao, Wei Wang, Yanwei Fu
date	2026
venue	CVPR 2026

Modeling Spatiotemporal Neural Frames for High Resolution Brain Dynamics

Overview

A CVPR 2026 paper presenting an EEG-conditioned framework for reconstructing dynamic fMRI as continuous neural sequences with high spatial fidelity and strong temporal coherence at the cortical-vertex level. Key innovation: null-space intermediate-frame reconstruction for handling real-world fMRI sampling irregularities.

Core Contributions

1. EEG-to-fMRI Reconstruction: Leverages millisecond-level EEG temporal cues to reconstruct high-resolution fMRI
2. Spatiotemporal Neural Frames: Continuous neural sequences at cortical-vertex level
3. Null-space Intermediate Reconstruction: Handles sampling irregularities in real fMRI acquisitions
4. Measurement-Consistent Completion: Guarantees arbitrary intermediate frame completion

Methodology

Framework Architecture

class SpatiotemporalNeuralFrames(nn.Module):
    """
    EEG-conditioned fMRI reconstruction with null-space completion
    """
    def __init__(self, 
                 eeg_channels=64,
                 fmri_vertices=59412,  # Cortical surface vertices
                 latent_dim=512,
                 num_frames=10):
        super().__init__()
        
        # EEG encoder (temporal features)
        self.eeg_encoder = EEGEncoder(
            in_channels=eeg_channels,
            temporal_dim=latent_dim // 2
        )
        
        # Spatial decoder (vertex-wise reconstruction)
        self.spatial_decoder = CorticalVertexDecoder(
            latent_dim=latent_dim,
            num_vertices=fmri_vertices
        )
        
        # Null-space completion module
        self.nullspace_completer = NullSpaceCompleter(
            latent_dim=latent_dim
        )
        
        # Temporal coherence module
        self.temporal_coherence = TemporalCoherence(
            latent_dim=latent_dim
        )
        
    def forward(self, eeg_sequence, known_fmri_indices, known_fmri_frames):
        """
        Args:
            eeg_sequence: [batch, channels, time] - EEG recording
            known_fmri_indices: [num_known] - Indices of known fMRI frames
            known_fmri_frames: [batch, vertices, num_known] - Available fMRI frames
        
        Returns:
            complete_sequence: [batch, vertices, num_frames] - Full fMRI sequence
        """
        # Encode EEG temporal dynamics
        eeg_features = self.eeg_encoder(eeg_sequence)
        
        # Generate initial frame estimates from EEG
        initial_frames = self.spatial_decoder(eeg_features)
        
        
        completed_frames = .nullspace_completer(
            initial_frames,
            known_fmri_indices,
            known_fmri_frames
        )
        
        
        coherent_sequence = .temporal_coherence(completed_frames)
        
         coherent_sequence

 (nn.Module):
    
     ():
        ().__init__()
        .latent_dim = latent_dim
        .num_iterations = num_iterations
        
        
        .P_measured = nn.Linear(latent_dim, latent_dim)
        .P_null = nn.Linear(latent_dim, latent_dim)
        
     ():
        
        
        x = initial_frames.clone()
        
        
         _  (.num_iterations):
            
            x_measured = x.clone()
            x_measured[..., known_indices] = known_values
            
            
            z_measured = .P_measured(x_measured.transpose(-, -))
            z_null = .P_null(x.transpose(-, -))
            
            
            z_combined = z_measured + z_null
            
            
            x = z_combined.transpose(-, -)
            
         x

 (nn.Module):
    
     ():
        ().__init__()
        
        
        .temporal_model = nn.LSTM(
            latent_dim, latent_dim,
            num_layers=, bidirectional=
        )
        
        
        .smoothness_weight = 
        
     ():
        
        
        x = frames.transpose(-, -)  
        
        
        smoothed, _ = .temporal_model(x)
        
        
        smoothed = smoothed[..., :frames.size(-)] + smoothed[..., frames.size(-):]
        
        
        temporal_diff = smoothed[:, :] - smoothed[:, :-]
        smoothness_loss = torch.mean(temporal_diff ** )
        
         smoothed.transpose(-, -), smoothness_loss

EEG Feature Extraction

class EEGEncoder(nn.Module):
    """
    Extract temporal features from EEG signals
    """
    def __init__(self, in_channels=64, temporal_dim=256):
        super().__init__()
        
        # Multi-scale temporal convolution
        self.temporal_conv = nn.ModuleList([
            nn.Conv1d(in_channels, 64, kernel_size=k, padding=k//2)
            for k in [3, 7, 15, 31]  # Different temporal scales
        ])
        
        # Attention across scales
        self.scale_attention = nn.MultiheadAttention(64 * 4, num_heads=4)
        
        # Frequency analysis
        self.freq_bands = {
            'delta': (0.5, 4),
            'theta': (4, 8),
            'alpha': (8, 13),
            'beta': (13, 30),
            'gamma': (30, 100)
        }
        
        self.freq_encoder = nn.Sequential(
            nn.Linear(len(self.freq_bands) * 64, temporal_dim),
            nn.LayerNorm(temporal_dim),
            nn.GELU()
        )
        
     ():
        
         scipy  signal
        
        batch, channels, time = eeg.shape
        band_powers = []
        
         band_name, (low, high)  .freq_bands.items():
            
            sos = signal.butter(, [low, high], btype=, fs=, output=)
            filtered = signal.sosfilt(sos, eeg.cpu().numpy(), axis=-)
            
            
            power = torch.from_numpy(filtered ** ).to(eeg.device)
            band_powers.append(power.mean(dim=-, keepdim=))
            
         torch.cat(band_powers, dim=-)  
        
     ():
        
        
        temporal_features = []
         conv  .temporal_conv:
            feat = F.relu(conv(eeg))
            temporal_features.append(feat.mean(dim=-))  
            
        multi_scale = torch.cat(temporal_features, dim=-)  
        
        
        freq_features = .extract_frequency_bands(eeg)
        freq_features = freq_features.view(freq_features.size(), -)
        
        
        combined = torch.cat([multi_scale, freq_features], dim=-)
        
         .freq_encoder(combined)

Cortical Vertex Decoder

class CorticalVertexDecoder(nn.Module):
    """
    Decode latent features to cortical vertex activations
    """
    def __init__(self, latent_dim=512, num_vertices=59412):
        super().__init__()
        
        # Surface-aware graph convolution
        self.surface_encoder = SurfaceGraphEncoder(latent_dim)
        
        # Vertex-wise MLP
        self.vertex_decoder = nn.Sequential(
            nn.Linear(latent_dim, 1024),
            nn.LayerNorm(1024),
            nn.GELU(),
            nn.Linear(1024, 512),
            nn.LayerNorm(512),
            nn.GELU(),
            nn.Linear(512, 1)  # Single activation per vertex
        )
        
        # Temporal expansion
        self.temporal_expansion = nn.Linear(1, num_frames)
        
    def forward(self, eeg_features, surface_mesh):
        """
        Args:
            eeg_features: [batch, latent_dim]
            surface_mesh: Graph structure of cortical surface
        
        Returns:
            vertex_activations: [batch, num_vertices, num_frames]
        """
        # Encode surface geometry
        surface_features = self.surface_encoder(eeg_features, surface_mesh)
        
        # Decode to vertices
        activations = self.vertex_decoder(surface_features)  # [batch, vertices, 1]
        
        # Expand temporally
        activations_temporal = self.temporal_expansion(activations)
        
         activations_temporal

 (nn.Module):
    
     ():
        ().__init__()
         torch_geometric.nn  GCNConv
        
        .conv1 = GCNConv(latent_dim, )
        .conv2 = GCNConv(, )
        .conv3 = GCNConv(, latent_dim)
        
     ():
        
        x, edge_index = surface_mesh.x, surface_mesh.edge_index
        
        
        x = x + eeg_features.unsqueeze()  
        
        
        x = F.relu(.conv1(x, edge_index))
        x = F.relu(.conv2(x, edge_index))
        x = .conv3(x, edge_index)
        
         x

Key Innovations

1. Null-space Completion: Decomposes reconstruction into measurement + null spaces
2. Cortical-Vertex Level: Operates at fine-grained surface vertices (59k+ vertices)
3. Temporal Coherence: Enforces smoothness across time using bidirectional LSTM
4. Multi-scale EEG: Uses multiple temporal scales and frequency bands

Results

Dataset: CineBrain

• Superior voxel-wise reconstruction quality
• Robust temporal consistency across whole brain
• Preserves functional information for downstream tasks
• Supports visual decoding from reconstructed fMRI

Metrics

Metric	Performance
Voxel-wise Reconstruction	State-of-the-art
Temporal Consistency	Robust
Functional Preservation	Excellent
Visual Decoding	Supported

Applications

1. High-resolution fMRI estimation from EEG: Cost-effective alternative
2. Missing data imputation: Complete irregularly sampled fMRI
3. Temporal super-resolution: Increase temporal resolution of fMRI
4. Visual decoding: Reconstruct perceived stimuli from neural activity

Implementation

def train_model(model, train_loader, num_epochs=100):
    optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
    
    for epoch in range(num_epochs):
        for batch in train_loader:
            eeg, fmri, sampling_mask = batch
            
            # Known frames from sampling mask
            known_indices = torch.where(sampling_mask)[0]
            known_frames = fmri[..., known_indices]
            
            # Forward
            reconstructed = model(eeg, known_indices, known_frames)
            
            # Losses
            recon_loss = F.mse_loss(reconstructed, fmri)
            smooth_loss = model.temporal_coherence.smoothness_weight
            
            loss = recon_loss + smooth_loss
            
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

References

• Paper: "Modeling Spatiotemporal Neural Frames for High Resolution Brain Dynamic" (arXiv:2603.24176)
• Authors: Wanying Qu, Jianxiong Gao, Wei Wang, Yanwei Fu
• Venue: CVPR 2026

Trigger Words

• EEG fMRI reconstruction, spatiotemporal neural frames, null-space completion, cortical-vertex reconstruction, temporal coherence brain, multimodal neuroimaging, measurement-consistent completion