primate-ventral-visual-stream-dynamic

name	primate-ventral-visual-stream-dynamic
description	Framework for modeling temporal dynamics in the primate ventral visual stream across intrinsic dynamics, dynamic visual stimuli, and active sensing during eye movements. Activation: primate vision, ventral visual stream, VVS dynamics, active sensing.

Dynamic Computations in Primate Ventral Visual Stream

Comprehensive framework for modeling dynamic neural computations in the primate ventral visual stream (VVS) beyond static snapshot approaches.

Metadata

• Source: arXiv:2601.12258
• Authors: Matteo Dunnhofer, Maren Wehrheim, Hamidreza Ramezanpour, Sabine Muzellec, Kohitij Kar
• Published: 2026-01-18
• Category: q-bio.NC

Core Methodology

Three Domains of VVS Dynamics

1. Intrinsic Dynamics

• Dynamics elicited by static images
• Recurrent interactions and intrinsic circuit dynamics
• Time-varying responses beyond feedforward processing

2. Dynamic Visual Stimuli

• Responses to moving objects and changing scenes
• Temporal evolution of neural representations
• Motion processing and temporal integration

3. Active Sensing During Eye Movements

• Saccadic eye movements and fixation dynamics
• Real-world vision with active sampling
• Top-down modulation during visual exploration

Key Insights

• VVS responses are rich dynamical signals shaped by:
  • Retinal input
  • Intrinsic circuit dynamics
  • Recurrent interactions
  • Widespread top-down modulation

Required Model Components

1. Multi-area recurrence: Interactions between visual areas
2. Structured E/I interactions: Excitatory/inhibitory balance
3. Temporal objectives: Natural behavior alignment
4. Multi-timescale dynamics: Fast and slow processing

Implementation Guide

Prerequisites

import torch
import torch.nn as nn
import torchvision

Current Static Approaches vs Dynamic Framework

class StaticVVSModel(nn.Module):
    """Traditional feedforward model for static snapshots."""
    def __init__(self):
        super().__init__()
        self.backbone = torchvision.models.resnet50(pretrained=True)
    
    def forward(self, static_image):
        # Single snapshot processing
        return self.backbone(static_image)

class DynamicVVSModel(nn.Module):
    """
    Dynamic model for temporal visual processing.
    
    Components:
    - Recurrent connections for temporal integration
    - Multi-area processing
    - Structured E/I dynamics
    - Active sensing (eye movements)
    """
    def __init__(self, num_areas=4, hidden_dim=512, num_timescales=3):
        super().__init__()
        self.num_areas = num_areas
        self.num_timescales = num_timescales
        
        # Multi-area recurrent processing
        self.areas = nn.ModuleList([
            RecurrentArea(hidden_dim, num_timescales) 
            for _ in range(num_areas)
        ])
        
        # Inter-area connections
        self.connections = InterAreaConnections(num_areas, hidden_dim)
        
        # Active sensing module (eye movement prediction)
        self.active_sensing = ActiveSensingModule(hidden_dim)
        
    def forward():
        
        T = video_sequence.shape[]
        states = [[  _  (.num_timescales)] 
                   _  (.num_areas)]
        outputs = [[]  _  (.num_areas)]
        eye_movements = []
        
         t  (T):
            
            feat = .extract_features(video_sequence[t])
            
            
             t < T - :
                saccade = .active_sensing(
                    feat, eye_position[t]  eye_position     
                )
                eye_movements.append(saccade)
            
            
             i, area  (.areas):
                
                recurrent_input = .connections(states, i)
                
                
                states[i] = area(feat, states[i], recurrent_input)
                outputs[i].append(states[i][-])  
        
         [torch.stack(o)  o  outputs], torch.stack(eye_movements)

 (nn.Module):
    
     ():
        ().__init__()
        .num_timescales = num_timescales
        
        
        .exc = nn.ModuleList([
            TimescaleGRUCell(dim, dim, tau=**i)  
             i  (num_timescales)
        ])
        
        
        .inh = nn.GRUCell(dim * num_timescales, dim)
        
        
        .ei_balance = nn.Parameter(torch.zeros(dim))
        
     ():
        
        
        exc_states = []
         i, (exc_cell, prev)  ((.exc, prev_states  []*.num_timescales)):
             prev  :
                prev = torch.zeros_like(input_feat)
            state = exc_cell(input_feat + recurrent_input, prev)
            exc_states.append(state)
        
        
        combined_exc = torch.cat(exc_states, dim=-)
        inh_state = .inh(combined_exc, torch.zeros_like(input_feat) 
                             prev_states[]    prev_states[])
        
        
        new_states = []
         i, exc  (exc_states):
            
            balance = torch.sigmoid(.ei_balance * (i + ))
            modulated = exc - balance * inh_state
            new_states.append(torch.relu(modulated))
        
         new_states

 (nn.Module):
    
     ():
        ().__init__()
        .tau = tau  
        .gru = nn.GRUCell(input_size, hidden_size)
        
     ():
        
        new_hidden = .gru(, hidden)
        
        
        alpha =  / .tau
         hidden + alpha * (new_hidden - hidden)

 (nn.Module):
    
     ():
        ().__init__()
        .num_areas = num_areas
        
        
        .feedforward = nn.ModuleList([
            nn.Linear(dim, dim)  _  (num_areas - )
        ])
        .feedback = nn.ModuleList([
            nn.Linear(dim, dim)  _  (num_areas - )
        ])
        
     ():
        
        inputs = []
        
        
         target_area >   states[target_area - ]   :
            ff_input = .feedforward[target_area - ](
                states[target_area - ][-]  
            )
            inputs.append(ff_input)
        
        
         target_area < .num_areas -   states[target_area + ]   :
            fb_input = .feedback[target_area](
                states[target_area + ][-]
            )
            inputs.append(fb_input)
        
         inputs:
             (inputs) / (inputs)
         torch.zeros_like(states[][])  states[]  

 (nn.Module):
    
     ():
        ().__init__()
        .saccade_predictor = nn.Sequential(
            nn.Linear(dim + , ),  
            nn.ReLU(),
            nn.Linear(, ),
            nn.ReLU(),
            nn.Linear(, ),  
            nn.Tanh()  
        )
        
     ():
         current_position  :
            current_position = torch.zeros()
        
        combined = torch.cat([visual_features, current_position], dim=-)
        saccade = .saccade_predictor(combined)
        
         saccade

Training with Temporal Objectives

def train_dynamic_vvs(model, dataloader, epochs=100):
    """
    Train with temporal objectives reflecting natural behavior.
    """
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
    
    for epoch in range(epochs):
        for batch in dataloader:
            video, labels, eye_positions = batch
            
            # Forward pass
            area_outputs, predicted_saccades = model(video, eye_positions)
            
            # Multi-objective loss
            # 1. Classification (highest area)
            classification_loss = F.cross_entropy(
                area_outputs[-1], labels
            )
            
            # 2. Temporal consistency (smoothness)
            temporal_loss = temporal_consistency_loss(area_outputs)
            
            # 3. Active sensing prediction
            saccade_loss = F.mse_loss(predicted_saccades[:-1], eye_positions[1:])
            
            # 4. Representational alignment with brain data (optional)
            neural_loss = alignment_loss(area_outputs, neural_recordings)
            
            total_loss = (
                classification_loss + 
                0.1 * temporal_loss + 
                0.5 * saccade_loss +
                0.01 * neural_loss
            )
            
            optimizer.zero_grad()
            total_loss.backward()
            optimizer.step()

def temporal_consistency_loss(area_outputs):
    """Encourage smooth temporal evolution."""
    losses = []
    for output in area_outputs:
        # L2 smoothness penalty
        diff = output[1:] - output[:-1]
        losses.append(torch.mean(diff**))
     (losses)

 ():
    
    
     sklearn.metrics  r2_score
    
    model_repr = model_outputs[-].detach().cpu().numpy()
    neural_repr = neural_data
    
    
    model_rdm =  - np.corrcoef(model_repr)
    neural_rdm =  - np.corrcoef(neural_repr)
    
     np.mean((model_rdm - neural_rdm)**)

Key Missing Ingredients

1. Behavioral context integration: Task-dependent modulation
2. Multi-scale temporal processing: From milliseconds to seconds
3. Predictive coding: Top-down predictions and prediction errors
4. Attention mechanisms: Selective processing of relevant information
5. Reward-based learning: Reinforcement learning for active sensing

Comparison with Static Models

Aspect	Static Models	Dynamic VVS Models
Input	Single image	Video sequences
Processing	Feedforward	Recurrent + feedback
Temporal	Time-averaged	Time-resolved
Eye movements	None	Active saccadic sampling
Brain alignment	Snapshot responses	Temporal dynamics
Real-world	Limited	Natural vision conditions

Pitfalls

• Computational cost: Recurrent processing is slower than feedforward
• Training instability: Multi-timescale dynamics can be hard to train
• Data requirements: Need video datasets with temporal labels
• Hyperparameter sensitivity: E/I balance and timescales need careful tuning

Related Skills

• vision-bottleneck-v1
• primary-visual-cortex-v1-functions
• untrained-cnns-match-backprop-v1
• neural-population-dynamics

References

• arXiv:2601.12258 - Modeling Dynamic Computations in the Primate Ventral Visual Stream