cross-modal-convergence-dispersion

name	cross-modal-convergence-dispersion
description	Measuring cross-modal neural network convergence using single-stimulus intra-modal dispersion. Generalized Procrustes Algorithm for quantifying how stimuli with low intra-modal dispersion elicit higher cross-modal alignment. Activation triggers: cross-modal convergence, neural network alignment, vision-language alignment, representational similarity.

Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion

A methodology based on the Generalized Procrustes Algorithm to measure how intra-modal representational convergence at the single-stimulus level modulates cross-modal alignment between vision and language models.

Metadata

• Source: arXiv:2604.21836v1
• Authors: Eghbal A. Hosseini, Brian Cheung, Evelina Fedorenko, Alex H. Williams
• Published: 2026-04-23
• Category: Representation Learning, Multi-Modal AI, Neural Network Analysis

Core Methodology

Problem Statement

Neural networks exhibit remarkable representational convergence:

• Across diverse architectures
• Across training objectives
• Even across data modalities

This convergence predicts alignment with brain representations. However, it's unclear how individual stimuli elicit convergent representations across networks.

Key Question: An image can be perceived in multiple ways and expressed differently using words. What determines when different networks converge on similar representations?

Key Finding

Intra-modal dispersion strongly modulates cross-modal convergence.

Stimuli with low intra-modal dispersion (high agreement among vision models) elicit significantly higher cross-modal alignment than stimuli with high dispersion.

Hypothesis

Representational convergence arises from learning the underlying structure of the environment in similar ways. When vision models agree on how to represent a stimulus (low dispersion), that stimulus is more likely to align with language model representations.

Methodology: Generalized Procrustes Analysis

Intra-Modal Dispersion

Concept: Measure how much vision models disagree about a single stimulus.

Vision Model A → Representation r_A(stimulus_i)
Vision Model B → Representation r_B(stimulus_i)
Vision Model C → Representation r_C(stimulus_i)

Intra-Modal Dispersion = variance([r_A, r_B, r_C]) after alignment

Low dispersion: All vision models represent the stimulus similarly High dispersion: Vision models have different representations

Cross-Modal Convergence

Concept: Measure alignment between vision and language models for the same stimulus.

Vision Model → Representation r_V(stimulus_i)
Language Model → Representation r_L(text_description_i)

Cross-Modal Alignment = similarity(r_V, r_L)

The Relationship

Low Intra-Modal Dispersion ──────→ High Cross-Modal Alignment
        ↓                                    ↓
Vision models agree              Vision-language models align
        ↓                                    ↓
Clear, unambiguous               Consistent representation
visual structure                 across modalities

High Intra-Modal Dispersion ──────→ Low Cross-Modal Alignment
        ↓                                    ↓
Vision models disagree           Vision-language models misalign
        ↓                                    ↓
Ambiguous or complex             Inconsistent representation
visual content                   across modalities

Generalized Procrustes Algorithm

Purpose: Align representations from different models for fair comparison.

# Standard Procrustes Problem
Given: Two matrices X, Y (representations from two models)
Find: Orthogonal Q, translation b, scale s minimizing:
      ||s * X * Q + b - Y||²

# Generalized Procrustes (for multiple models)
Given: Matrices X₁, X₂, ..., Xₙ (representations from n models)
Find: Transformations for each minimizing:
      Σᵢ ||transform(Xᵢ) - consensus||²

Implementation Guide

Prerequisites

# Core dependencies
numpy
scipy
scikit-learn
torch

# For model access
timm          # Vision models
transformers  # Language models

Step 1: Extract Representations

import torch
import numpy as np
from transformers import CLIPModel, CLIPProcessor
import timm

class RepresentationExtractor:
    """
    Extract representations from vision and language models.
    """
    def __init__(self, device='cuda'):
        self.device = device
        
        # Load multiple vision models
        self.vision_models = {
            'resnet50': timm.create_model('resnet50', pretrained=True, num_classes=0).to(device),
            'vit': timm.create_model('vit_base_patch16_224', pretrained=True, num_classes=0).to(device),
            'dino': timm.create_model('vit_base_patch16_224_dino', pretrained=True, num_classes=0).to(device),
            'deit': timm.create_model('deit_base_patch16_224', pretrained=True, num_classes=0).to(device),
        }
        
        # Load language model (CLIP for vision-language alignment)
        self.clip = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").to(device)
        self.clip_processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
        
        for model in self.vision_models.values():
            model.eval()
        self.clip.()
    
     ():
        
        representations = {}
        
         torch.no_grad():
             name, model  .vision_models.items():
                features = model(image.to(.device))
                representations[name] = features.cpu().numpy()
        
         representations
    
     ():
        
        inputs = .clip_processor(text=[text], return_tensors=, padding=)
        
         torch.no_grad():
            text_features = .clip.get_text_features(**inputs)
        
         text_features.cpu().numpy()
    
     ():
        
        inputs = .clip_processor(images=image, return_tensors=)
        
         torch.no_grad():
            vision_features = .clip.get_image_features(**inputs)
        
         vision_features.cpu().numpy()

Step 2: Generalized Procrustes Analysis

from scipy.linalg import orthogonal_procrustes
from scipy.spatial.distance import cosine

def procrustes_alignment(X, Y):
    """
    Align Y to X using orthogonal Procrustes.
    
    Args:
        X: Reference representation [n_samples, n_features]
        Y: Target representation [n_samples, n_features]
    
    Returns:
        Y_aligned: Aligned Y
        R: Orthogonal rotation matrix
    """
    R, _ = orthogonal_procrustes(Y, X)
    Y_aligned = Y @ R
    return Y_aligned, R


def generalized_procrustes(representations_dict, max_iter=100, tol=1e-6):
    """
    Align multiple representations using Generalized Procrustes Analysis.
    
    Args:
        representations_dict: Dict of {model_name: representation_matrix}
        max_iter: Maximum iterations
        tol: Convergence tolerance
    
    Returns:
        aligned_reps: Dict of aligned representations
        consensus: Consensus (mean) representation
    """
    model_names = list(representations_dict.keys())
    n_models = len(model_names)
    n_samples = representations_dict[model_names[0]].shape[0]
    
    # Normalize each representation
    normalized = {}
    for name, rep in representations_dict.items():
        # Center and scale
        rep_centered = rep - rep.mean(axis=0)
        rep_scaled = rep_centered / np.linalg.norm(rep_centered, axis=1, keepdims=True)
        normalized[name] = rep_scaled
    
    # Initialize consensus as mean
    consensus = np.mean(list(normalized.values()), axis=0)
    
    # Iterative alignment
    for iteration in range(max_iter):
        aligned = {}
        
         name, rep  normalized.items():
            
            aligned_rep, _ = procrustes_alignment(consensus, rep)
            aligned[name] = aligned_rep
        
        
        new_consensus = np.mean((aligned.values()), axis=)
        
        
        diff = np.linalg.norm(new_consensus - consensus)
         diff < tol:
            
        
        consensus = new_consensus
    
     aligned, consensus


 ():
    
    
    reps_stack = np.stack((aligned_representations.values()), axis=)
    
    
    dispersion = np.var(reps_stack, axis=).mean(axis=)
    
     dispersion

Step 3: Measure Cross-Modal Alignment

def compute_cross_modal_alignment(vision_reps, language_reps, metric='cosine'):
    """
    Compute cross-modal alignment for each stimulus.
    
    Args:
        vision_reps: Vision representations [n_stimuli, n_features]
        language_reps: Language representations [n_stimuli, n_features]
        metric: Similarity metric ('cosine' or 'euclidean')
    
    Returns:
        alignment: [n_stimuli] vector of alignment scores
    """
    if metric == 'cosine':
        # Cosine similarity
        vision_norm = vision_reps / (np.linalg.norm(vision_reps, axis=1, keepdims=True) + 1e-8)
        language_norm = language_reps / (np.linalg.norm(language_reps, axis=1, keepdims=True) + 1e-8)
        
        # Element-wise cosine similarity for corresponding stimuli
        alignment = np.sum(vision_norm * language_norm, axis=1)
    else:
        # Negative Euclidean distance (higher = more aligned)
        diff = vision_reps - language_reps
        alignment = -np.linalg.norm(diff, axis=1)
    
    return alignment

Step 4: Analyze Relationship

import matplotlib.pyplot as plt
from scipy import stats

def analyze_dispersion_alignment_relationship(intra_modal_dispersion,
                                               cross_modal_alignment,
                                               bins=5):
    """
    Analyze how intra-modal dispersion modulates cross-modal alignment.
    
    Args:
        intra_modal_dispersion: [n_stimuli] dispersion values
        cross_modal_alignment: [n_stimuli] alignment values
        bins: Number of dispersion bins
    
    Returns:
        analysis_results: Dict with statistics and binned results
    """
    # Compute correlation
    correlation, p_value = stats.pearsonr(intra_modal_dispersion, cross_modal_alignment)
    
    # Bin by dispersion
    percentiles = np.percentile(intra_modal_dispersion, 
                                 np.linspace(0, 100, bins + 1))
    
    binned_results = []
    for i in range(bins):
        lower = percentiles[i]
        upper = percentiles[i + 1]
        
        mask = (intra_modal_dispersion >= lower) & (intra_modal_dispersion < upper)
        mean_alignment = np.mean(cross_modal_alignment[mask])
        std_alignment = np.std(cross_modal_alignment[mask])
        
        binned_results.append({
            'dispersion_range': (lower, upper),
            'mean_alignment': mean_alignment,
            'std_alignment': std_alignment,
            'n_stimuli': mask.sum()
        })
    
    return {
        'correlation': correlation,
        'p_value': p_value,
        'binned_results': binned_results
    }


def visualize_relationship(intra_modal_dispersion, cross_modal_alignment,
                          model_names=None):
    
    fig, axes = plt.subplots(, , figsize=(, ))
    
    
    ax1 = axes[]
    ax1.scatter(intra_modal_dispersion, cross_modal_alignment, alpha=)
    ax1.set_xlabel()
    ax1.set_ylabel()
    ax1.set_title()
    
    
    z = np.polyfit(intra_modal_dispersion, cross_modal_alignment, )
    p = np.poly1d(z)
    ax1.plot(np.sort(intra_modal_dispersion), 
             p(np.sort(intra_modal_dispersion)), 
             , alpha=)
    
    
    ax2 = axes[]
    analysis = analyze_dispersion_alignment_relationship(
        intra_modal_dispersion, cross_modal_alignment
    )
    
    means = [b[]  b  analysis[]]
    stds = [b[]  b  analysis[]]
    labels = [ 
               b  analysis[]]
    
    ax2.bar(((means)), means, yerr=stds)
    ax2.set_xticks(((means)))
    ax2.set_xticklabels(labels, rotation=)
    ax2.set_xlabel()
    ax2.set_ylabel()
    ax2.set_title()
    
    plt.tight_layout()
     fig, analysis

Step 5: Complete Analysis Pipeline

def analyze_cross_modal_convergence(stimuli, text_descriptions, images):
    """
    Complete pipeline for analyzing cross-modal convergence.
    
    Args:
        stimuli: List of stimulus identifiers
        text_descriptions: List of text descriptions (parallel to stimuli)
        images: List of image tensors (parallel to stimuli)
    
    Returns:
        results: Dict with all analysis results
    """
    # Initialize extractor
    extractor = RepresentationExtractor()
    
    # Extract vision representations from multiple models
    print("Extracting vision representations...")
    vision_reps = {name: [] for name in extractor.vision_models.keys()}
    
    for img in images:
        reps = extractor.extract_vision_representations(img)
        for name, rep in reps.items():
            vision_reps[name].append(rep[0])  # [0] to remove batch dim
    
    # Convert to matrices
    vision_matrices = {name: np.stack(reps) for name, reps in vision_reps.items()}
    
    # Extract language representations
    print("Extracting language representations...")
    language_reps = []
    for text in text_descriptions:
        rep = extractor.extract_language_representation(text)
        language_reps.append(rep[0])
    language_matrix = np.stack(language_reps)
    
    # Generalized Procrustes for vision models
    print("Aligning vision representations...")
    aligned_vision, vision_consensus = generalized_procrustes(vision_matrices)
    
    # Compute intra-modal dispersion
    print("Computing intra-modal dispersion...")
    intra_modal_dispersion = compute_intra_modal_dispersion(aligned_vision)
    
    # Compute cross-modal alignment
    ()
    
    cross_modal_alignment = compute_cross_modal_alignment(
        aligned_vision[], 
        language_matrix
    )
    
    
    ()
    analysis = analyze_dispersion_alignment_relationship(
        intra_modal_dispersion, 
        cross_modal_alignment
    )
    
    
    fig, detailed_analysis = visualize_relationship(
        intra_modal_dispersion, 
        cross_modal_alignment,
        (extractor.vision_models.keys())
    )
    
     {
        : intra_modal_dispersion,
        : cross_modal_alignment,
        : analysis[],
        : analysis[],
        : analysis[],
        : fig
    }

Applications

1. Model Selection - Choose stimuli that maximize cross-modal alignment
2. Dataset Curation - Filter ambiguous stimuli that reduce alignment
3. Brain Alignment - Predict which stimuli will align with neural recordings
4. Multi-Modal Training - Design better vision-language pretraining datasets
5. Interpretability - Understand what makes stimuli "alignable" across modalities

Key Findings

• Up to 2x improvement: Low dispersion stimuli show 2x higher cross-modal alignment
• Robust effect: Generalizes across different vision-language model pairings
• Interpretable: High dispersion = ambiguous/complex content; Low dispersion = clear structure

Pitfalls

1. Model Selection - Results depend on which vision models are compared
2. Procrustes Limitations - Orthogonal transformations may not capture all alignment types
3. Stimulus Selection Bias - Care needed to avoid circular reasoning
4. Computational Cost - Multiple forward passes required
5. Dimensionality - High-dimensional representations may need dimensionality reduction

Related Skills

• vlm-visual-cortex-alignment - Vision-language model brain alignment
• brain-llm-key-neurons-grammar - Brain-LLM analogy
• meta-learning-in-context-brain-decoding - Cross-subject brain decoding

References

@article{hosseini2026crossmodal,
  title={Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion},
  author={Hosseini, Eghbal A. and Cheung, Brian and Fedorenko, Evelina and Williams, Alex H.},
  journal={arXiv preprint arXiv:2604.21836},
  year={2026}
}