free-probability-rnn-spectral-analysis

name	free-probability-rnn-spectral-analysis
description	Free probability approach to analyzing stationary covariance spectra of random recurrent neural networks. Derives closed functional equations for moment generating functions of limiting stationary covariance spectra with random non-normal Gaussian weights.
trigger_words	["free probability","stationary covariance","random recurrent","non-normal dynamics","moment generating function","covariance spectrum","RNN spectral analysis"]
category	neuroscience

Overview

This methodology (arXiv:2606.31944) uses free probability theory to derive the stationary covariance spectrum of discrete-time random recurrent neural networks with non-normal weight matrices. Critical for understanding how variance distributes among principal components in noise-driven RNN dynamics.

Core Theory

Problem Setup

• Input: Random non-normal Gaussian weight matrix W, noise-driven dynamics
• Output: Stationary covariance matrix spectrum (eigenvalue distribution)
• Key insight: For discrete-time dynamics, a closed functional equation exists for the moment generating function

Free Probability Approach

1. Model weight matrix W as a random matrix ensemble
2. Use free probability tools to compute resolvent of covariance matrix
3. Derive moment generating function M(z) via fixed-point equation
4. Extract eigenvalue distribution from imaginary part of M(z)
5. Analyze tail eigenvalues in critical regime

Key Results

1. Discrete-time: Closed scalar functional equation for limiting spectrum
2. Continuous-time: Leads to infinite hierarchy of Schwinger-Dyson equations (no closed form)
3. Critical regime: Tail eigenvalue behavior determines PCA interpretability

Implementation

Moment Generating Function Equation

For discrete-time dynamics x_{t+1} = W x_t + noise:

• The moment generating function satisfies: M(z) = f(M(z), z, σ²)
• Where σ² is the noise variance and f depends on the weight distribution
• The eigenvalue density ρ(λ) = Im[M(λ + iε)] / π

Critical Regime Analysis

• Near criticality (spectral radius of W ≈ 1), tail eigenvalues dominate
• Use the derived functional equation to predict tail behavior
• Compare predictions with empirical PCA on neural data

Pitfalls

• Discrete vs continuous: The closed-form result only applies to discrete-time dynamics. Continuous-time requires solving an infinite hierarchy.
• Non-normal assumption: Results assume random non-normal weights, not structured/learned weights.
• Gaussian assumption: Derivation assumes Gaussian weight distribution; heavy-tailed weights may need different treatment.

Applications

• Neural data analysis: Compare model predictions to empirical neural covariance
• RNN interpretability: Understand which directions in state space carry most variance
• Network design: Engineer weight matrices for desired spectral properties

Verification

1. Generate random non-normal weight matrix with known spectral properties
2. Simulate noise-driven dynamics and compute empirical covariance
3. Compare empirical eigenvalue distribution with theoretical prediction
4. Verify tail behavior matches free-probability prediction in critical regime

Activation

free probability, stationary covariance, random recurrent, non-normal weights, covariance spectrum, moment generating function, RNN analysis, PCA interpretation, neural data analysis, critical regime