Free-probability framework for analyzing stationary covariance spectra in non-normal random recurrent neural networks. Derives closed functional equations for moment generating functions and analyzes tail eigenvalue behavior in critical regimes. arXiv:2606.31944
Free-probability framework for analyzing stationary covariance spectra in non-normal random recurrent neural networks. Derives closed functional equations for moment generating functions and analyzes tail eigenvalue behavior in critical regimes. arXiv:2606.31944
Stationary Covariance Spectra of Discrete-Time Non-Normal Random Recurrent Dynamics
Overview
This skill implements theoretical and computational frameworks for analyzing stationary covariance spectra in random recurrent neural networks using free-probability theory. The methodology provides closed-form functional equations for moment generating functions and characterizes tail eigenvalue behavior in critical regimes.
Key Contributions:
Closed functional equation for stationary covariance spectrum moment generating function
Analysis of tail eigenvalues in critical regime
Comparison between discrete-time and continuous-time dynamics
Free-probability approach vs infinite Schwinger-Dyson hierarchy
Core Methodology
1. Problem Setup
Consider discrete-time random recurrent dynamics:
x[t+1] = W · φ(x[t]) + ε[t]
where:
W = random non-normal Gaussian weight matrix (variance g²/N)
φ = element-wise nonlinear activation
ε = stationary noise process
N = network size
Stationary covariance matrix:
C = lim_{T→∞} (1/T) Σ_{t=1}^T (x[t] - μ)(x[t] - μ)^T
2. Free-Probability Framework
Moment generating function:
M(z) = Σ_{k=0}^∞ m_k z^{-k}
where m_k = (1/N) Tr(C^k) are normalized moments of covariance spectrum.
Closed functional equation (discrete-time):
M(z) = F(M(z), z, g, σ)
where:
g = spectral radius parameter (g < 1 for stability)
σ = noise variance
F = function derived via free-probability tools
Key insight: Discrete-time dynamics yield a closed scalar equation, unlike continuous-time which produces infinite Schwinger-Dyson hierarchy.
3. Critical Regime Analysis
At critical point g → 1⁻:
Tail eigenvalues exhibit power-law scaling
Covariance spectrum becomes heavy-tailed
Network operates at edge of chaos
Critical scaling:
λ_max ~ N^{1/2} · (1-g)^{-α}
where α depends on activation function statistics.
4. Implementation
4.1 Numerical Verification
import numpy as np
from scipy.linalg import sqrtm
defsimulate_covariance_spectrum(N=1000, g=0.95, T=10000, phi=np.tanh):
"""
Simulate discrete-time random recurrent network and compute covariance spectrum.
Args:
N: network size
g: spectral radius parameter
T: simulation timesteps
phi: activation function
Returns:
eigenvalues: sorted eigenvalues of covariance matrix
"""# Random non-normal weight matrix
W = np.random.randn(N, N) * g / np.sqrt(N)
# Simulate dynamics
x = np.random.randn(N)
trajectory = []
for t inrange(T):
x = phi(W @ x) + 0.01 * np.random.randn(N)
trajectory.append(x)
# Compute covariance matrix
X = np.array(trajectory)
C = np.cov(X.T)
# Eigenvalue spectrum
eigenvalues = np.linalg.eigvalsh(C)
eigenvalues = np.sort(eigenvalues)[::-1]
return eigenvalues
defcompute_moment_generating_function(eigenvalues, z_values):
"""
Compute empirical moment generating function M(z) = Σ m_k z^{-k}
Args:
eigenvalues: covariance spectrum eigenvalues
z_values: complex z values
Returns:
M_z: moment generating function values
"""
N = len(eigenvalues)
moments = np.array([np.sum(eigenvalues**k) / N for k inrange(1, 20)])
M_z = np.zeros(len(z_values), dtype=complex)
for i, z inenumerate(z_values):
M_z[i] = np.sum([m * z**(-k-1) for k, m inenumerate(moments)])
return M_z
4.2 Critical Scaling Analysis
defanalyze_critical_scaling(g_values, N_values, num_trials=10):
"""
Analyze how maximum eigenvalue scales near critical point g → 1.
Args:
g_values: array of g values (e.g., np.linspace(0.5, 0.99, 20))
N_values: array of network sizes
num_trials: number of trials per parameter
Returns:
scaling_exponent: fitted critical exponent α
"""
lambda_max = np.zeros((len(g_values), len(N_values), num_trials))
for i, g inenumerate(g_values):
for j, N inenumerate(N_values):
for trial inrange(num_trials):
eigs = simulate_covariance_spectrum(N=N, g=g)
lambda_max[i, j, trial] = eigs[0]
# Average over trials
lambda_max_mean = lambda_max.mean(axis=2)
# Fit power law: λ_max ~ (1-g)^{-α}# Use log-linear regression
log_lambda = np.log(lambda_max_mean[:, 0])
log_delta = np.log(1 - g_values)
# Linear fit
coeffs = np.polyfit(log_delta, log_lambda, 1)
scaling_exponent = -coeffs[0]
return scaling_exponent
4.3 Free-Probability Theoretical Prediction
deffree_probability_prediction(g, sigma, phi_deriv_squared):
"""
Compute theoretical moment generating function via free-probability.
Args:
g: spectral radius parameter
sigma: noise variance
phi_deriv_squared: average squared derivative of activation
Returns:
M_theory: theoretical moment generating function
"""# This requires solving the closed functional equation# M = F(M, z, g, σ) iteratively# Simplified version for Marchenko-Pastur regime# Full implementation requires numerical root findingdeffixed_point_iteration(M_guess, z, g, sigma, max_iter=100):
"""Iterate to find self-consistent solution."""
M = M_guess
for _ inrange(max_iter):
# Self-consistency equation (simplified)
M_new = sigma * z**(-1) * (1 + g**2 * phi_deriv_squared * M)
if np.abs(M_new - M) < 1e-8:
break
M = M_new
return M
return fixed_point_iteration
Experimental Validation
Test 1: Marchenko-Pastur Limit (g=0)
When g=0, network reduces to noise:
x[t] = ε[t]
Covariance spectrum should follow Marchenko-Pastur law:
defmarchenko_pastur_density(sigma, N, T):
"""
Marchenko-Pastur density for white noise covariance.
λ± = σ²(1 ± √(N/T))²
"""
lambda_plus = sigma**2 * (1 + np.sqrt(N/T))**2
lambda_minus = sigma**2 * (1 - np.sqrt(N/T))**2return lambda_minus, lambda_plus
Test 2: Critical Scaling
As g → 1, verify:
Tail eigenvalues grow as power law
Covariance spectrum becomes heavy-tailed
Network approaches edge of chaos
Test 3: Discrete vs Continuous Comparison
Compare discrete-time (closed equation) with continuous-time (infinite hierarchy):
