memory-uncertainty-relation-recurrent-networks

# Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks ## Overview **Paper**: "Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks" **Authors**: Taichi Haruna, Kohei Nakajima **arXiv**: [2605.24628](https://arxiv.org/abs/2605.24628) **Published**: 2026-05-23 **Categories**: nlin.AO, cond-mat.dis-nn, cs.NE ## Core Contribution This work establishes a fundamental **uncertainty-type inequality** for short-term memory in dynamical systems: $$\text{STM}(\mathcal{S}) \cdot \text{Fluct}(\mathcal{S}) \geq C$$ Where: - **STM**: Short-term memory capacity measure - **Fluct**: Size of state-space fluctuations induced by input signals - **C**: A positive constant (lower bound) This is analogous to Heisenberg's uncertainty principle — you cannot simultaneously minimize both memory and fluctuations below the bound. ## Key Definitions ### Short-Term Memory (STM) Capacity Following Jaeger (2001), STM measures how well a reservoir can recall past inputs: $$\text{STM} = \sum_{k=1}^{\infty} r^2(s_t, u_{t-k})$$ Where $r^2$ is the coefficient of determination between the state $s_t$ and a delayed input $u_{t-k}$. ### Harmonic Memory The **harmonic memory** $H(\mathcal{S})$ is the suboptimal lower bound achievable by the best linear readout weight: $$H(\mathcal{S}) = \frac{\left(\sum_k \text{cov}(s_t, u_{t-k})\right)^2}{\text{var}(s_t)}$$ This is always achievable (tight) — making it a **constructive lower bound**. ### State-Space Fluctuation Measured as the variance of the reservoir state induced by the input process: $$\text{Fluct}(\mathcal{S}) = \text{Tr}[\text{Var}(s_t)]$$ ## Main Results ### Theorem 1: Memory Uncertainty Inequality For any reservoir system $\mathcal{S}$ with state process $s_t$ driven by input $u_t$: $$\text{STM}(\mathcal{S}) \geq H(\mathcal{S}) \geq \frac{C}{\text{Fluct}(\mathcal{S})}$$ **Interpretation**: To have high memory capacity, you need either large fluctuations OR the system must be close to the harmonic optimum. ### When Equality Holds 1. **Exact equality**: When the state is a simple linear function of past inputs 2. **Asymptotic equality**: When reservoir spectral radius → 1 (edge of chaos) 3. **Strict inequality**: For nonlinear reservoirs with complex internal dynamics ### Effect of State-Space Regularization Adding L2 regularization to reservoir states: - Reduces fluctuations - May decrease memory capacity proportionally - The ratio STM/Fluct is conserved — the bound is "tight" under regularization ## Implementation Guide ### Measuring STM in a Reservoir ```python import numpy as np from sklearn.linear_model import LinearRegression def measure_stm(states, inputs, max_delay=50): """Measure short-term memory capacity of a reservoir. Args: states: (T, N) reservoir states inputs: (T,) input time series max_delay: maximum delay to consider Returns: stm: total STM capacity r2_per_delay: R² for each delay """ T, N = states.shape r2_per_delay = [] for k in range(1, max_delay + 1): # Align states with delayed inputs s = states[k:, :] u_delayed = inputs[:T - k].reshape(-1, 1) # Fit linear readout reg = LinearRegression().fit(s, u_delayed) r2 = reg.score(s, u_delayed) r2_per_delay.append(max(0, r2)) # Clamp to [0, 1] stm = sum(r2_per_delay) return stm, r2_per_delay def measure_harmonic_memory(states, inputs, max_delay=50): """Compute harmonic memory lower bound. Args: states: (T, N) reservoir states (use single-neuron for simplicity) inputs: (T,) input time series Returns: harmonic_memory: lower bound on STM """ T, N = states.shape # Sum of covariances: sum_k cov(s_t, u_{t-k}) cov_sum = 0 for k in range(1, min(T // 2, 200)): cov = np.cov(states[k:, 0], inputs[:T - k])[0, 1] cov_sum += cov # Variance of state state_var = np.var(states[:, 0]) if state_var < 1e-10: return 0.0 return (cov_sum ** 2) / state_var def measure_fluctuation(states): """Measure state-space fluctuation (trace of covariance).""" return np.trace(np.cov(states.T)) ``` ### Reservoir with Tunable Spectral Radius ```python class EchoStateNetwork: """Simple Echo State Network for memory capacity experiments.""" def __init__(self, N=100, spectral_radius=0.9, input_scaling=0.1, density=0.1, seed=42): rng = np.random.RandomState(seed) # Random recurrent weights W = rng.randn(N, N) # Sparse connectivity mask = rng.uniform(0, 1, (N, N)) < density W *= mask # Scale to desired spectral radius sr = np.max(np.abs(np.linalg.eigvals(W))) self.W = spectral_radius * W / sr # Input weights self.W_in = input_scaling * rng.randn(N, 1) self.N = N self.spectral_radius = spectral_radius def run(self, inputs, washout=100): """Run reservoir on input sequence. Args: inputs: (T,) input time series washout: initial steps to discard Returns: states: (T - washout, N) reservoir states """ T = len(inputs) states = np.zeros((T, self.N)) x = np.zeros(self.N) for t in range(T): x = np.tanh(self.W @ x + self.W_in.flatten() * inputs[t]) states[t] = x return states[washout:] def memory_profile(self, inputs, max_delay=100): """Full memory capacity profile.""" states = self.run(inputs, washout=200) stm, r2s = measure_stm(states, inputs[200:], max_delay) harmonic = measure_harmonic_memory(states, inputs[200:], max_delay) fluct = measure_fluctuation(states) return { 'stm': stm, 'harmonic_memory': harmonic, 'fluctuation': fluct, 'r2_profile': r2s, 'memory_uncertainty_product': stm * fluct, } ``` ### Analyzing the Uncertainty Bound ```python import matplotlib.pyplot as plt def analyze_memory_uncertainty(spectral_radii=np.linspace(0.5, 0.99, 20)): """Analyze how the memory-fluctuation trade-off changes with spectral radius.""" results = [] rng = np.random.RandomState(0) inputs = rng.randn(2000) for sr in spectral_radii: esn = EchoStateNetwork(N=100, spectral_radius=sr) profile = esn.memory_profile(inputs) profile['spectral_radius'] = sr results.append(profile) # Plot fig, axes = plt.subplots(1, 3, figsize=(15, 4)) sr_vals = [r['spectral_radius'] for r in results] stms = [r['stm'] for r in results] flucts = [r['fluctuation'] for r in results] products = [r['memory_uncertainty_product'] for r in results] axes[0].plot(sr_vals, stms, 'b-o', label='STM') axes[0].set_xlabel('Spectral Radius') axes[0].set_ylabel('STM Capacity') axes[0].set_title('Memory vs Spectral Radius') axes[1].plot(sr_vals, flucts, 'r-o', label='Fluctuation') axes[1].set_xlabel('Spectral Radius') axes[1].set_ylabel('State Fluctuation') axes[1].set_title('Fluctuation vs Spectral Radius') axes[2].plot(sr_vals, products, 'g-o', label='STM × Fluct') axes[2].axhline(y=min(products), color='k', linestyle='--', label='Lower Bound') axes[2].set_xlabel('Spectral Radius') axes[2].set_ylabel('STM × Fluctuation') axes[2].set_title('Uncertainty Product (should be ≥ bound)') axes[2].legend() plt.tight_layout() return results, fig ``` ## Theoretical Implications ### For Reservoir Computing Design | Design Goal | Implication from Uncertainty Relation | |------------|--------------------------------------| | **High STM** | Must accept high fluctuations OR operate near harmonic optimum | | **Stable dynamics** | Low fluctuations → bounded memory capacity | | **Edge of chaos** (sr → 1) | Approaches equality — maximum memory per unit fluctuation | | **Deep nonlinear reservoirs** | Strict inequality — some memory "wasted" on nonlinear processing | ### Connection to Physics - Analogous to **Heisenberg uncertainty**: ΔxΔp ≥ ℏ/2 - Here: STM × Fluct ≥ C - The "harmonic memory" is the minimum uncertainty state — like a coherent state in QM ### Practical Guidelines 1. **Choose spectral radius near 1** for best memory efficiency 2. **Use harmonic memory as a diagnostic**: if STM ≈ H, the reservoir is close to optimal 3. **Regularization trade-off**: L2 regularization reduces fluctuations but proportionally reduces memory 4. **Nonlinearity**: Strong nonlinear activation (tanh saturation) increases the STM/H gap ## Applications 1. **Reservoir computing optimization**: Tune spectral radius to the harmonic optimum 2. **Memory-efficient RNNs**: Design architectures that approach the harmonic bound 3. **Cognitive modeling**: Memory capacity bounds as a model for working memory limitations 4. **Anomaly detection**: Compare measured STM vs harmonic bound to detect sub-optimal network states ## Pitfalls - **Washout period**: Always discard initial transients when measuring STM - **Input statistics matter**: The bound depends on input process statistics - **Finite-time estimation**: Need long time series for accurate STM estimation (T ≥ 10 × max_delay) - **Nonlinear reservoirs**: The harmonic bound may be loose for highly nonlinear systems ## Citation ```bibtex @article{haruna2026memory, title={Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks}, author={Haruna, Taichi and Nakajima, Kohei}, journal={arXiv preprint arXiv:2605.24628}, year={2026} } ```