scalable-perturbation-learning-esn

name	scalable-perturbation-learning-esn
description	Scalable Perturbation Learning for Online Self-Supervised Echo State Networks - orthogonal decomposition reduces perturbation dimension from reservoir to input dimension, enabling scalable hardware-compatible online learning
tags	["echo-state-networks","perturbation-learning","online-learning","self-supervised","neuromorphic","reservoir-computing"]
activation_words	["ESN","perturbation learning","online self-supervised","orthogonal decomposition","reservoir","scalable learning"]
arxiv_id	2607.06079

Scalable Perturbation Learning for Online Self-Supervised ESNs

Paper Info

• Title: Scalable Perturbation Learning for Online Self-Supervised Echo State Networks
• arXiv: 2607.06079
• Authors: Taiki Yamada, Kantaro Fujiwara
• Date: 2026-07-08
• Categories: cs.LG, cs.NE

Core Contribution

A perturbation-based learning rule for online self-supervised learning in Echo State Networks (ESNs) that achieves dimension-independent variance by exploiting an orthogonal decomposition of the self-supervised learning cost.

Key Innovation: Orthogonal Decomposition

The self-supervised learning cost in ESNs can be decomposed into:

1. Input-dependent component — varies with incoming data
2. Redundant component — determined entirely by fixed ESN parameters

By perturbing only the input-dependent component, the effective perturbation dimension drops from the reservoir dimension (N) to the input dimension (d), where d << N.

Problem Addressed

Perturbation-based learning suffers from variance that grows with the dimension of perturbed variables. In large ESN reservoirs (thousands of neurons), this makes perturbation learning impractical despite its advantages:

• Self-supervised adaptation (no labeled data needed)
• Online learning (continuous adaptation)
• Hardware compatibility (scalar feedback only)

Methodology

Design Principle

Online learning should be restricted to the dynamically necessary low-dimensional component of the objective.

Algorithm Steps

1. Decompose the self-supervised cost J into J = J_input(u) + J_fixed(θ)
2. Compute gradient ∂J_input/∂u (depends only on input, not reservoir state)
3. Apply perturbation only to the input-dependent pathway
4. Update readout weights using scalar feedback signal

Complexity

• Standard perturbation: O(N) variance growth with reservoir size
• Proposed method: O(d) variance growth with input dimension (d << N)
• Preserves all three properties: self-supervised, online, scalar-feedback

Applications

1. Neuromorphic hardware — low-dimensional perturbation compatible with analog circuits
2. Continual learning — online adaptation without catastrophic forgetting
3. Edge AI — memory-efficient implementation for resource-constrained devices
4. Robotics — real-time adaptation to changing environments

Limitations

• Requires identifiable orthogonal decomposition (may not generalize to all network architectures)
• Demonstrated on ESNs specifically; extension to general RNNs not validated
• Self-supervised objective choice affects decomposition quality

Related Skills

• [[reservoir-computing]]
• [[perturbation-learning]]
• [[online-learning]]
• [[neuromorphic-computing]]

Implementation Notes

The key insight is architectural: by constraining the learning to the input-dependent subspace, we get a natural dimensionality reduction that preserves learning signal while eliminating variance explosion. This is analogous to how backpropagation through time (BPTT) can be replaced by real-time recurrent learning (RTRL) when the network is small, but here the decomposition makes perturbation learning scalable even for large networks.