emergent-generalization-representation-learning

name

emergent-generalization-representation-learning

description

Emergent generalization by representation learning in artificial neural networks. An explicit information bottleneck forcing an RNN to learn a low-dimensional representation is necessary for rotational and out-of-distribution generalization in time-series prediction. Uses information-theoretic causal emergence to characterize the memorization-to-generalization transition (non-monotonic down-min-up trajectory) and finds analogous dynamics in CA1 hippocampal activity of mice learning an alternating maze. Supports a causal role for learned representations in cognition. Activation: neural manifold generalization, information bottleneck RNN, causal emergence representation, out-of-distribution generalization, memorization to generalization transition, CA1 hippocampal dynamics, low-dimensional representation necessary generalization

Emergent Generalization by Representation Learning in Artificial Neural Networks

Overview

Low-dimensional neural manifolds (identified via dimensionality reduction) have improved the interpretability of population-level neural coding. But whether such compact representations are biologically functional or merely descriptive remains contested. This paper shows that an explicit information bottleneck forcing a recurrent neural network (RNN) to learn a low-dimensional representation is necessary for rotational and out-of-distribution (OOD) generalization in a time-series prediction task.

Paper: Emergent Generalization by Representation Learning in Artificial Neural Networks

arXiv: 2607.10430v1 (July 11, 2026) Authors: Hardik Rajpal, Dan Goodman

Core Findings

1. Low-D representation is causal, not incidental: an explicit information bottleneck that compresses the RNN's hidden state to a low-D manifold is required for rotational and OOD generalization (not just correlated with it).
2. Causal-emergence trajectory is non-monotonic: across the memorization→generalization transition, the information-theoretic measure of causal emergence first decreases, hits a minimum, then rises to a maximum — even while prediction loss falls monotonically.
3. Scales with task complexity: more complex tasks produce larger-magnitude emergent structure; the magnitude of emergent structure reliably predicts generalization performance.
4. Biological validation: analysis of CA1 hippocampal activity in mice learning an alternating maze reveals analogous non-monotonic emergence dynamics that track behavioral performance — linking the ANN result to real neural computation.

Why It Matters

• Reframes neural manifolds from descriptive tools to functional/causal ones
• Provides a concrete, measurable signal (causal-emergence trajectory) for when a network has genuinely generalized vs merely memorized
• Bridges ANN and systems neuroscience (hippocampus) with a shared dynamical signature
• Practical for evaluating representation quality in RNNs without held-out OOD sets

Methodology Pattern

1. Train RNN on time-series prediction; insert an information bottleneck
   (e.g., bottleneck/compress hidden state -> low-D latent -> reconstruct/output).
2. Measure causal emergence (information-theoretic): EI or similar metric of
   coarse-graining robustness of the latent dynamics.
3. Track trajectory of EI across training (memorize -> generalize):
   expect down -> min -> up.
4. Correlate EI magnitude / trajectory shape with OOD generalization score.
5. (Optional) Compare against biological recordings (e.g., CA1 calcium/hd imaging)
   on a comparable task for convergent dynamics.

Use When

• Evaluating whether an RNN/spiking network has learned a generalizable representation
• Designing representation-learning objectives that explicitly enforce compactness
• Linking artificial and biological neural dynamics via shared manifold signatures
• Analyzing hippocampal / replay data for learning-phase transitions
• You need an early signal of OOD generalization beyond validation loss

Pitfalls

• Bottleneck strength matters: too tight kills task performance; too loose yields no emergence signal. Tune the compression rate.
• Causal-emergence metric choice: results depend on the specific information-theoretic measure (EI variant, coarse-graining scheme); report which.
• Non-monotonicity is subtle: the minimum can be shallow; requires enough training resolution to resolve the down→min→up shape.
• Biological analogy is correlational: convergent CA1 dynamics support but do not prove causal equivalence between ANN and brain.
• Activation Keywords: neural manifold generalization, information bottleneck RNN, causal emergence representation, out-of-distribution generalization, memorization to generalization transition, CA1 hippocampal dynamics, low-dimensional representation necessary generalization

References

• arXiv: 2607.10430v1
• Categories: q-bio.NC, cs.LG, cs.NE
• Related skills: dynamic-neural-manifolds-control (parameterizable dynamic manifolds on neuromorphic hardware), spiking-polar-trajectory-generator (manifold-riding SNN trajectories), neural-dynamics-analysis-methodology (generic neural-dynamics analysis framework)