attractor-models-language-reasoning

name

attractor-models-language-reasoning

description

Attractor Models for language and reasoning — backbone proposes output embeddings, attractor module refines them by solving for fixed point via implicit differentiation. Constant memory for effective depth, adaptive iteration count, equilibrium internalization phenomenon. Outperforms standard and looped Transformers across language modeling and challenging reasoning tasks (Sudoku-Extreme 91.4%, Maze-Hard 93.1% with 27M params). Use when designing recurrent/iterative refinement architectures, fixed-point models, energy-based reasoning, or efficient looped Transformers. arXiv: 2605.12466 (cs.LG, cs.AI, cs.CL, cs.NE). Fein-Ashley, Rashidinejad.

Attractor Models for Language and Reasoning

Fixed-point attractor models turn recurrence into scalable iterative refinement with implicit differentiation, achieving equilibrium internalization — solver removable at inference with minimal loss.

Metadata

• Source: arXiv:2605.12466
• Authors: Jacob Fein-Ashley, Paria Rashidinejad
• Published: 2026-05-12
• Subjects: cs.LG, cs.AI, cs.CL, cs.NE

Core Problem

Looped Transformers improve language modeling and reasoning through iterative refinement, but face:

1. Training instability — recurrent architectures are hard to optimize
2. Fixed recurrence depth — constrained to small, pre-defined iteration counts
3. High deployment cost — scaling recurrence is computationally expensive

Key Innovation

Attractor Models: Two-stage architecture that decouples proposal from refinement:

1. Backbone module: Proposes initial output embeddings
2. Attractor module: Refines embeddings by solving for the fixed point
   • Gradients via implicit differentiation (not backprop-through-time)
   • Training memory is constant in effective depth
   • Iteration count chosen adaptively by convergence

Novel Phenomenon: Equilibrium Internalization

Fixed-point training places the model's initial output embedding near equilibrium, allowing the solver to be removed at inference time with little degradation. The model learns to internalize the fixed-point computation.

Results

Language Modeling

• 770M Attractor Model outperforms 1.3B Transformer trained on 2× more tokens
• Pareto improvement over standard and stable looped Transformers across sizes
• Up to 46.6% perplexity improvement, 19.7% downstream accuracy gain
• Reduced training cost

Reasoning (27M params, ~1000 examples)

• Sudoku-Extreme: 91.4% accuracy
• Maze-Hard: 93.1% accuracy
• Outperforms Claude and GPT-o3 (which fail completely)
• Specialized recursive reasoners collapse at larger sizes; Attractor Models scale favorably

Implementation Guide

Architecture Design

Input → Backbone → Initial Embeddings → Attractor → Fixed Point → Output
                  (feed-forward)        (iterative, implicit diff)

Training

• Implicit differentiation for gradients through fixed point
• No need for BPTT or truncated backprop
• Memory efficient — constant regardless of effective depth

Inference

• Adaptive convergence: iterate until solution stabilizes
• After training: solver may be removed (equilibrium internalization)
• Backbone alone produces near-fixed-point outputs

Applications

• Efficient language model pretraining with iterative refinement
• Reasoning tasks requiring recursive/iterative computation
• Memory-constrained deployment of recurrent architectures
• Energy-based inference systems
• Neuro-inspired reasoning models with attractor dynamics

Pitfalls

• Fixed-point convergence not guaranteed for all inputs
• Implicit differentiation requires stable Jacobian at fixed point
• Equilibrium internalization is an emergent property, not enforced

Related Skills

• attractor-fcm-gradient-descent
• attractor-metadynamics-neural
• attention-residuals
• memory-efficient-looped-transformer
• neuro-attractor-landscape-working-memory