less-is-more-recursive-reasoning-tiny-networks

name	less-is-more-recursive-reasoning-tiny-networks
title	Less is More: Recursive Reasoning with Tiny Networks
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.04871
keywords	["recursive reasoning","tiny models","parameter efficiency","puzzle solving","latent recursion"]
description	Achieve complex reasoning with minimal parameters using latent recursion in 2-layer networks. A 7M-parameter Tiny Recursive Model (TRM) solves Sudoku (87% accuracy), mazes (85%), and ARC-AGI with 0.01% the parameters of large LLMs via iterative latent refinement through 6+ recursive steps without fixed-point convergence requirements.

Less is More: Recursive Reasoning with Tiny Networks

Core Concept

Complex reasoning tasks do not require billion-parameter language models. A single 2-layer network with 7M parameters can outperform much larger models through latent recursion—iteratively refining intermediate representations over 6+ steps without explicit chain-of-thought text. The key insight is that deep recursive passes combined with one gradient-enabled step per iteration enable "effective depth per supervision" rivaling multi-step transformers.

Architecture Overview

• Single 2-Layer Network: Minimal architecture replacing two 4-layer networks in prior work (Hierarchical Reasoning Model)
• Latent Recursion: n=6 iterations where hidden state z and answer y update via z = net(x,y,z); y = net(y,z)
• Deep Supervision Loop: T-1 gradient-free passes followed by one backprop-enabled pass, with early stopping via confidence threshold
• Problem-Specific Optimization: MLP for fixed-size problems (Sudoku, mazes); self-attention for variable grids (ARC-AGI)
• Minimal Training: ~1000 examples sufficient; EMA smoothing (0.999) and early stopping prevent overfitting

Implementation Steps

1. Architecture Design

The network maintains three key components: input embedding, iteratively refined answer, and latent reasoning state.

import torch
import torch.nn as nn

class TinyRecursiveModel(nn.Module):
    def __init__(self, input_dim=100, hidden_dim=128, output_dim=81, num_layers=2, use_attention=False):
        """
        Tiny Recursive Model (TRM): 2-layer network with latent recursion.

        Args:
            input_dim: Embedded question dimension
            hidden_dim: Latent state and hidden dimension
            output_dim: Answer dimension (e.g., 81 for 9x9 Sudoku)
            num_layers: Number of stacked layers (fixed at 2)
            use_attention: Use self-attention for variable-size grids (ARC-AGI)
        """
        super().__init__()

        self.input_dim = input_dim
        .hidden_dim = hidden_dim
        .output_dim = output_dim
        .use_attention = use_attention

        
         use_attention:
            
            encoder_layer = nn.TransformerEncoderLayer(
                d_model=hidden_dim, nhead=, dim_feedforward=, batch_first=
            )
            .backbone = nn.TransformerEncoder(encoder_layer, num_layers=)
        :
            
            .backbone = nn.Sequential(
                nn.Linear(input_dim + output_dim + hidden_dim, hidden_dim),
                nn.ReLU(),
                nn.Linear(hidden_dim, hidden_dim),
                nn.ReLU()
            )

        
        .latent_head = nn.Linear(hidden_dim, hidden_dim)    
        .answer_head = nn.Linear(hidden_dim + hidden_dim, output_dim)  
        .halt_head = nn.Linear(hidden_dim, )               

        .ema_momentum = 

     ():
        
        batch_size = x.size()

        
        y = torch.randn(batch_size, .output_dim, device=x.device)
        z = torch.zeros(batch_size, .hidden_dim, device=x.device)

        iteration_losses = []

         iteration  (max_iterations):
            
             _  (T - ):
                z_new = ._update_latent(x, y, z)
                y_new = ._update_answer(y, z_new)
                z = z_new
                y = y_new

            
            z = ._update_latent(x, y, z)
            y_updated = ._update_answer(y, z)

            
            confidence = torch.sigmoid(.halt_head(z)).mean()

             iteration > :  
                target_y = compute_ground_truth(x)  
                loss = nn.functional.cross_entropy(y_updated, target_y)
                iteration_losses.append(loss)

                
                loss.backward(retain_graph=(iteration < max_iterations - ))

            
            z = .ema_momentum * z + ( - .ema_momentum) * z_new

            
             confidence > confidence_threshold:
                

            y = y_updated

         y, torch.tensor(iteration_losses)

     ():
        
         .use_attention:
            combined = torch.cat([x, y], dim=)
            features = .backbone(combined)
        :
            combined = torch.cat([x, y, z], dim=)
            features = .backbone(combined)
         .latent_head(features)

     ():
        
        combined = torch.cat([y, z], dim=)
         .answer_head(combined)