looped-elastic-depth-transformers

name	looped-elastic-depth-transformers
title	LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.11451
keywords	["Transformer Architecture","Elastic Depth","Looped Processing","Budget-Conditioned Reasoning","Self-Distillation"]
description	Enable budget-conditioned reasoning by repeatedly applying a shared transformer block stack with trajectory-based conditioning on time and step size. Train via shortcut-consistency loss to align shorter and full-length trajectories, enabling variable-depth inference without retraining.

LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning

Problem Context

Transformers with fixed depth have fixed computation. Extended reasoning requires more depth, but retraining is expensive. LoopFormer enables variable-depth inference by repeatedly applying the same transformer blocks, with trajectories conditioned on normalized time (0 to 1 over steps) and step size. Shorter trajectories remain informative while longer ones refine.

Core Concept

LoopFormer uses: (1) a shared stack of K transformer blocks applied repeatedly, (2) trajectory-based conditioning via time t ∈ [0,1] and step size Δt indicating position in loop, (3) shortcut-consistency training that aligns trajectories of different lengths, (4) AdaLN-style modulation of attention/FFN residual strengths.

Implementation

Step 1: Trajectory-based conditioning

import torch
import torch.nn as nn
import math
from typing import Tuple

class TrajectoryConditioner:
    """Encode time and step size for loop iteration."""

    def __init__(self, dim: int = 768):
        self.dim = dim

    def encode_trajectory_position(
        self,
        iteration_idx: int,
        total_iterations: int,
        step_size: float = None
    ) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Encode normalized time t and step size Δt.

        Args:
            iteration_idx: Current loop iteration (0-based)
            total_iterations: Total number of iterations
            step_size: Size of current step (fraction of sequence)

        Returns:
            (time_embedding, step_embedding): [dim] tensors
        """
        # Normalized time: 0 to 1 over course of all iterations
        t = iteration_idx / max(1, total_iterations - )

        
         step_size  :
            step_size =  / total_iterations

        
        time_embedding = ._sinusoidal_encode(t)
        step_embedding = ._sinusoidal_encode(step_size)

         time_embedding, step_embedding

     () -> torch.Tensor:
        
        embedding = torch.zeros(.dim)

         i  (, .dim, ):
            omega =  / ( ** (i / .dim))
             i < .dim:
                embedding[i] = math.sin(value * omega)
             i +  < .dim:
                embedding[i + ] = math.cos(value * omega)

         embedding

     () -> torch.Tensor:
        
        combined = time_embedding + step_embedding  

        
        
        scale =  +  * combined.mean()
        bias =  * combined.mean()

         hidden_states * scale + bias