emergent-temporal-abstraction

name	emergent-temporal-abstraction
title	Emergent Temporal Abstractions in AR Models for Hierarchical RL
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.20605
keywords	["reinforcement-learning","hierarchy","autoregressive","temporal-abstraction","internal-rl"]
description	Discover hierarchical temporal abstractions within autoregressive models via internal RL, enabling efficient exploration of sparse-reward tasks. Metacontroller learns abstract action sequences modifying residual streams, switching gates enable quasi-binary patterns, and abstract-space RL achieves many orders-of-magnitude speedup over token-level learning.

Overview

This technique enables autoregressive models to learn hierarchical behaviors through discovery of temporal abstractions, dramatically accelerating learning on sparse-reward tasks.

Core Technique

Internal RL with Discovered Abstractions:

class HierarchicalARModel:
    def __init__(self):
        self.base_ar_model = PretrainedAutoregressive()
        self.metacontroller = MetacontrollerPolicy()
        self.abstract_controllers = nn.ModuleList()

    def forward_hierarchical(self, state):
        # Metacontroller generates abstract action sequence
        abstract_actions = self.metacontroller.sample_actions(state)

        # Each abstract action is a sequence of residual stream modifications
        output = self.base_ar_model.initial_forward(state)

        for t, abstract_action in enumerate(abstract_actions):
            # Apply abstract action via residual stream modification
            controller_output = self.abstract_controllers[abstract_action](output)
            output = output + controller_output  # Residual addition
            
            # Check switching condition
            if self.should_switch(output, t):
                break  # Move to next abstract action

        return output

Switching Gates and Temporal Patterns:

def switching_gate_mechanism(features, temperature=1.0):
    """
    Binary switching via gating, creating sparse temporal patterns.
    """
    gate_logits = nn.Linear(hidden_dim, )(features)
    gate_prob = sigmoid(gate_logits / temperature)
    
    
    gate_sample = gumbel_softmax(gate_prob)
    
     gate_sample