bottom-up-policy

name	bottom-up-policy
title	Bottom-up Policy Optimization: Internal Policies in LMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.19673
keywords	["reinforcement-learning","interpretability","layer-wise","reasoning","llm"]
description	Optimize language model policies layer-by-layer rather than monolithically to understand internal reasoning structure. Decompose models into per-layer and per-module policies via residual streams, analyze entropy patterns revealing exploration→convergence phases, and optimize layers sequentially—improving reasoning on math tasks by up to 4.69 points.

Overview

Bottom-up Policy Optimization (BuPO) treats language models as compositional reasoning systems rather than monolithic policies. By analyzing internal layer policies via residual streams, the framework reveals that models naturally exhibit a universal structure: early layers explore solution spaces while top layers converge to predictions. Sequential optimization respects this structure.

Core Technique

The key insight is that residual streams enable additive decomposition of layer and module policies.

Internal Policy Decomposition: Define policies at different architectural levels using hidden states and the unembedding matrix.

# Layer and module policy definition
class InternalPolicyDecomposition:
    def __init__(self, model):
        self.model = model
        self.num_layers = len(model.layers)
        self.unembedding = model.unembedding

    def layer_policy(self, layer_idx):
        """
        Define policy for individual layer via its residual contribution.
        Policy: hidden_state @ unembedding → logits
        """
        def pi_layer(residual_stream, target_idx):
            # Extract this layer's residual contribution
            layer_output = residual_stream[layer_idx]
            # Convert to logits via unembedding
            logits = layer_output @ self.unembedding.weight
            return logits
        return pi_layer

    def module_policy(self, layer_idx, module_type):
        """
        Define policy for individual module (attention vs FFN).
        Isolate each module's contribution to reasoning.
        """
        if module_type == 'attention':
              x: .model.layers[layer_idx].self_attn(x)
         module_type == :
              x: .model.layers[layer_idx].mlp(x)