gain-rl-angle-concentration

name	gain-rl-angle-concentration
title	Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signals
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.02281
keywords	["reinforcement-learning","data-selection","angle-concentration","efficiency","gradient-signals"]
description	Improve RL training efficiency by 2.5× using angle concentration between token hidden states as a cost-effective data scheduling signal, selecting high-gradient samples dynamically.

Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signals

Core Concept

GAIN-RL identifies a powerful cost-free signal hidden in every LLM: the geometric relationship between token hidden states (cosine similarity / "angle concentration"). The key discovery: models with concentrated angles between tokens exhibit larger gradient norms, indicating higher learning capacity for that example. Rather than expensive external metrics or heuristics, the framework uses intrinsic model signals to schedule training data dynamically. This enables 2.5× training speedup using only 50% of data while surpassing full-dataset performance, through three components: angle-based data ranking, Gaussian probability sampling, and dynamic curriculum adjustment.

The approach is remarkably simple yet effective: higher angle concentration → higher learning potential → prioritize early in training.

Architecture Overview

• Angle Concentration Metric: Cosine similarity between consecutive token hidden states as learning capacity signal
• Three Concentration Patterns: Layer-wise, epoch-wise, and data-wise patterns govern learning dynamics
• Data Reordering: Single offline pass ranking examples by combined intra/inter-segment angles
• Gaussian Probability Sampling: Curriculum learning starting high-concentration, transitioning to lower-concentration
• Dynamic Mean Shift: Progressively sample harder examples as training progresses

Implementation

1. Angle Concentration Calculation: Compute cosine similarity between token states

def compute_angle_concentration(model, example, device='cuda'):
    """
    Calculate intra-segment and inter-segment angle concentration.
    Angle concentration = cosine similarity between consecutive token hidden states.
    Higher concentration indicates higher learning potential.
    """
    # Forward pass to get hidden states
    with torch.no_grad():
        outputs = model(
            example['input_ids'].to(device),
            output_hidden_states=True
        )

    hidden_states = outputs.hidden_states  # [num_layers, seq_len, hidden_dim]

    angle_concentration = {}

    # Compute intra-segment angles (within reasoning segment)
    intra_angles = []
     layer_idx, layer_states  (hidden_states):
         i  ((layer_states) - ):
            
            h_i = layer_states[i]
            h_next = layer_states[i + ]

            
            h_i_norm = h_i / (torch.norm(h_i) + )
            h_next_norm = h_next / (torch.norm(h_next) + )

            
            cosine_sim = torch.dot(h_i_norm, h_next_norm).item()
            intra_angles.append(cosine_sim)

    angle_concentration[] = np.mean(intra_angles)

    
    
    answer_boundary = example.get(, (hidden_states[-]) // )

    inter_angles = []
     layer_idx, layer_states  (hidden_states):
         answer_boundary < (layer_states):
            
            reasoning_token = layer_states[answer_boundary - ]
            answer_token = layer_states[answer_boundary]

            reasoning_norm = reasoning_token / (torch.norm(reasoning_token) + )
            answer_norm = answer_token / (torch.norm(answer_token) + )

            cosine_sim = torch.dot(reasoning_norm, answer_norm).item()
            inter_angles.append(cosine_sim)

    angle_concentration[] = np.mean(inter_angles)  inter_angles  

    
    combined_signal =  * angle_concentration[] +  * angle_concentration[]

     {
        : angle_concentration[],
        : angle_concentration[],
        : combined_signal
    }