sr-grpo-stable-rank

name	sr-grpo-stable-rank
title	SR-GRPO: Stable Rank as Intrinsic Reward for LLM Alignment
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.02807
keywords	["llm-alignment","reinforcement-learning","intrinsic-rewards","representation-geometry","grpo"]
description	Uses stable rank (variance distribution across semantic dimensions) as annotation-free reward signal in GRPO to incentivize high-dimensional representation structures, eliminating dependency on human preference annotations or learned reward models.

Summary

SR-GRPO introduces stable rank as an intrinsic, annotation-free reward signal for LLM alignment. Stable rank measures how information distributes across semantic dimensions in hidden representations. The approach leverages this geometric property in reinforcement learning to guide policy optimization, rewarding responses that maintain higher-dimensional representational structure rather than collapsing into narrow activation patterns.

Core Technique

Stable Rank Definition: Measures the ratio of total variance to dominant-direction variance:

SR(X) = (sum of all eigenvalues)² / (sum of squared eigenvalues)

Higher stable rank indicates information spread across dimensions; lower indicates collapse.

Intrinsic Reward Signal: Compute stable rank of hidden states at each generation step:

reward_t = stable_rank(hidden_states_t) - stable_rank(hidden_states_ref)

Encourage diversity of representations without external labels.

SR-GRPO Integration: Add stable rank to group relative policy optimization:

total_reward = task_reward + λ_sr * sr_reward

Implementation

Stable rank computation:

def compute_stable_rank(X):
    # X: [batch, seq_len, hidden_dim]
    # Flatten to [N, hidden_dim]
    X_flat = X.reshape(-1, X.shape[-1])

    # SVD or eigendecomposition
    _, singular_values, _ = torch.svd(X_flat)
    sv_normalized = singular_values / singular_values.sum()

    # Stable rank formula
    sr = (sv_normalized.sum() ** 2) / (sv_normalized ** 2).sum()
    return sr

Reference model setup: Use frozen reference model for baseline:

reference_model = load_pretrained_model()
reference_model.eval()
with torch.no_grad():
    ref_hidden = reference_model(input_ids)
    ref_sr = compute_stable_rank(ref_hidden)