vespo-variational-sequence-rl

name	vespo-variational-sequence-rl
title	VESPO: Variational Sequence-Level Soft Policy Optimization
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.10693
keywords	["reinforcement learning","off-policy learning","LLM training","importance weighting","policy stability"]
description	Stabilize off-policy RL training for LLMs by deriving principled importance weight reshaping from variational optimization. Instead of heuristic clamping, VESPO uses closed-form exponential weighting W^c1 * exp(c2*(1-W)) to suppress overweighted samples while maintaining smooth gradients. Enables stable training at 64× policy staleness and under fully asynchronous execution with sequence-level operations that avoid length-dependent biases.

VESPO: Principled Off-Policy Weighting for Stable LLM RL

Large language model training with reinforcement learning often requires collecting interaction data from older policy versions due to computational constraints. This off-policy distribution shift—where training samples come from stale policies—destabilizes learning, causing gradient explosions or policy collapse. Standard approaches use heuristic importance weight functions (clipping, softmax) to suppress overweighted samples, but these lack principled justification and often require extensive hyperparameter tuning.

The challenge is discovering weight transformation functions that maintain training stability while preserving signal from informative samples. Existing methods operate at token level and apply length normalization, introducing biases where longer sequences receive different effective learning rates than short ones.

Core Concept

VESPO reframes importance weight transformation through measure-change perspective. Rather than manually designing weight functions, the method formulates an optimization problem seeking a proposal distribution that:

1. Remains efficient (close to the behavior distribution for importance-sampled learning)
2. Incorporates the target policy (reduces bias)
3. Constrains variance (bounds importance weight magnitudes)

The closed-form solution is an exponential reshaping kernel: W^c₁ × exp(c₂(1-W)), where W represents sequence-level importance weights. This provides smooth, principled suppression that scales correctly with off-policy staleness.

Architecture Overview

• Sequence-Level Importance: Compute importance weight per full sequence (not per-token), avoiding length-dependent biases
• Variational Optimization: Formulate weight transformation as constrained optimization problem over proposal distribution
• Exponential Reshaping: Apply closed-form kernel W^c₁ × exp(c₂(1-W)) to transform raw importance weights
• Smooth Gradient Flow: Exponential decay prevents hard clipping artifacts and enables stable backpropagation
• Asynchronous Compatible: Works with fully asynchronous data collection (no synchronization requirements)

Implementation

Compute sequence-level importance weights from policy log probabilities:

def compute_sequence_importance_weights(
    sequences, target_logprobs, behavior_logprobs, eps=1e-6
):
    """
    Compute importance weights at sequence level (not token level).
    sequences: (B, T) token ids
    target_logprobs: (B, T) log probs under current policy
    behavior_logprobs: (B, T) log probs under data collection policy
    Returns: (B,) importance weights
    """
    
    target_log_prob = target_logprobs.(dim=)  
    behavior_log_prob = behavior_logprobs.(dim=)  

    
    log_importance = target_log_prob - behavior_log_prob
    importance_weights = torch.exp(log_importance.clamp(-, ))

     importance_weights