soft-grpo-soft-thinking-rl-lms

name	soft-grpo-soft-thinking-rl-lms
title	SofT-GRPO: Surpassing Discrete-Token LLM RL via Gumbel-Reparameterized Soft-Thinking
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.06411
keywords	["Reinforcement Learning","Soft Tokens","Policy Gradients","LLM Training","Gumbel Reparameterization"]
description	Enable policy gradient optimization on soft LLM tokens by injecting Gumbel noise and applying Gumbel-Softmax reparameterization—allowing soft-thinking patterns to match discrete-token RL performance while maintaining continuous optimization advantages.

Optimize Soft-Thinking LLMs with Gumbel-Reparameterized Policy Gradients

Soft-thinking enables LLMs to decompose reasoning through continuous internal tokens rather than discrete outputs. However, applying reinforcement learning to soft tokens has been problematic—naive approaches produce out-of-distribution tokens or fail to optimize effectively. SofT-GRPO bridges this gap through three innovations: Gumbel noise injection, Gumbel-Softmax reparameterization, and policy gradient adaptation.

The result is that soft-thinking patterns can now match or exceed discrete-token RL performance. On reasoning tasks, soft-thinking achieves +0.13% improvement on single-attempt metrics and substantial +2.19% gains on multi-attempt evaluation, closing a previously intractable performance gap.

Core Concept

SofT-GRPO treats soft token generation as a continuous optimization problem solvable via policy gradients. The key insight is that standard RL fails because soft tokens drift out of the pre-trained embedding space. By injecting Gumbel noise and applying the Gumbel-Softmax trick, we maintain tokens within the learned embedding space while enabling full differentiability for policy optimization.

The architecture combines soft-thinking internal reasoning with group relative policy optimization (GRPO), enabling agents to develop deeper reasoning patterns while maintaining the benefits of token-level optimization.

Architecture Overview

• Soft Token Generator: Produces continuous representations instead of discrete tokens
• Gumbel Noise Injection: Adds Gumbel-distributed noise to logits for stochasticity
• Gumbel-Softmax Layer: Maps noisy logits to embedded space while preserving differentiability
• Reparameterization Trick: Enables gradients to flow through the sampling process
• GRPO Loss Module: Computes policy gradients using relative rewards across rollouts
• Embedding Space Validation: Ensures generated tokens remain in pre-trained vocabulary space

Implementation Steps

Step 1: Gumbel Noise Injection

Add Gumbel-distributed noise to logits to introduce stochasticity required for exploration in policy learning.

import torch
import torch.nn.functional as F

def gumbel_noise(shape, device, eps=1e-20):
    """
    Sample Gumbel(0, 1) noise.

    Args:
        shape: Tensor shape for noise
        device: torch device
        eps: Small value for numerical stability

    Returns:
        gumbel_samples: Gumbel-distributed noise
    """
    
    uniform = torch.rand(shape, device=device)

    
    gumbel = -torch.log(-torch.log(uniform + eps) + eps)
     gumbel

 ():
    
    gumbel = gumbel_noise(logits.shape, logits.device)
    noisy_logits = logits + gumbel
     noisy_logits / temperature