nabla-reasoner

name	nabla-reasoner
title	∇-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.04948
keywords	["Inference Time Scaling","Reasoning","Gradient Descent","Reward Optimization","Latent Space Optimization"]
description	Improves LLM reasoning quality at inference time by optimizing token logits using gradient descent, combining reward model signals with KL-regularization. Bridges parametric training-time and non-parametric test-time scaling through token-level optimization.

∇-Reasoner: Achieving Better Reasoning Through Test-Time Gradient Descent on Token Logits

LLM reasoning during inference typically relies on sampling and rejection—generate multiple candidate responses and select based on reward. This zeroth-order search is inefficient in high-dimensional token spaces, especially for sparse reward landscapes. ∇-Reasoner reformulates inference-time reasoning as first-order optimization: instead of blind sampling, use gradients from reward and language models to refine token predictions during generation.

The key insight is that token logits are differentiable with respect to both the language model's likelihood and a learned reward function. By applying gradient descent to these logits, the model can iteratively improve reasoning quality while maintaining linguistic fluency through KL regularization.

Core Concept

Standard decoding: x_1, x_2, ... ~ π_θ(·|prefix) (sample from fixed policy)

∇-Reasoner: Optimize logits using gradient descent on:

L = -R(x) + λ KL(π_optimized || π_θ)

where R(x) is reward from a learned reward model and KL regularization keeps logits close to the base model's predictions. This combines two objectives: maximize rewards while staying faithful to the base language model.

The optimization operates in logit space (differentiable) rather than discrete token space, enabling gradient-based refinement. After optimization, resample tokens from improved logit distributions.

Architecture Overview

• Token Logit Optimization: Perform gradient descent on token logits at each generation step
• Dual Objective: Maximize reward while minimizing KL divergence from base model
• Iterative Refinement: Generate full trajectory, optimize, then resample with acceptance filtering
• Gradient Caching: Reuse gradients when token predictions stabilize to reduce computation
• Rollout Sharing: Leverage KV cache across steps to amortize cost

Implementation Steps

Implement gradient-based optimization of token logits combined with rejection sampling for conservative updates.

Base Gradient-Based Token Optimization

import torch
import torch.nn.functional as F

def optimize_token_logits_for_step(
    prompt,
    generated_prefix,
    logits_initial,
    reward_model,
    language_model,
    num_optimization_steps=5,
    learning_rate=,
    kl_weight=
):
    
    logits_opt = logits_initial.clone().detach().requires_grad_()
    optimizer = torch.optim.Adam([logits_opt], lr=learning_rate)

    
     torch.no_grad():
        logits_ref = language_model.get_logits(prompt, generated_prefix)

    best_loss = ()
    best_logits = logits_opt.clone().detach()

     step  (num_optimization_steps):
        optimizer.zero_grad()

        
        probs = F.softmax(logits_opt, dim=)
        token_sampled = torch.multinomial(probs, num_samples=).item()

        
        candidate_seq = torch.cat([prompt, generated_prefix, torch.tensor([token_sampled])])

        
         probs[token_sampled] > :  
             torch.enable_grad():
                reward = reward_model(candidate_seq)
                reward_loss = -reward  
        :
            reward_loss = torch.tensor(, device=logits_opt.device)

        
        kl_div = torch.(
            F.softmax(logits_ref, dim=) * (F.log_softmax(logits_ref, dim=) - F.log_softmax(logits_opt, dim=))
        )

        
        loss = reward_loss + kl_weight * kl_div

        
        loss.backward()
        optimizer.step()

         loss.item() < best_loss:
            best_loss = loss.item()
            best_logits = logits_opt.clone().detach()

     best_logits,   


 ():
    
    probs_opt = F.softmax(logits_optimized, dim=)
    probs_orig = F.softmax(logits_original, dim=)

    
    token_candidate = torch.multinomial(probs_opt, num_samples=).item()

    
    acceptance_prob = (, probs_opt[token_candidate].item() / (probs_orig[token_candidate].item() + ))

    
    accepted = torch.rand().item() < acceptance_prob

     token_candidate, accepted