hcapo-hindsight-credit-assignment

name	hcapo-hindsight-credit-assignment
title	Hindsight Credit Assignment for Long-Horizon LLM Agents
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.08754
keywords	["Credit Assignment","RL","Long-Horizon","LLM","Agents","GRPO"]
description	Compute step-level credit assignments via hindsight generative verification: condition the LLM on successful outcomes to compute importance ratios that filter credit by causal relevance. Synergizes macro stability with micro precision.

Technique: Hindsight Importance Ratios for Causal Step-Level Credit

Long-horizon tasks suffer from credit assignment myopia: trajectory-level rewards don't reveal which intermediate steps were responsible for success. HCAPO addresses this by using the LLM itself as a post-hoc critic: it conditions on successful outcomes to estimate hindsight importance ratios that amplify credit for causally relevant steps while suppressing less instrumental ones.

Rather than training separate value models, this method leverages self-normalized importance sampling with the base LLM, making it training-free and readily integrated into existing RLVR frameworks like GRPO.

Core Concept

The key innovation is computing hindsight Q-values by:

1. Self-Conditioning on Success: Inject the successful outcome into the LLM's prompt context
2. Importance Ratio Estimation: Measure how step probability changes when conditioned on success
3. Causal Filtering: Steps whose probability increases given success are causally responsible
4. Multi-Scale Integration: Combine macro-level trajectory rewards with micro-level Q-values

This produces step-wise credit signals grounded in the model's own conditional distributions, avoiding external value function errors.

Architecture Overview

• Base LLM policy: Unchanged from baseline RLVR method
• Hindsight conditional: Re-prompt the model with outcome information
• Importance ratio computation: Token-level log-probability deltas
• Self-normalized importance sampling: Aggregate step-level credits
• Multi-scale reward aggregation: Blend trajectory and step signals

Implementation Steps

Step 1: Compute Self-Normalized Importance Ratios

For each step, estimate how probability changes when conditioned on successful outcome.

import torch
import torch.nn.functional as F

def compute_hindsight_importance_ratios(
    model,
    trajectory,
    input_ids,
    successful_outcome
):
    """
    Compute importance ratios ρ for each step in trajectory.

    trajectory: list of (token_id, log_prob) tuples
    input_ids: original input token sequence
    successful_outcome: target outcome to condition on
    """
    trajectory_length = (trajectory)
    importance_ratios = []

    
    original_log_probs = torch.tensor(
        [log_prob  _, log_prob  trajectory],
        dtype=torch.float32
    )

    
    
    hindsight_input = torch.cat([
        input_ids,
        torch.tensor([[model.tokenizer.encode(successful_outcome)]])
    ], dim=)

     torch.no_grad():
        hindsight_outputs = model(hindsight_input)
        hindsight_logits = hindsight_outputs.logits

    
    hindsight_log_probs = []
     t, (token_id, _)  (trajectory):
        logits = hindsight_logits[, input_ids.shape[] + t]
        log_prob = F.log_softmax(logits, dim=-)[token_id]
        hindsight_log_probs.append(log_prob.item())

    hindsight_log_probs = torch.tensor(hindsight_log_probs, dtype=torch.float32)

    
    
    log_ratios = hindsight_log_probs - original_log_probs
    importance_ratios = torch.exp(torch.clamp(log_ratios, =-, =))

     importance_ratios