info-driven-policy-optimization-agents

name	info-driven-policy-optimization-agents
title	InfoPO: Information-Driven Policy Optimization for User-Centric Agents
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.00656
keywords	["Multi-Turn Agents","Information Gain","Credit Assignment","Reinforcement Learning","User Interaction"]
description	Optimize multi-turn agent policies by measuring turn-level information gain via counterfactual reasoning. Provide dense reward signals identifying which clarifying questions and observations improve the agent's decision distribution, then adaptively blend information rewards with outcome rewards.

InfoPO: Information-Driven Policy Optimization for Multi-Turn Agents

Sparse outcome rewards make early-stage multi-turn agent training inefficient. Standard GRPO-based methods struggle to credit valuable clarification steps that improve downstream decisions but don't directly affect final task success. InfoPO solves this through turn-level counterfactual information-gain rewards that identify which observations genuinely shift the agent's action distribution.

The core insight is to measure how much each observation changes what the agent would do next, then reward observations that drive meaningful behavioral shifts. This approach provides dense supervision during exploration and automatically down-weights information once discriminative outcome signals emerge.

Core Concept

InfoPO treats each interaction turn as a decision point with two scenarios:

1. Factual: The agent receives real feedback; its action distribution reflects both prior knowledge and the new observation
2. Counterfactual: The feedback is masked; the agent relies only on prior context

The information gain is the KL divergence between these two distributions—a principled measure of how much the observation matters to the agent's thinking.

The method then uses adaptive variance gating to balance two reward sources:

• Early training: Information gain drives learning when outcome signals are sparse
• Late training: Outcome rewards dominate as discriminative feedback emerges

Architecture Overview

• Input: Multi-turn trajectory with observations at each step {o₁, o₂, ..., oₜ}
• Counterfactual Masking: For each step, create masked version with observation set to null/padding
• Policy Evaluation: Run forward passes to compute action distributions with real and masked observations
• Information Gain Computation: Compute KL divergence between the two distributions
• Variance Gating: Blend information and outcome rewards based on outcome signal strength
• Output: Dense per-turn rewards for policy optimization

Implementation Steps

Step 1: Prepare factual and counterfactual trajectories

For each interaction turn, create two versions of the context: one with the real observation and one with it masked.

def prepare_counterfactual_pair(trajectory, turn_idx):
    
    
    factual_context = trajectory[:turn_idx + ]

    
    counterfactual_context = trajectory[:turn_idx]
    
    counterfactual_context.append(mask_observation(trajectory[turn_idx]))

     factual_context, counterfactual_context


trajectory = [context, obs1, obs2, obs3, obs4, obs5]
pairs = [
    prepare_counterfactual_pair(trajectory, t)
     t  (, (trajectory))
]