at2po-agentic-tree-search-optimization

name	at2po-agentic-tree-search-optimization
title	AT²PO: Agentic Turn-based Policy Optimization via Tree Search
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.04767
keywords	["Agent Reinforcement Learning","Tool Use","Tree Search","Multi-Hop Reasoning"]
description	Optimize multi-turn agent policies via entropy-guided tree expansion and turn-level credit assignment. AT²PO addresses exploration diversity, sparse credit signal, and policy misalignment problems in LLM agents through structured tree search and turn-aware policy updates.

When to Use This Skill

• Multi-turn agent tasks requiring tool interaction (web search, knowledge bases)
• Multi-hop reasoning where information gathering spans multiple steps
• Sparse reward scenarios where only final task completion provides feedback
• Agents making discrete tool-calling decisions (HotpotQA, WikiQA, retrieval tasks)
• Environments with 3-6 tool calls per trajectory

When NOT to Use This Skill

• Single-turn decision making (overkill for one-step tasks)
• Continuous action spaces (method designed for discrete tool calling)
• Scenarios with dense intermediate rewards (tree search provides marginal benefit)
• Real-time systems where planning overhead is prohibitive

Problem Summary

Multi-turn agent reinforcement learning faces three critical challenges: (1) limited exploration diversity when policy entropy is low, (2) sparse credit assignment where only final success provides feedback (no signal for intermediate steps), and (3) misaligned policy optimization—token-level policy updates may not reflect the turn-level decisions agents actually make. These problems compound in multi-hop reasoning where agents must gather information across multiple tool calls before arriving at answers.

Solution: AT²PO Tree Search + Turn-Level Policy Framework

Combine entropy-guided tree exploration with turn-aware policy optimization that aligns updates to actual agent decision structure.

class AT2PO:
    def __init__(self, model, tree_depth=2, branching_factor=10):
        self.model = model
        self.tree_depth = tree_depth
        self.max_branches = branching_factor

    def entropy_guided_tree_expansion(self, root_state, num_iterations=2):
        """Expand tree from uncertain turns to promote diverse exploration"""
        expanded_nodes = []

        for iteration in range(num_iterations):
            # Score all leaf nodes by policy entropy
            leaf_scores = []
             leaf  .tree.leaves:
                action_logits = .model(leaf.state)
                entropy = compute_entropy(action_logits)
                
                score = entropy - BRANCHING_PENALTY * leaf.depth
                leaf_scores.append((leaf, score))

            
            K = (, (leaf_scores))
            selected_leaves = (leaf_scores, key= x: x[], reverse=)[:K]

            
             leaf, _  selected_leaves:
                
                 _  (.max_branches):
                    new_trajectory = .sample_continuation(leaf)
                    expanded_nodes.append(new_trajectory)

         expanded_nodes

     ():
        
        node_values = {}

        
         node  (tree.nodes):
             node.is_leaf:
                
                node_values[node] = node.reward
            :
                
                descendant_rewards = [
                    node_values[child]  child  node.children
                ]
                entropy_weights = [
                    compute_entropy(child.action_logits)
                     child  node.children
                ]
                
                node_values[node] = weighted_average(
                    descendant_rewards, entropy_weights
                )

         node_values

     ():
        
        turn_losses = []

         trajectory  trajectories:
             turn_idx, turn  (trajectory.turns):
                
                turn_tokens = turn.tokens
                turn_old_logprobs = turn.old_log_probs

                
                turn_advantage = node_values[turn.end_node] - baseline(turn.start_node)

                
                turn_ratio = torch.exp(
                    turn.new_log_probs - turn_old_logprobs
                )
                clipped_ratio = torch.clamp(
                    turn_ratio,  - CLIP_EPS,  + CLIP_EPS
                )

                
                turn_loss = -torch.(
                    turn_ratio * turn_advantage,
                    clipped_ratio * turn_advantage
                )
                turn_losses.append(turn_loss)

         torch.mean(torch.cat(turn_losses))