| name | at2po-agentic-tree-search-optimization |
| title | AT²PO: Agentic Turn-based Policy Optimization via Tree Search |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2601.04767 |
| keywords | ["Agent Reinforcement Learning","Tool Use","Tree Search","Multi-Hop Reasoning"] |
| description | Optimize multi-turn agent policies via entropy-guided tree expansion and turn-level credit assignment. AT²PO addresses exploration diversity, sparse credit signal, and policy misalignment problems in LLM agents through structured tree search and turn-aware policy updates. |
When to Use This Skill
- Multi-turn agent tasks requiring tool interaction (web search, knowledge bases)
- Multi-hop reasoning where information gathering spans multiple steps
- Sparse reward scenarios where only final task completion provides feedback
- Agents making discrete tool-calling decisions (HotpotQA, WikiQA, retrieval tasks)
- Environments with 3-6 tool calls per trajectory
When NOT to Use This Skill
- Single-turn decision making (overkill for one-step tasks)
- Continuous action spaces (method designed for discrete tool calling)
- Scenarios with dense intermediate rewards (tree search provides marginal benefit)
- Real-time systems where planning overhead is prohibitive
Problem Summary
Multi-turn agent reinforcement learning faces three critical challenges: (1) limited exploration diversity when policy entropy is low, (2) sparse credit assignment where only final success provides feedback (no signal for intermediate steps), and (3) misaligned policy optimization—token-level policy updates may not reflect the turn-level decisions agents actually make. These problems compound in multi-hop reasoning where agents must gather information across multiple tool calls before arriving at answers.
Solution: AT²PO Tree Search + Turn-Level Policy Framework
Combine entropy-guided tree exploration with turn-aware policy optimization that aligns updates to actual agent decision structure.
class AT2PO:
def __init__(self, model, tree_depth=2, branching_factor=10):
self.model = model
self.tree_depth = tree_depth
self.max_branches = branching_factor
def entropy_guided_tree_expansion(self, root_state, num_iterations=2):
"""Expand tree from uncertain turns to promote diverse exploration"""
expanded_nodes = []
for iteration in range(num_iterations):
leaf_scores = []
leaf .tree.leaves:
action_logits = .model(leaf.state)
entropy = compute_entropy(action_logits)
score = entropy - BRANCHING_PENALTY * leaf.depth
leaf_scores.append((leaf, score))
K = (, (leaf_scores))
selected_leaves = (leaf_scores, key= x: x[], reverse=)[:K]
leaf, _ selected_leaves:
_ (.max_branches):
new_trajectory = .sample_continuation(leaf)
expanded_nodes.append(new_trajectory)
expanded_nodes
():
node_values = {}
node (tree.nodes):
node.is_leaf:
node_values[node] = node.reward
:
descendant_rewards = [
node_values[child] child node.children
]
entropy_weights = [
compute_entropy(child.action_logits)
child node.children
]
node_values[node] = weighted_average(
descendant_rewards, entropy_weights
)
node_values
():
turn_losses = []
trajectory trajectories:
turn_idx, turn (trajectory.turns):
turn_tokens = turn.tokens
turn_old_logprobs = turn.old_log_probs
turn_advantage = node_values[turn.end_node] - baseline(turn.start_node)
turn_ratio = torch.exp(
turn.new_log_probs - turn_old_logprobs
)
clipped_ratio = torch.clamp(
turn_ratio, - CLIP_EPS, + CLIP_EPS
)
turn_loss = -torch.(
turn_ratio * turn_advantage,
clipped_ratio * turn_advantage
)
turn_losses.append(turn_loss)
torch.mean(torch.cat(turn_losses))
