bavt-budget-aware-search

name	bavt-budget-aware-search
title	Spend Less, Reason Better: Budget-Aware Value Tree Search for LLM Agents
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.12634
keywords	["Tree Search","Budget-Aware","Value Estimation","Multi-Hop Reasoning","LLM"]
description	Allocate LLM reasoning budget optimally via value tree search: use residual value prediction to estimate step utility, then dynamically shift exploration-exploitation balance as budget depletes. Outperform high-budget baselines at 1/4 cost.

Technique: Budget-Conditioned Value Tree Search with Dynamic UCB

Agentic reasoning requires exploring multiple paths, but computational budgets are finite. Budget-Aware Value Tree (BAVT) search makes principled allocation decisions: it estimates marginal utility per step using residual value prediction, then dynamically modulates exploration strength as budget depletes.

The key insight is a power-law scaling exponent inversely proportional to remaining budget, creating smooth transitions from broad exploration to aggressive greedy exploitation.

Core Concept

BAVT operates through three mechanisms:

1. Step-Level Value Estimation: Critics predict marginal progress (residual value) rather than absolute trajectory value
2. Budget-Conditioned Node Selection: Dynamic UCB scaling (αt = 1/rt) where rt is remaining budget
3. Training-Free Dual-Role LLM: Single model alternates between generator and critic roles

This achieves superior performance under budget constraints: 4× budget-efficient compared to baseline.

Architecture Overview

• Generator role: Proposes next reasoning steps
• Critic role: Estimates residual value (marginal progress)
• Value predictor: MLP head for step-level value
• Tree search engine: Maintains search tree with UCB-based selection
• Budget manager: Tracks and allocates remaining compute

Implementation Steps

Step 1: Residual Value Prediction

Critic estimates incremental progress, not absolute returns.

import torch
import torch.nn as nn

class ResidualValuePredictor(nn.Module):
    def __init__(self, hidden_dim=768):
        super().__init__()
        self.hidden_dim = hidden_dim

        # MLP for residual value prediction
        self.predictor = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim * 2),
            nn.ReLU(),
            nn.Linear(hidden_dim * 2, )  
        )

     ():
        
        residual_value = .predictor(context_embeddings)

         residual_value

 (nn.Module):
     ():
        ().__init__()
        .model = model
        .value_predictor = ResidualValuePredictor(hidden_dim)

     ():
        
        
        hidden_state = .model.encode(trajectory_text)  

        
        residual = .value_predictor(hidden_state)

         residual.item()