config-agents

name	config-agents
title	Learning to Configure Agentic AI Systems
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.11574
keywords	["Agent Configuration","Hierarchical RL","Reinforcement Learning","Hyperparameter Optimization","System Design"]
description	Learn optimal configurations for agentic AI systems through hierarchical RL that treats configuration as a query-wise decision problem. Structure policy selects workflows/tools/budgets while prompt policy composes specific instructions, achieving 25% accuracy improvement with 35% cost reduction.

Learning to Configure Agentic AI Systems

Problem Context

Agentic AI systems have enormous configuration spaces: tool selection, workflow choice, computational budget, prompt templates, reasoning strategies, etc. Over 8,600 possible structural configurations exist in simple systems. Manual tuning is infeasible. Static heuristics fail across different query types. Different queries benefit from different configurations (some need web search, others need local reasoning; some need 1-step, others need 5-step reasoning). The challenge: learn to configure agents dynamically per query while balancing accuracy and cost.

Core Concept

ARC (Agentic Resource & Configuration learner) treats configuration as a query-wise decision problem solved through hierarchical RL. A two-level policy system:

1. Structure Policy: High-level decisions

   • Which workflows (sequential, hierarchical, parallel)
   • Which tools (web search, calculator, code executor)
   • Computational budgets (tokens, steps, time)

2. Prompt Policy: Low-level decisions

   • Specific task instructions
   • CoT strategies
   • Output format constraints

Rather than exhaustive grid search, masked RL with SFT on successful trajectories enables efficient learning under sparse rewards.

Architecture Overview

• Hierarchical Policy Structure: Two-level decision hierarchy
• Structure Policy: Workflow/tool/budget selection
• Prompt Policy: Instruction composition and constraints
• Action Masking: Prevent invalid action combinations
• Sparse Reward Signal: Only on task completion
• Masked RL: PPO variant with invalid action masking
• Supervised Fine-Tuning: Learn from successful trajectories
• Query-Wise Adaptation: Per-query configuration decisions

Implementation

Hierarchical policy architecture:

class HierarchicalConfigurationPolicy(nn.Module):
    """
    Two-level policy: structure policy + prompt policy.
    Structure decides what capabilities to use.
    Prompt decides how to ask for them.
    """

    def __init__():
        ().__init__()
        .model = model
        .tools = tool_library
        .workflows = workflow_library

        
        .structure_policy = nn.Sequential(
            nn.Linear(, ),
            nn.ReLU(),
            nn.Linear(, )
        )

        
        .workflow_head = nn.Linear(, (workflow_library))

        
        .tool_head = nn.Linear(, (tool_library))

        
        .budget_head = nn.Linear(, )  

        
        .prompt_policy = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(, ),
            num_layers=
        )

     ():
        
        
        query_features = .model.encode(query)

        
        structure_hidden = .structure_policy(query_features)

        
        workflow_logits = .workflow_head(structure_hidden)
        tool_logits = .tool_head(structure_hidden)
        budget_logits = .budget_head(structure_hidden)

        
        workflow = torch.softmax(workflow_logits, dim=-)
        tools = torch.sigmoid(tool_logits)  
        budget = torch.softmax(budget_logits, dim=-)

         {
            : workflow,
            : tools,
            : budget,
            : {
                : workflow_logits,
                : tool_logits,
                : budget_logits
            }
        }

     ():
        
        workflow = structure_decision[].argmax()
        tools = structure_decision[]
        budget = structure_decision[].argmax()

        
        constraints = {
            : [, ],  
            : [],  
            : []  
        }

        
        mask = torch.ones_like(tools)

         tool_idx, tool_name  (.tools.keys()):
            compatible_workflows = constraints.get(tool_name, [])

            
             compatible_workflows:
                workflow_name = .workflows[workflow]
                 workflow_name   compatible_workflows:
                    mask[tool_idx] = 

         mask

     ():
        
        
        workflow = structure_decision[].argmax()
        selected_tools = structure_decision[].nonzero(as_tuple=)[]
        budget_level = structure_decision[].argmax()

        
        prompt_parts = []

        
        workflow_name = (.workflows.keys())[workflow]
         workflow_name == :
            prompt_parts.append(
                )
         workflow_name == :
            prompt_parts.append(
                )
         workflow_name == :
            prompt_parts.append(
                )

        
         tool_idx  selected_tools:
            tool_name = (.tools.keys())[tool_idx]
            tool_instruction = .tools[tool_name][]
            prompt_parts.append()

        
        budget_values = [, , , , , , ,
                        , , ]
        max_tokens = budget_values[budget_level]
        prompt_parts.append(
            )

        
        prompt_parts.append()

        composed_prompt = .join(prompt_parts)

         {
            : composed_prompt,
            : workflow_name,
            : [(.tools.keys())[i]  i  selected_tools],
            : budget_values[budget_level]
        }

     ():
        
        structure = .structure_decision(query)
        mask = .action_masking(structure)
        prompt = .prompt_decision(query, structure)

         {
            : structure,
            : mask,
            : prompt
        }