multi-agent-tool-policy-opt

name	multi-agent-tool-policy-opt
title	Multi-Agent Tool-Integrated Policy Optimization: Single LLM with Role-Specific RL Training
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.04678
keywords	["multi-agent","reinforcement-learning","tool-use","credit-assignment","prompt-engineering"]
description	Train planner and worker agent roles within a single LLM via role-specific prompts and RL, avoiding multi-instance overhead while preserving specialization. Trigger: improve tool-use planning robustness to noisy outputs without deploying separate models.

Multi-Agent Tool-Integrated Policy Optimization (MATPO)

Core Concept

MATPO enables a single LLM instance to embody both planner and worker agent roles through specialized prompts and reinforcement learning. Rather than deploying multiple LLM instances (expensive), the framework trains role-specific behaviors within one model using credit assignment across planner and worker rollouts. This maintains specialization benefits while reducing memory and compute overhead.

The key insight: A single model can learn distinct reasoning patterns when prompted appropriately and trained with role-aware reward signals.

Architecture Overview

• Dual Roles in Single Model: Planner (strategic reasoning, tool selection) and Worker (execution, tool invocation)
• Role-Specific Prompts: Distinct system prompts guide behavior for each role
• Unified RL Training: Both roles optimized simultaneously via shared parameters
• Credit Assignment Mechanism: Principled reward flow from final outcomes to planner and worker decisions
• Tool Response Integration: Framework handles noisy/partial tool outputs robustly

Implementation Steps

1. Design Role-Specific Prompts

Create distinct system prompts that guide the model toward planner vs. worker behavior. The prompts encode role expectations without requiring separate model instances.

PLANNER_PROMPT = """You are a strategic planner agent. Your role:
1. Analyze the user request and decompose into sub-tasks
2. Decide which tools are needed and in what order
3. Specify parameters for each tool call
4. Handle errors from worker and re-plan if needed

Format your response as: PLAN: <step1> | PLAN: <step2> | ...
Then output: DELEGATE: <tool_call_json>"""

WORKER_PROMPT = """You are a worker agent executing tool calls. Your role:
1. Receive a tool call specification from the planner
2. Execute the tool with exact parameters
3. Process the output and report results
4. Flag errors for planner to handle

Format your response as: EXECUTING: <tool_name> | RESULT: <output>
If error occurs: ERROR: <error_type> with recommendation for replanning."""

2. Implement Rollout Collection with Role Switching

Collect training data by rolling out the model in both roles. The key is tracking which role is active during each decision for proper credit assignment.

class DualAgentRollout:
    def __init__():
        .model = model
        .tools = tools_registry

     ():
        trajectory = {
            : [],
            : [],
            : [],
            : []
        }

        context = 
        planner_history = 
        worker_history = 

         turn  (max_turns):
            
            planner_output = .model.generate(
                context + PLANNER_PROMPT + planner_history,
                max_tokens=
            )

            
            tool_calls = parse_tool_calls(planner_output)
            trajectory[].append({
                : turn,
                : planner_output,
                : tool_calls
            })

            
            worker_results = []
            worker_output = 

             tool_call  tool_calls:
                :
                    result = .tools[tool_call[]](
                        **tool_call[]
                    )
                    worker_output += 
                    worker_output += 
                    worker_results.append(result)
                 Exception  e:
                    worker_output += 

            trajectory[].append({
                : turn,
                : worker_output,
                : worker_results
            })

            
            context += 
            context += 

            
            task_reward = evaluate_task_progress(
                worker_results, task
            )
            trajectory[].append(task_reward)

             task_reward > COMPLETION_THRESHOLD:
                trajectory[].append()
                
            :
                trajectory[].append()

         trajectory