tool-orchestra-coordination

name	tool-orchestra-coordination
title	ToolOrchestra: Elevating Intelligence via Model and Tool Orchestration
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.21689
keywords	["tool-use","orchestration","reinforcement-learning","agent-training","multi-modal-agents"]
description	8B parameter orchestrator trained with end-to-end RL balancing outcome, efficiency, and user preference rewards to strategically coordinate diverse tools and models. Generate realistic tool-use data via ToolScale synthetic pipeline for verifiable multi-turn scenarios.

Summary

ToolOrchestra introduces an 8B-parameter language model trained to serve as an orchestrator that strategically coordinates diverse tools and models. The key innovation is an end-to-end reinforcement learning framework with multi-objective reward design balancing outcome correctness, computational efficiency, and user preference alignment. ToolScale provides the synthetic dataset pipeline.

Core Technique

Multi-Objective Reward Design: Balance three competing objectives:

1. Outcome Reward: Correctness of final answer
2. Efficiency Reward: Computational cost of tool invocations
3. User Preference Reward: Alignment with user preferences (speed, format, confidence)

Formulate as: R = w_o * R_outcome + w_e * R_efficiency + w_p * R_preference

ToolScale Dataset Generation: Synthetic pipeline creating verifiable multi-turn tool-use examples across domains by:

• Sampling problems from multiple domains
• Generating grounded solution paths with tool calls
• Collecting execution traces and outcomes
• Labeling efficiency and preference metrics

End-to-End RL Training: Train the orchestrator using policy gradient methods (PPO or GRPO) where the policy is the LLM choosing which tool to call next.

Implementation

Reward computation: At each step:

# Outcome reward: +1 if final answer is correct, 0 otherwise
r_outcome = 1.0 if is_correct(final_answer) else 0.0

# Efficiency reward: penalize tool calls and tokens
r_efficiency = -0.01 * num_tool_calls - 0.0001 * num_tokens

# User preference: e.g., preference for fast simple solutions
r_preference = 0.5 if solved_quickly else 0.1

# Combined reward
total_reward = w_o * r_outcome + w_e * r_efficiency + w_p * r_preference

Tool specification: Define available tools:

tools = {
    'calculator': (cost=0.1, latency=10ms),
    : (cost=, latency=500ms),
    : (cost=, latency=100ms),
    ...
}