toolrm-outcome-reward-tool-calling

name	toolrm-outcome-reward-tool-calling
title	ToolRM: Outcome Reward Models for Tool-Calling Large Language Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2509.11963
keywords	["reward-models","tool-calling","LLMs","outcome-evaluation","function-calling","reinforcement-learning","preference-optimization","evaluation-benchmarks"]
description	Specialized outcome reward models (1.7B-14B) for evaluating tool-calling performance in LLMs. Addresses the critical gap in reward modeling where general-purpose models miss key signals of effective tool use. Enables better Best-of-N sampling, data filtering, and RL-based policy training through FC-RewardBench evaluation framework.

ToolRM: Evaluating Tool Use Outcomes in LLMs

The Outcome: Accurate Tool-Calling Reward Signals

As LLMs increasingly orchestrate complex workflows through tool calling (APIs, databases, calculators), the ability to score whether a tool invocation achieved the user's goal has become essential. ToolRM delivers outcome reward models (1.7B-14B parameters) that accurately evaluate tool-calling success—enabling 25% performance improvements in Best-of-N sampling, robust data filtering for RL training, and effective policy optimization for autonomous agent systems.

Problem Context

Large language models now serve as orchestrators of external tools and services. Yet existing reward models trained on natural language outputs frequently miss the key signals of effective tool use: Did the model invoke the right tools? With correct arguments? In the right sequence? General-purpose reward models like those on RewardBench show poor correlation with actual tool-calling performance, leaving practitioners without reliable signals for preference optimization or reinforcement learning.

The gap is acute because tool-calling scenarios differ fundamentally from open-ended generation:

• Binary outcome clarity: Tool calls either invoke the correct function or they don't—grounding is explicit.
• Structured reasoning: Tool selection and argument filling follow deterministic patterns, not creative prose.
• Multi-turn complexity: Tool calls often require sequential reasoning with dependencies (search result → next query → database lookup).
• Robustness requirements: Models need signals that degrade gracefully under input noise, missing APIs, or partial information.

Prior work on reward modeling addresses code verification, math problem solving, and process-level step supervision, but no systematic framework existed for evaluating tool-calling reward models specifically.

Core Concept

ToolRM tackles outcome reward modeling for tool-calling tasks through three aligned contributions:

1. FC-RewardBench: The first benchmark (1,500 data points) specifically designed for evaluating reward models on function-calling tasks. Each example contains a user query, tool catalog, correct tool calls, and incorrect candidates generated from a diverse pool of 25 LLMs (0.5B–685B parameters).

2. ToolRM Suite: Outcome reward models trained end-to-end on tool-calling data, available in four scales (1.7B, 7B, 14B parameters). These models score the entire tool call sequence in context, capturing correctness signals that general-purpose models miss.

3. Empirical Validation: Demonstration that ToolRM outperforms general-purpose baselines across diverse scenarios (Best-of-N sampling, data filtering, RL-guided fine-tuning), with correlation analysis showing ToolRM scores predict downstream task success.