Specialized outcome reward models (1.7B-14B) for evaluating tool-calling performance in LLMs. Addresses the critical gap in reward modeling where general-purpose models miss key signals of effective tool use. Enables better Best-of-N sampling, data filtering, and RL-based policy training through FC-RewardBench evaluation framework.
Specialized outcome reward models (1.7B-14B) for evaluating tool-calling performance in LLMs. Addresses the critical gap in reward modeling where general-purpose models miss key signals of effective tool use. Enables better Best-of-N sampling, data filtering, and RL-based policy training through FC-RewardBench evaluation framework.
ToolRM: Evaluating Tool Use Outcomes in LLMs
The Outcome: Accurate Tool-Calling Reward Signals
As LLMs increasingly orchestrate complex workflows through tool calling (APIs, databases, calculators), the ability to score whether a tool invocation achieved the user's goal has become essential. ToolRM delivers outcome reward models (1.7B-14B parameters) that accurately evaluate tool-calling success—enabling 25% performance improvements in Best-of-N sampling, robust data filtering for RL training, and effective policy optimization for autonomous agent systems.
Problem Context
Large language models now serve as orchestrators of external tools and services. Yet existing reward models trained on natural language outputs frequently miss the key signals of effective tool use: Did the model invoke the right tools? With correct arguments? In the right sequence? General-purpose reward models like those on RewardBench show poor correlation with actual tool-calling performance, leaving practitioners without reliable signals for preference optimization or reinforcement learning.
The gap is acute because tool-calling scenarios differ fundamentally from open-ended generation:
Binary outcome clarity: Tool calls either invoke the correct function or they don't—grounding is explicit.
Structured reasoning: Tool selection and argument filling follow deterministic patterns, not creative prose.
Multi-turn complexity: Tool calls often require sequential reasoning with dependencies (search result → next query → database lookup).
Robustness requirements: Models need signals that degrade gracefully under input noise, missing APIs, or partial information.
Prior work on reward modeling addresses code verification, math problem solving, and process-level step supervision, but no systematic framework existed for evaluating tool-calling reward models specifically.
Core Concept
ToolRM tackles outcome reward modeling for tool-calling tasks through three aligned contributions:
1. FC-RewardBench: The first benchmark (1,500 data points) specifically designed for evaluating reward models on function-calling tasks. Each example contains a user query, tool catalog, correct tool calls, and incorrect candidates generated from a diverse pool of 25 LLMs (0.5B–685B parameters).
2. ToolRM Suite: Outcome reward models trained end-to-end on tool-calling data, available in four scales (1.7B, 7B, 14B parameters). These models score the entire tool call sequence in context, capturing correctness signals that general-purpose models miss.
3. Empirical Validation: Demonstration that ToolRM outperforms general-purpose baselines across diverse scenarios (Best-of-N sampling, data filtering, RL-guided fine-tuning), with correlation analysis showing ToolRM scores predict downstream task success.
The key insight: outcome reward models work better than process-level scoring for tool-calling because the correctness of a tool invocation is fundamentally tied to the final outcome—whether the query was answered correctly. A single wrong parameter ruins the entire call.
Architecture Overview
ToolRM adopts a lightweight, practical design for integrating into LLM training pipelines:
Base Architecture: Transformer-based, decoder-only models (similar to Llama, Mistral) as foundational encoders
Input Representation: Concatenates user query, tool catalog (function names, descriptions, parameter schemas), and the proposed tool call sequence into a single prompt context
Scoring Head: Trained classification or regression head that outputs a scalar reward or preference logit
Training Signal: Synthetic preference pairs (correct vs. incorrect tool calls) from diverse LLMs, augmented with permissive-license data sources
Scales: 1.7B, 7B, 14B parameters, allowing trade-offs between latency and accuracy
No Tool Execution: The model evaluates tool calls statically (syntactic correctness, semantic alignment) without actually executing functions
Robustness: Evaluation protocol includes noise injection (missing APIs, malformed arguments) to assess graceful degradation
The architecture intentionally avoids process-level supervision (rewarding intermediate steps) in favor of outcome evaluation, because tool-calling correctness is largely deterministic—the presence of a single error invalidates the entire sequence.
Implementation
Step 1: Prepare FC-RewardBench Evaluation Data
Create a dataset with 1,500 function-calling examples following the BFCL-v3 format: user query → tool catalog → correct calls → incorrect candidates.
import json
import random
# Load BFCL-v3 single-turn datawithopen("bfcl-v3-single-turn.jsonl") as f:
examples = [json.loads(line) for line in f]
# Structure for FC-RewardBenchdefcreate_eval_example(query, tools, correct_calls, incorrect_candidates):
return {
"user_query": query,
"tool_catalog": tools, # List of tool definitions with name, desc, params"correct_calls": correct_calls, # Reference tool call sequences"incorrect": incorrect_candidates, # Negative examples from diverse LLMs"difficulty": assess_complexity(query, tools)
}
# Generate negatives using 25 LLMs (0.5B to 685B parameters)
lm_pool = ["gpt2", "pythia-1b", "llama-7b", "gpt-3.5", "llama-70b"]
incorrect_calls = {}
for lm in lm_pool:
try:
# Prompt each LM with: "Query: {query}\nTools: {tools}\nCall:"
response = call_llm_with_prompt(query, tools, lm_id=lm)
incorrect_calls[lm] = parse_tool_call(response)
except Exception as e:
pass# Skip if LM call fails
eval_data = {
"examples": [
create_eval_example(ex["query"], ex["tools"], ex["calls"], incorrect_calls)
for ex in examples[:1500]
]
}
withopen("fc_rewardbench.json", "w") as f:
json.dump(eval_data, f)
Step 2: Synthesize Training Data for ToolRM
Generate preference pairs (correct call wins, incorrect call loses) using permissively licensed LLMs. This avoids licensing concerns with proprietary models and ensures reproducibility.
Step 3: Train ToolRM with Outcome Reward Objective
Fine-tune a base LLM (Llama 7B, Mistral 7B, etc.) on preference pairs using a binary classification loss. The model learns to output high scores for correct tool calls and low scores for incorrect ones.
Preference optimization: Use ToolRM preference signals in DPO or IPO to align LLMs toward better tool use
Deployment monitoring: Track tool-calling quality in production by scoring call sequences real-time
When NOT to Use ToolRM
Open-ended generation tasks: ToolRM is specialized for structured tool calls; use general-purpose reward models (RewardBench top models) for natural language preference
Process-level feedback: If your task requires step-by-step reasoning guidance (intermediate steps matter), use process reward models instead; ToolRM gives final-outcome signals only
Real-time tool execution feedback: ToolRM provides static evaluation; if you need rewards from actually executing tools (e.g., database queries), integrate tool execution directly into the loop
Extreme domain drift: ToolRM trained on BFCL tools (web search, code, APIs) may not generalize to proprietary or highly specialized tool sets; fine-tune on your domain
Multi-agent coordination: ToolRM scores individual tool calls; for multi-agent orchestration, model inter-agent dependencies explicitly
Low-latency inference: 7B/14B models require GPU; use 1.7B version if latency is critical, or distill further
Common Pitfalls and Mitigations
Pitfall 1: Training on same LLM errors repeatedly
The negative examples matter. If you only use errors from 2-3 LMs, the model learns narrow failure patterns.
Mitigation: Synthesize negatives from 25+ diverse models as in the paper; rotate models during training.
Pitfall 2: Overfitting to tool catalog format
If all training examples use the same tool description style, ToolRM fails on differently-formatted catalogs.
Mitigation: Vary tool schemas, description lengths, and parameter naming during data generation; test on out-of-distribution catalogs.
Pitfall 3: Missing low-signal negatives
Generating negatives only from weak models (0.5B params) means they're too obvious to distinguish. Strong negatives train better discernment.
Mitigation: Include negatives from models at or above your target policy quality (7B–70B range).
Pitfall 4: Margin too large or too small
Margin 2.0 in preference loss makes the training too strict; margin 0.1 allows incorrect calls to remain competitive.
Mitigation: Start at 0.5, measure AUC on held-out eval data, adjust if AUC plateaus or diverges.
Pitfall 5: Ignoring robustness under noise
ToolRM trained on clean tool catalogs may collapse on missing APIs, malformed arguments, or incomplete context.
Mitigation: Explicitly corrupt training data (drop 10% of tools, mangle 5% of arguments) to teach graceful degradation.
Reference
Paper: ToolRM: Outcome Reward Models for Tool-Calling Large Language Models
Authors: Mayank Agarwal, Ibrahim Abdelaziz, Kinjal Basu, Merve Unuvar, Luis A. Lastras, Yara Rizk, Pavan Kapanipathi (IBM Research)
Published: September 2025 | Version: v2 (January 2026)
