enterprise-tool-calling-finetuning

name	enterprise-tool-calling-finetuning
title	Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.03336
keywords	["Tool Calling","Enterprise APIs","Disambiguation","Fine-Tuning","Interactive Agents"]
description	Train LLMs to disambiguate tool calls in enterprise settings where multiple similar APIs exist and parameters are incomplete. Generates synthetic multi-turn dialogues with realistic ambiguity to improve tool selection accuracy by 27+ percentage points.

DiaFORGE: Disambiguation-Centric Fine-Tuning for Reliable Tool-Calling

In enterprise environments, a single business query often maps to multiple near-duplicate APIs, and most real calls arrive missing required parameters. Current LLMs trained on clean, fully-specified tool-calling benchmarks fail dramatically when deployed against production APIs. DiaFORGE solves this mismatch by generating realistic, ambiguous multi-turn dialogues where the agent must disambiguate between competing tools and ask clarifying questions—turning a brittle single-turn problem into a robust interactive one.

The gap between benchmark performance and production reality is massive: static benchmarks use fully-specified queries with one obvious tool, while real APIs have overlapping functionality, required parameters, and user confusion. DiaFORGE bridges this gap through synthetic dialogue generation that mimics real deployment dynamics.

Core Concept

The framework operates on the insight that realistic tool-calling is fundamentally interactive. Rather than expecting agents to infer all details from a single query, train them to:

1. Recognize ambiguity: Detect when multiple tools could satisfy the request
2. Ask clarifying questions: Engage users to disambiguate and collect missing parameters
3. Validate completeness: Ensure all required parameters are present before calling the tool
4. Recover from errors: Gracefully handle ambiguous or impossible requests

This interactive approach reduces both false positives (wrong tool calls) and false negatives (refusing to act).

Architecture Overview

• UTC-Gen engine: Multi-agent system generating realistic multi-turn dialogues with ground-truth tools and semantic distractors
• Dialogue structure: Simulated user gradually reveals information while assistant asks clarifying questions
• Validation layer: Rule-based and LLM-based checks ensure generated data is correct
• Dynamic evaluation: Interactive rollout where trained agent engages with user simulator (not just static metrics)
• Parameter handling: Explicit tracking of required vs. provided parameters across turns

Implementation

Generate synthetic dialogues using the UTC-Gen multi-agent system. The engine seeds conversations with a ground-truth tool and includes semantic "distractor" tools:

from diaforge.generator import UTCGen

gen = UTCGen(api_catalog="enterprise_apis.json")


dialogue = gen.generate_dialogue(
    ground_truth_tool=,
    
    distractors=[
        ,
        ,
        
    ],
    
    information_stages=[
        {: , : []},
        {: , : [, ]},
        {: , : [, ]}
    ]
)



(dialogue)