Skip to main content Inicio Creadores adu2021 skillxiv dissecting-tool-integrated-reasoning
dissecting-tool-integrated-reasoning Evaluate and optimize tool-integrated reasoning in LLMs through empirical benchmarking, performance-cost metrics (PAC, AUC-PCC), and measurement frameworks for diverse reasoning tasks.
Ir a la instalación Skills Marketplace Descubre y explora habilidades de IA creadas por la comunidad.
Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.
Mostrar detalles del prompt Un comando directo omite el prompt de revisión. Revisa el origen antes de ejecutarlo.
npx skills add https://github.com/ADu2021/skillXiv --skill dissecting-tool-integrated-reasoningEl comando permanece en una sola línea. Desplázate horizontalmente para revisarlo antes de copiarlo.
¿Prefieres una copia local? Descarga los archivos que SkillsMP tiene disponibles ahora.
Descargar Zip Descargando... Más de este repositorio meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
Ocupaciones relacionadas SOC
Basado en la clasificación ocupacional SOC
name dissecting-tool-integrated-reasoning title Tool-Integrated Reasoning: Empirical Benchmarking and Efficiency Metrics version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2508.15754 keywords ["tool-integration","reasoning-efficiency","chain-of-thought","external-tools","benchmarking"] description Evaluate and optimize tool-integrated reasoning in LLMs through empirical benchmarking, performance-cost metrics (PAC, AUC-PCC), and measurement frameworks for diverse reasoning tasks.
Tool-Integrated Reasoning: Empirical Analysis and Efficiency Metrics
Core Concept
Tool-Integrated Reasoning (TIR) enables language models to offload computational tasks to external tools (calculators, code interpreters, etc.) rather than solving everything through token generation. This approach reduces "overthinking" and produces more streamlined reasoning traces while improving accuracy. The key innovation is measuring TIR effectiveness beyond raw accuracy through efficiency metrics like Performance-At-Cost (PAC) and Area-Under-Curve-PCC (AUC-PCC), enabling systematic evaluation across diverse reasoning domains.
Architecture Overview
ReasonZoo Benchmark : Nine reasoning categories for comprehensive evaluationExternal Tool Integration : Seamless API binding to computation toolsStreamlined Reasoning Traces : Reduced token generation through strategic tool delegationDual Metrics Framework : Accuracy paired with efficiency measurementsMulti-Domain Coverage : Mathematical, logical, and natural reasoning tasks
Implementation Steps
1. Design ReasonZoo Benchmark Categories
Create comprehensive benchmark covering diverse reasoning types:
from enum import Enum
from dataclasses import dataclass
class ReasoningCategory (Enum ):
ARITHMETIC = "arithmetic"
ALGEBRA = "algebra"
GEOMETRY = "geometry"
LOGIC = "logic"
COUNTING = "counting"
KNOWLEDGE = "knowledge"
SYMBOLIC = "symbolic"
COMMONSENSE = "commonsense"
MULTI_STEP = "multi_step"
@dataclass
class BenchmarkTask :
task_id: str
category: ReasoningCategory
question: str
expected_answer: str
tools_required: list [str
bool
int
def
create_reasonzoo_benchmark
list
"""
Construct comprehensive benchmark across nine reasoning categories.
"""
"arith_001"
"What is 47 * 13 + 892?"
"1503"
"calculator"
False
1
"multi_001"
"If a store has 50 items and sells 12 on Monday and 18 on Tuesday, how many remain?"
"20"
"calculator"
True
2
return
2. Implement Tool Integration Layer Create abstraction for external tool binding:
from abc import ABC, abstractmethod
from typing import Any , Dict
class ExternalTool (ABC ):
@abstractmethod
def execute (self, input_str: str ) -> str :
pass
class Calculator (ExternalTool ):
def execute (self, expression: str ) -> str :
"""Evaluate mathematical expressions safely."""
try :
result = eval (expression, {"__builtins__" : {}}, {})
return str (result)
except Exception as e:
return f"Error: {str (e)} "
class CodeInterpreter (ExternalTool ):
def execute (self, code: str ) -> str :
"""Execute Python code safely in sandbox."""
import subprocess
try :
result = subprocess.run(
["python" , "-c" , code],
capture_output=True ,
text=True ,
timeout=5
)
return result.stdout or result.stderr
except Exception as e:
return f"Error: {str (e)} "
class ToolRouter :
def __init__ (self ):
self .tools = {
"calculator" : Calculator(),
"code_interpreter" : CodeInterpreter(),
"search" : SearchTool(),
}
def route_and_execute (self, tool_name: str , tool_input: str ) -> str :
"""Route execution request to appropriate tool."""
if tool_name not in self .tools:
return f"Unknown tool: {tool_name} "
return self .tools[tool_name].execute(tool_input)
3. Create Reasoning Trace Collection Pipeline Capture detailed reasoning execution for analysis:
@dataclass
class ReasoningTrace :
task_id: str
model_response: str
tool_calls: list [Dict [str , Any ]]
reasoning_steps: list [str ]
final_answer: str
correct: bool
tokens_generated: int
tool_execution_time: float
total_time: float
class ReasoningTraceCollector :
def __init__ (self, tool_router: ToolRouter ):
self .tool_router = tool_router
self .traces = []
def collect_trace (
self,
task: BenchmarkTask,
model: "LLM"
) -> ReasoningTrace:
"""
Execute task and collect detailed reasoning trace.
"""
trace = ReasoningTrace(
task_id=task.task_id,
model_response="" ,
tool_calls=[],
reasoning_steps=[],
final_answer="" ,
correct=False ,
tokens_generated=0 ,
tool_execution_time=0.0 ,
total_time=0.0
)
start_time = time.time()
response, tool_calls, steps = model.generate_with_tools(
task.question,
available_tools=task.tools_required
)
trace.model_response = response
trace.tokens_generated = len (response.split())
trace.reasoning_steps = steps
tool_start = time.time()
for tool_call in tool_calls:
result = self .tool_router.route_and_execute(
tool_call["tool_name" ],
tool_call["input" ]
)
tool_call["result" ] = result
trace.tool_calls.append(tool_call)
trace.tool_execution_time = time.time() - tool_start
trace.total_time = time.time() - start_time
trace.final_answer = extract_answer(response)
trace.correct = trace.final_answer == task.expected_answer
return trace
4. Implement Performance-At-Cost (PAC) Metric Measure efficiency beyond accuracy:
def compute_pac_metric (
traces: list [ReasoningTrace],
cost_weights: Dict [str , float ] = None
) -> float :
"""
Compute Performance-At-Cost metric balancing accuracy and efficiency.
PAC = Accuracy / (1 + normalized_cost)
where cost includes tokens and tool calls
"""
if cost_weights is None :
cost_weights = {
"token" : 0.001 ,
"tool_call" : 0.1 ,
"time" : 0.01
}
total_accuracy = 0.0
total_cost = 0.0
for trace in traces:
total_accuracy += 1.0 if trace.correct else 0.0
token_cost = trace.tokens_generated * cost_weights["token" ]
tool_cost = len (trace.tool_calls) * cost_weights["tool_call" ]
time_cost = trace.total_time * cost_weights["time" ]
total_cost += token_cost + tool_cost + time_cost
avg_accuracy = total_accuracy / len (traces)
normalized_cost = total_cost / len (traces)
pac = avg_accuracy / (1.0 + normalized_cost)
return pac
5. Compute AUC-PCC (Area Under Curve - Performance vs Cost Curve) Measure efficiency frontier:
from sklearn.metrics import auc
def compute_auc_pcc (
traces: list [ReasoningTrace],
cost_budgets: list [float ]
) -> float :
"""
Compute AUC of Performance vs Cost Curve.
Shows how accuracy scales with different cost budgets
(token limits, time limits, tool call limits)
"""
accuracies = []
costs = []
sorted_traces = sorted (traces, key=lambda t: t.total_time)
cumulative_cost = 0.0
cumulative_correct = 0
for i, trace in enumerate (sorted_traces):
cumulative_cost += trace.total_time
if trace.correct:
cumulative_correct += 1
accuracy = cumulative_correct / (i + 1 )
accuracies.append(accuracy)
costs.append(cumulative_cost / (i + 1 ))
min_cost = min (costs)
max_cost = max (costs)
normalized_costs = [(c - min_cost) / (max_cost - min_cost) for c in costs]
auc_pcc = auc(normalized_costs, accuracies)
return auc_pcc
def evaluate_tool_integration (
model: "LLM" ,
benchmark: list [BenchmarkTask]
) -> Dict [str , float ]:
"""
Comprehensive evaluation of tool-integrated reasoning.
"""
collector = ReasoningTraceCollector(ToolRouter())
traces = [collector.collect_trace(task, model) for task in benchmark]
metrics = {
"accuracy" : sum (1.0 for t in traces if t.correct) / len (traces),
"avg_tokens" : sum (t.tokens_generated for t in traces) / len (traces),
"avg_tool_calls" : sum (len (t.tool_calls) for t in traces) / len (traces),
"avg_time" : sum (t.total_time for t in traces) / len (traces),
"pac" : compute_pac_metric(traces),
"auc_pcc" : compute_auc_pcc(traces, cost_budgets=[0.5 , 1.0 , 2.0 ])
}
return metrics
Practical Guidance
When to Use Tool-Integrated Reasoning
Mathematical and computational tasks (arithmetic, algebra, calculus)
Tasks with high precision requirements beyond LLM capabilities
Multi-step reasoning combining thinking and computation
Knowledge-intensive questions requiring lookup tools
Code generation and execution validation
When NOT to Use
Creative generation tasks (poetry, narrative)
Real-time systems with strict latency requirements
Tasks where all knowledge should be contained in model weights
Domains without well-defined external tools
Key Hyperparameters
Tool Timeout : 1-5 seconds to prevent hangingMax Tool Calls : 5-20 per reasoning episodeToken Limit : 1000-4000 for reasoning tracesCost Weights : Adjust based on tool availability/cost tradeoffs
Performance Expectations
Accuracy Improvement: 10-30% on mathematical tasks
Token Reduction: 30-50% fewer tokens with tool use
Tool Call Frequency: 1-5 tool invocations per multi-step problem
End-to-end Speedup: 1.5-3x when tools are efficiently invoked
Reference Researchers. (2024). Dissecting Tool-Integrated Reasoning: An Empirical Study and Analysis. arXiv preprint arXiv:2508.15754.
