tir-bench-visual-tool-reasoning

name	tir-bench-visual-tool-reasoning
title	TIR-Bench: Agentic Thinking-with-Images Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.01833
keywords	["Visual Reasoning","Tool Use","Multimodal Reasoning","Benchmark","Image Manipulation"]
description	Evaluate multimodal models on agentic visual reasoning through 13 diverse tasks requiring novel tool use for image processing and manipulation in chain-of-thought, revealing that strong performance requires genuine thinking-with-images capabilities.

Title: Test Genuine Visual Reasoning With Interactive Tool Use

Existing visual benchmarks test pattern recognition but not reasoning. TIR-Bench evaluates whether models can act as agents: generating, manipulating, and reasoning over images as part of solving problems. The benchmark includes 13 diverse tasks (color VQA, symbolic reasoning, math, maze solving, etc.) requiring tool creation and deployment—highlighting gaps in current approaches and demonstrating that agentic fine-tuning substantially outperforms direct supervised approaches.

The key insight: visual reasoning is more than recognition—it requires agency.

Core Concept

Agentic Visual Reasoning Evaluation:

• 13 Diverse Tasks: Spanning perception, reasoning, and tool-dependent operations
• Tool Requirements: Enhancement, geometric, programmatic, drawing—models must create and apply tools
• Two Reasoning Modes: Direct vs. agentic fine-tuning evaluation
• 1,215 Examples: 665 multiple-choice + 550 free-form questions
• Universal Challenge: Even top models (o3-TU: 46%, non-agentic: 29%) struggle

Architecture Overview

• Task Categories: Vision-centric (puzzles, mazes) + perception (VQA, reading)
• Evaluation Metrics: Accuracy for MC + IOU for grounding
• Model Types Tested: 22 models spanning open-source and proprietary
• Fine-Tuning Approaches: Direct SFT vs. agentic (tool-use enabled)
• Benchmark Suite: Tool API definitions + grading functions + visualization tools

Implementation Steps

1. Define Task Types and Tool Requirements

Structure benchmark around tool-use patterns.

class TIRBenchTaskTypes:
    # Vision-centric: require visual reasoning
    VISION_TASKS = {
        'eyeballing_puzzles': {
            'description': 'Estimate relative sizes/distances',
            'tools': ['measurement', 'highlighting', 'comparison'],
            'metric': 'accuracy'
        },
        : {
            : ,
            : [, , ],
            : 
        },
        : {
            : ,
            : [, ],
            : 
        },
        : {
            : ,
            : [, ],
            : 
        },
        : {
            : ,
            : [, ],
            : 
        },
        : {
            : ,
            : [, , ],
            : 
        },
        : {
            : ,
            : [, ],
            : 
        }
    }

    
    PERCEPTION_TASKS = {
        : {
            : ,
            : [, ],
            : 
        },
        : {
            : ,
            : [, ],
            : 
        },
        : {
            : ,
            : [, ],
            : 
        },
        : {
            : ,
            : [, ],
            : 
        },
        : {
            : ,
            : [, ],
            : 
        },
        : {
            : ,
            : [, ],
            : 
        },
        : {
            : ,
            : [, ],
            : 
        }
    }

    ALL_TASKS = {**VISION_TASKS, **PERCEPTION_TASKS}