pyvision-agentic-vision-dynamic-tools

name	pyvision-agentic-vision-dynamic-tools
title	PyVision: Agentic Vision with Dynamic Tooling
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.07998
keywords	["Agentic Vision","Tool Generation","Visual Reasoning","Code Execution","Dynamic Programming"]
description	Enable multimodal language models to autonomously generate and execute Python-based tools during visual reasoning, boosting performance on vision benchmarks by up to 31% through interactive problem-solving without relying on predefined tool sets.

PyVision: Self-Directed Visual Reasoning Through Dynamic Tool Creation

Standard vision-language models solve visual tasks through direct reasoning over the image. PyVision adds a reasoning layer: models can write and execute Python code as tools to decompose complex visual problems. Instead of trying to answer directly, the model generates code for object detection, region extraction, color analysis, or custom domain-specific processing, executes it, and iteratively refines both the code and the answer.

This agentic approach yields substantial gains: +7.8% on V* benchmarks for GPT-4.1, +31.1% on VLMsAreBlind-mini for Claude-3-Sonnet. The model effectively teaches itself task-specific tools on the fly.

Core Concept

Complex visual reasoning often requires specialized processing: segmenting objects, extracting regions, analyzing textures, measuring distances, or applying domain-specific logic. Rather than embedding all this knowledge in the LLM's weights, PyVision enables models to write executable code. The model reasons about what tool would help, generates Python code, executes it on the image, and uses results to refine its answer.

This creates a virtuous cycle: problem decomposition suggests tools, tools provide insights, insights enable better solutions. Critically, the model can iterate: if initial code fails, it can rewrite and retry. This self-correction loop is powerful for complex visual reasoning.

Architecture Overview

• Vision-Language Backbone: Multimodal LLM (GPT-4.1, Claude-Sonnet, etc.) with code generation ability
• Code Execution Sandbox: Safe Python environment with image processing libraries (PIL, OpenCV, numpy)
• Tool Library: Automatically imported dependencies (cv2, PIL, numpy, scipy, torchvision, etc.)
• Interactive Loop: Model generates code → code executes → results feed back to model
• Error Handling: Failed executions provide error messages; model can debug and retry
• Iteration Limit: Typically 3-5 iterations per question to prevent infinite loops
• Output Extraction: Model learns to call special functions to finalize answers

Implementation

Step 1: Design the Code Execution Sandbox

Create a safe environment where models can run code on images:

import tempfile
import subprocess
import json
from typing import Dict, Any, 
 PIL  Image
 base64
 io

 :
    

     ():
        .image_path = image_path
        .image = Image.(image_path)
        .max_iterations = max_iterations
        .execution_history = []
        .variables = {}  

     () -> [, ]:
        
        
        script_template = 

        
         tempfile.NamedTemporaryFile(mode=, suffix=, delete=)  f:
            f.write(script_template)
            script_path = f.name

        
        :
            result = subprocess.run(
                [, script_path],
                capture_output=,
                text=,
                timeout=  
            )

             result.returncode == :
                
                :
                     (, )  f:
                        outputs = json.load(f)
                     {
                        : ,
                        : outputs,
                        : 
                    }
                :
                     {
                        : ,
                        : {: result.stdout},
                        : 
                    }
            :
                 {
                    : ,
                    : ,
                    : result.stderr
                }

         subprocess.TimeoutExpired:
             {
                : ,
                : ,
                : 
            }
         Exception  e:
             {
                : ,
                : ,
                : (e)
            }

     () -> :
        
         max_iterations  :
            max_iterations = .max_iterations

        conversation_history = [
            {: , : }
        ]

         iteration  (max_iterations):
            
            code_response = model_fn(
                image_path=.image_path,
                question=question,
                history=conversation_history
            )

            
            code = ._extract_code(code_response)

             code  :
                
                 code_response.strip()

            
            execution_result = .execute_tool_code(code)

            
            conversation_history.append({
                : ,
                : 
            })

             execution_result[] == :
                outputs_str = json.dumps(execution_result[])
                conversation_history.append({
                    : ,
                    : 
                })

                
                   execution_result[]:
                     execution_result[][]

            :
                error_msg = execution_result[]
                conversation_history.append({
                    : ,
                    : 
                })

        
        conclusion_prompt = 
        final_response = model_fn(
            image_path=.image_path,
            question=question,
            history=conversation_history + [
                {: , : conclusion_prompt}
            ]
        )

         final_response.strip()

     () -> []:
        
         re

        
        code_pattern = 
        matches = re.findall(code_pattern, response, re.DOTALL)

         matches:
             matches[]