Enable multimodal language models to autonomously generate and execute Python-based tools during visual reasoning, boosting performance on vision benchmarks by up to 31% through interactive problem-solving without relying on predefined tool sets.
Enable multimodal language models to autonomously generate and execute Python-based tools during visual reasoning, boosting performance on vision benchmarks by up to 31% through interactive problem-solving without relying on predefined tool sets.
PyVision: Self-Directed Visual Reasoning Through Dynamic Tool Creation
Standard vision-language models solve visual tasks through direct reasoning over the image. PyVision adds a reasoning layer: models can write and execute Python code as tools to decompose complex visual problems. Instead of trying to answer directly, the model generates code for object detection, region extraction, color analysis, or custom domain-specific processing, executes it, and iteratively refines both the code and the answer.
This agentic approach yields substantial gains: +7.8% on V* benchmarks for GPT-4.1, +31.1% on VLMsAreBlind-mini for Claude-3-Sonnet. The model effectively teaches itself task-specific tools on the fly.
Core Concept
Complex visual reasoning often requires specialized processing: segmenting objects, extracting regions, analyzing textures, measuring distances, or applying domain-specific logic. Rather than embedding all this knowledge in the LLM's weights, PyVision enables models to write executable code. The model reasons about what tool would help, generates Python code, executes it on the image, and uses results to refine its answer.
This creates a virtuous cycle: problem decomposition suggests tools, tools provide insights, insights enable better solutions. Critically, the model can iterate: if initial code fails, it can rewrite and retry. This self-correction loop is powerful for complex visual reasoning.
### Step 3: Evaluate Tool Usage Across Benchmarks
Analyze what types of tools models generate and their effectiveness:
```python
from collections import defaultdict
from typing import List, Dict
class ToolAnalyzer:
"""Analyze tool usage patterns across vision benchmarks."""
def __init__(self):
self.tool_types = defaultdict(int)
self.success_rate = defaultdict(float)
self.performance_by_tool = defaultdict(list)
def classify_tool_type(self, code: str) -> str:
"""Classify generated code by tool type."""
if "detect" in code.lower() or "yolo" in code.lower():
return "object_detection"
elif "segment" in code.lower():
return "segmentation"
elif "color" in code.lower():
return "color_analysis"
elif "edge" in code.lower():
return "edge_detection"
elif "measure" in code.lower() or "distance" in code.lower():
return "measurement"
elif "ocr" in code.lower() or "text" in code.lower():
return "text_recognition"
else:
return "custom_analysis"
def evaluate_benchmark(self, benchmark_name: str,
dataset: List[Dict],
solver: AgenticVisionSolver) -> Dict:
"""Evaluate solver on benchmark, tracking tool usage."""
results = {
"total": 0,
"correct": 0,
"tool_types": defaultdict(int),
"tool_success": defaultdict(int),
"accuracy_by_tool": defaultdict(list)
}
for sample in dataset:
image_path = sample["image"]
question = sample["question"]
ground_truth = sample["answer"]
# Solve with agentic approach
predicted = solver.solve(image_path, question)
# Check correctness
is_correct = (predicted.lower().strip() ==
ground_truth.lower().strip())
results["total"] += 1
if is_correct:
results["correct"] += 1
# TODO: Track which tools were used
# This requires extracting code from solver's history
accuracy = results["correct"] / results["total"] if results["total"] > 0 else 0
results["accuracy"] = accuracy
return results
def evaluate_pyvision_suite(solver: AgenticVisionSolver,
benchmarks: Dict[str, list]) -> Dict:
"""Evaluate PyVision across multiple benchmarks."""
analyzer = ToolAnalyzer()
results = {}
for bench_name, dataset in benchmarks.items():
print(f"Evaluating {bench_name}...")
bench_results = analyzer.evaluate_benchmark(bench_name, dataset, solver)
results[bench_name] = bench_results
print(f" Accuracy: {bench_results['accuracy']:.2%}")
return results
Step 4: Interactive Refinement Loop
Enable multi-turn refinement where the model improves its answer based on tool feedback:
definteractive_vqa(image_path: str,
question: str,
solver: AgenticVisionSolver,
max_turns: int = 5) -> str:
"""
Interactive VQA with user feedback for refinement.
"""
sandbox = VisionToolSandbox(image_path, max_iterations=max_turns)
answer = solver.solve(image_path, question)
print(f"Initial answer: {answer}")
for turn inrange(max_turns):
feedback = input("Is this answer correct? (yes/no/refine): ").strip().lower()
if feedback == "yes":
return answer
elif feedback == "no":
# Ask model to try a different approach
correction_prompt = f"That was incorrect. Try a different analysis approach."# TODO: Integrate feedback into solver loopelif feedback.startswith("refine"):
# Use specific feedback to improve
specific_feedback = feedback[7:].strip()
# TODO: Incorporate specific feedbackreturn answer
