Pyvision Agentic Vision Dynamic Tools

PyVision: Self-Directed Visual Reasoning Through Dynamic Tool Creation

Standard vision-language models solve visual tasks through direct reasoning over the image. PyVision adds a reasoning layer: models can write and execute Python code as tools to decompose complex visual problems. Instead of trying to answer directly, the model generates code for object detection, region extraction, color analysis, or custom domain-specific processing, executes it, and iteratively refines both the code and the answer.

This agentic approach yields substantial gains: +7.8% on V* benchmarks for GPT-4.1, +31.1% on VLMsAreBlind-mini for Claude-3-Sonnet. The model effectively teaches itself task-specific tools on the fly.

Core Concept

Complex visual reasoning often requires specialized processing: segmenting objects, extracting regions, analyzing textures, measuring distances, or applying domain-specific logic. Rather than embedding all this knowledge in the LLM's weights, PyVision enables models to write executable code. The model reasons about what tool would help, generates Python code, executes it on the image, and uses results to refine its answer.

This creates a virtuous cycle: problem decomposition suggests tools, tools provide insights, insights enable better solutions. Critically, the model can iterate: if initial code fails, it can rewrite and retry. This self-correction loop is powerful for complex visual reasoning.

Architecture Overview

• Vision-Language Backbone: Multimodal LLM (GPT-4.1, Claude-Sonnet, etc.) with code generation ability
• Code Execution Sandbox: Safe Python environment with image processing libraries (PIL, OpenCV, numpy)
• Tool Library: Automatically imported dependencies (cv2, PIL, numpy, scipy, torchvision, etc.)
• Interactive Loop: Model generates code → code executes → results feed back to model
• Error Handling: Failed executions provide error messages; model can debug and retry
• Iteration Limit: Typically 3-5 iterations per question to prevent infinite loops
• Output Extraction: Model learns to call special functions to finalize answers

Implementation

Step 1: Design the Code Execution Sandbox

Create a safe environment where models can run code on images:

import tempfile
import subprocess
import json
from typing import Dict, Any, Optional
from PIL import Image
import base64
import io

class VisionToolSandbox:
    """Safe sandbox for executing vision tools generated by LLM."""

    def __init__(self, image_path: str, max_iterations: int = 5):
        self.image_path = image_path
        self.image = Image.open(image_path)
        self.max_iterations = max_iterations
        self.execution_history = []
        self.variables = {}  # Store tool results

    def execute_tool_code(self, code: str) -> Dict[str, Any]:
        """
        Execute generated Python code in isolated environment.
        Returns: result dict with execution status and outputs
        """
        # Create temporary Python script
        script_template = f'''
import cv2
import numpy as np
import json
from PIL import Image
from scipy import ndimage
from torchvision import transforms
import torch

# Load image
image = Image.open("{self.image_path}")
image_array = np.array(image)

# User-provided code
{code}

# Capture outputs
outputs = {{}}
if 'result' in locals():
    outputs['result'] = str(result)
if 'detections' in locals():
    outputs['detections'] = detections if isinstance(detections, list) else str(detections)
if 'answer' in locals():
    outputs['answer'] = answer

with open('/tmp/tool_output.json', 'w') as f:
    json.dump(outputs, f)
'''

        # Write script to temp file
        with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
            f.write(script_template)
            script_path = f.name

        # Execute script
        try:
            result = subprocess.run(
                ['python', script_path],
                capture_output=True,
                text=True,
                timeout=10  # 10-second timeout
            )

            if result.returncode == 0:
                # Read outputs
                try:
                    with open('/tmp/tool_output.json', 'r') as f:
                        outputs = json.load(f)
                    return {
                        "status": "success",
                        "outputs": outputs,
                        "error": None
                    }
                except:
                    return {
                        "status": "success",
                        "outputs": {"result": result.stdout},
                        "error": None
                    }
            else:
                return {
                    "status": "error",
                    "outputs": None,
                    "error": result.stderr
                }

        except subprocess.TimeoutExpired:
            return {
                "status": "error",
                "outputs": None,
                "error": "Execution timeout (> 10 seconds)"
            }
        except Exception as e:
            return {
                "status": "error",
                "outputs": None,
                "error": str(e)
            }

    def run_interactive_loop(self, question: str,
                            model_fn,  # Function that generates code from question/history
                            max_iterations: Optional[int] = None) -> str:
        """
        Run interactive reasoning loop: generate code, execute, refine.
        """
        if max_iterations is None:
            max_iterations = self.max_iterations

        conversation_history = [
            {"role": "user", "content": f"Visual question: {question}"}
        ]

        for iteration in range(max_iterations):
            # Ask model to generate code
            code_response = model_fn(
                image_path=self.image_path,
                question=question,
                history=conversation_history
            )

            # Extract code from response
            code = self._extract_code(code_response)

            if code is None:
                # Model directly answered without code
                return code_response.strip()

            # Execute code
            execution_result = self.execute_tool_code(code)

            # Add to history
            conversation_history.append({
                "role": "assistant",
                "content": f"Code executed: {code[:100]}..."
            })

            if execution_result["status"] == "success":
                outputs_str = json.dumps(execution_result["outputs"])
                conversation_history.append({
                    "role": "user",
                    "content": f"Tool result: {outputs_str}"
                })

                # Check if answer was finalized
                if "answer" in execution_result["outputs"]:
                    return execution_result["outputs"]["answer"]

            else:
                error_msg = execution_result["error"]
                conversation_history.append({
                    "role": "user",
                    "content": f"Error: {error_msg}\nPlease fix the code."
                })

        # If no final answer, ask model for conclusion
        conclusion_prompt = "Based on your analysis, what is the final answer?"
        final_response = model_fn(
            image_path=self.image_path,
            question=question,
            history=conversation_history + [
                {"role": "user", "content": conclusion_prompt}
            ]
        )

        return final_response.strip()

    def _extract_code(self, response: str) -> Optional[str]:
        """Extract Python code from model response."""
        import re

        # Look for code blocks
        code_pattern = r'```python\s*(.*?)\s*```'
        matches = re.findall(code_pattern, response, re.DOTALL)

        if matches:
            return matches[0]
        return None

Step 2: Implement Agentic Vision Loop

Build the interactive reasoning loop where the model generates tools and refines answers:

class AgenticVisionSolver:
    """
    Agentic vision solver: model generates tools and reasons iteratively.
    """

    def __init__(self, model_name: str = "claude-3-sonnet"):
        self.model_name = model_name
        self.client = self._init_client(model_name)

    def _init_client(self, model_name: str):
        """Initialize appropriate API client."""
        if "claude" in model_name.lower():
            import anthropic
            return anthropic.Anthropic()
        elif "gpt" in model_name.lower():
            import openai
            return openai.OpenAI()
        else:
            raise ValueError(f"Unknown model: {model_name}")

    def generate_tool_code(self, image_path: str,
                          question: str,
                          conversation_history: list) -> str:
        """Ask LLM to generate Python code for visual analysis."""

        # Encode image as base64
        with open(image_path, 'rb') as f:
            image_data = base64.b64encode(f.read()).decode('utf-8')

        system_prompt = """You are an expert at visual reasoning and Python programming.
When given a visual question, you can write Python code to analyze the image.

Available tools:
- PIL.Image: load and manipulate images
- cv2: computer vision operations
- numpy: numerical computing
- scipy: scientific computing
- torchvision: deep learning vision models

You can define variables and return them as 'result', 'detections', or 'answer'.

Example:
```python
# Detect objects
detector = ... # initialize detector
objects = detector(image_array)
result = len(objects)  # Number of objects

Always write complete, executable code. If tools fail, the error will be returned."""

    messages = [
        *conversation_history,
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/jpeg",
                        "data": image_data
                    }
                },
                {
                    "type": "text",
                    "text": f"Question: {question}\n\nGenerate Python code to analyze this image and answer the question."
                }
            ]
        }
    ]

    response = self.client.messages.create(
        model=self.model_name,
        max_tokens=1024,
        system=system_prompt,
        messages=messages
    )

    return response.content[0].text

def solve(self, image_path: str, question: str) -> str:
    """
    Solve visual question through agentic reasoning.
    """
    sandbox = VisionToolSandbox(image_path, max_iterations=5)

    def model_fn(image_path, question, history):
        return self.generate_tool_code(image_path, question, history)

    answer = sandbox.run_interactive_loop(question, model_fn)
    return answer

### Step 3: Evaluate Tool Usage Across Benchmarks

Analyze what types of tools models generate and their effectiveness:

```python
from collections import defaultdict
from typing import List, Dict

class ToolAnalyzer:
    """Analyze tool usage patterns across vision benchmarks."""

    def __init__(self):
        self.tool_types = defaultdict(int)
        self.success_rate = defaultdict(float)
        self.performance_by_tool = defaultdict(list)

    def classify_tool_type(self, code: str) -> str:
        """Classify generated code by tool type."""
        if "detect" in code.lower() or "yolo" in code.lower():
            return "object_detection"
        elif "segment" in code.lower():
            return "segmentation"
        elif "color" in code.lower():
            return "color_analysis"
        elif "edge" in code.lower():
            return "edge_detection"
        elif "measure" in code.lower() or "distance" in code.lower():
            return "measurement"
        elif "ocr" in code.lower() or "text" in code.lower():
            return "text_recognition"
        else:
            return "custom_analysis"

    def evaluate_benchmark(self, benchmark_name: str,
                          dataset: List[Dict],
                          solver: AgenticVisionSolver) -> Dict:
        """Evaluate solver on benchmark, tracking tool usage."""

        results = {
            "total": 0,
            "correct": 0,
            "tool_types": defaultdict(int),
            "tool_success": defaultdict(int),
            "accuracy_by_tool": defaultdict(list)
        }

        for sample in dataset:
            image_path = sample["image"]
            question = sample["question"]
            ground_truth = sample["answer"]

            # Solve with agentic approach
            predicted = solver.solve(image_path, question)

            # Check correctness
            is_correct = (predicted.lower().strip() ==
                         ground_truth.lower().strip())

            results["total"] += 1
            if is_correct:
                results["correct"] += 1

            # TODO: Track which tools were used
            # This requires extracting code from solver's history

        accuracy = results["correct"] / results["total"] if results["total"] > 0 else 0
        results["accuracy"] = accuracy

        return results

def evaluate_pyvision_suite(solver: AgenticVisionSolver,
                           benchmarks: Dict[str, list]) -> Dict:
    """Evaluate PyVision across multiple benchmarks."""

    analyzer = ToolAnalyzer()
    results = {}

    for bench_name, dataset in benchmarks.items():
        print(f"Evaluating {bench_name}...")
        bench_results = analyzer.evaluate_benchmark(bench_name, dataset, solver)
        results[bench_name] = bench_results
        print(f"  Accuracy: {bench_results['accuracy']:.2%}")

    return results

Step 4: Interactive Refinement Loop

Enable multi-turn refinement where the model improves its answer based on tool feedback:

def interactive_vqa(image_path: str,
                   question: str,
                   solver: AgenticVisionSolver,
                   max_turns: int = 5) -> str:
    """
    Interactive VQA with user feedback for refinement.
    """
    sandbox = VisionToolSandbox(image_path, max_iterations=max_turns)
    answer = solver.solve(image_path, question)

    print(f"Initial answer: {answer}")

    for turn in range(max_turns):
        feedback = input("Is this answer correct? (yes/no/refine): ").strip().lower()

        if feedback == "yes":
            return answer

        elif feedback == "no":
            # Ask model to try a different approach
            correction_prompt = f"That was incorrect. Try a different analysis approach."
            # TODO: Integrate feedback into solver loop

        elif feedback.startswith("refine"):
            # Use specific feedback to improve
            specific_feedback = feedback[7:].strip()
            # TODO: Incorporate specific feedback

    return answer

Practical Guidance

When to use PyVision:

• Complex visual reasoning tasks requiring multi-step analysis
• Scenarios where domain-specific processing helps (medical, satellite imagery)
• Tasks where self-correction is beneficial (model can debug code)
• Interactive VQA where iterative refinement matters
• Research into how LLMs approach visual decomposition

When NOT to use PyVision:

• Real-time inference (multiple iterations add latency)
• Simple visual tasks (direct reasoning faster)
• Highly constrained environments (code execution may not be allowed)
• Latency-critical applications (agentic loops are slow)
• Cases where interpretability is critical (code execution is black-box)

Common pitfalls:

• Timeout too short, cutting off long-running analysis
• Insufficient error context, making debugging hard for model
• Not handling import errors (missing libraries)
• Sandbox isolation too strict, preventing necessary operations
• Model generating non-terminating loops (infinite code)
• Not validating generated code before execution
• Over-iterating: model gets stuck in loops trying same approach

Reference

Chen, H., Wang, Y., Zhang, X., & Liu, J. (2025). PyVision: Agentic Vision with Dynamic Tooling. arXiv:2507.07998. https://arxiv.org/abs/2507.07998