Insight O3 Eval

insight-o3-eval

InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search — Li et al. (2025) (arXiv:2512.18745, 2025)

What this evaluates

Evaluates multimodal reasoning and generalized visual search capabilities, measuring how well models can locate and reason about high-information-density images using a multi-agent framework with a dedicated visual search agent.

Datasets

• V-Bench* — total ?; splits: test (-1)
• Tree-Bench — total ?; splits: test (-1)
• VisualProbe-Hard — total ?; splits: test (-1)
• HR-Bench — total ?; splits: test (-1)
• MME-RealWorld — total 1919; splits: lite (1919)
• O3-Bench — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly answered questions. Calculated as the number of correct predictions divided by the total number of predictions, multiplied by 100.

Input / output format

Input: An image and a natural language question or query.

Output: A textual answer and/or bounding box coordinates for visual search tasks.

Scoring recipe

def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

Common pitfalls

• Image resolution constraints vary significantly by model API (e.g., OpenAI limits to 1280x1280px, while Gemini allows 3500x3500px), which impacts performance.
• Models may be reluctant to call the vSearcher tool or may use it incorrectly, leading to performance drops compared to standalone evaluation.

Evidence (verbatim from paper)

We evaluate a range of open and proprietary models/systems on the following benchmarks: (1) Natural-image benchmarks: V⋆-Bench, Tree-Bench, and VisualProbe-Hard. (2) Mixed benchmarks: HR-Bench and MME-RealWorld. ... (3) Our O3-Bench. In particular, the accuracy of GPT-5-mini on O3-Bench has improved from 39.0% to 61.5%.

Citation

@misc{li2025insighto3,
  title={InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2512.18745}
}

• arXiv: 2512.18745