qhjqhj00 deepeyes-eval
DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning — Zheng et al. (2025) (arXiv:2505.14362, 2025)
What this evaluates
Evaluates large vision-language models on fine-grained visual perception, grounding, hallucination mitigation, and multimodal reasoning. It specifically probes the model's ability to autonomously use image zoom-in tools for interleaved visual-linguistic reasoning (iMCoT) to solve high-resolution and complex visual tasks.
Datasets
- V Bench* — total ?; splits: test (-1)
- HR-Bench — total ?; splits: test (-1)
- refCOCO / refCOCO+ / refCOCOg / ReasonSeg — total ?; splits: test (-1)
- POPE — total ?; splits: test (-1)
- MathVista — total ?; splits: test (-1)
- MathVerse — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions. Computed as (number of correct predictions / total number of instances) * 100.
Intersection-over-Union (IoU)— range: [0, 1]- Area of overlap between the model's predicted bounding box (from tool cropping) and the ground-truth bounding box, divided by the area of their union. Used to quantify grounding quality during tool use.
Input / output format
Input: One or more images paired with a text prompt/question. The model may optionally invoke a built-in zoom-in tool to crop and inspect specific regions during its reasoning process.
Output: A text response containing interleaved reasoning steps, tool calls (if used), and a final answer.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
return (correct / len(gold)) * 100
def compute_iou(pred_box, gt_box):
# pred_box and gt_box are [x_min, y_min, x_max, y_max]
inter_x = max(0, min(pred_box[2], gt_box[2]) - max(pred_box[0], gt_box[0]))
inter_y = max(0, min(pred_box[3], gt_box[3]) - max(pred_box[1], gt_box[1]))
inter_area = inter_x * inter_y
union_area = (pred_box[2]-pred_box[0])*(pred_box[3]-pred_box[1]) + \
(gt_box[2]-gt_box[0])*(gt_box[3]-gt_box[1]) - inter_area
return inter_area / union_area if union_area > 0 else 0.0
Common pitfalls
- High-resolution benchmarks require explicit tool-use capabilities; standard VLMs without zoom-in tools will fail regardless of base reasoning ability.
- Hallucination evaluation (POPE) relies on yes/no classification; models may over-hallinate without explicit grounding verification.
- Tool-use behavior evolves during RL training; evaluating intermediate checkpoints yields misleadingly high tool counts but low accuracy.
Evidence (verbatim from paper)
From Table 2, our model achieves higher accuracy on the grounding task and shows substantial improvement in reducing hallucinations. This improvement stems from our model's ability to focus on specific regions of interest during the visual reasoning process and perform detailed analysis of these cropped areas, thereby more confidently confirming the presence or absence of objects.
Citation
@misc{zheng2025deepeyes,
title={DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning},
author={Zheng et al. (2025)},
year={2025},
note={arXiv:2505.14362}
}
- arXiv: 2505.14362