Crop And Zoom Tool Use Eval

crop-and-zoom-tool-use-eval

What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom — Yan Ma et al. (2026) (arXiv:2602.01334, 2026)

What this evaluates

Evaluates vision-language models' ability to use a crop-and-zoom tool for high-resolution visual question answering, disentangling intrinsic capability improvements from tool-induced gains and harms across multiple benchmarks.

Datasets

• VStar, HR-Bench 4k/8k, VisualProbe Easy/Medium/Harm — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Binary correctness of the final textual answer against the ground truth. Computed separately for tool-available (Acc_w) and tool-free (Acc_wo) protocols.
• tool_induced_gap — range: [-1, 1]
  • Difference between tool-available and tool-free accuracy (G(t) = Acc_w - Acc_wo), decomposed into Gross Gain (Call Gain + Schema Gain) and Gross Harm (Call Harm + Schema Harm).

Input / output format

Input: An image and a natural language question.

Output: A final textual answer to the question.

Scoring recipe

# Compute baseline and tool-assisted accuracy
acc_wo = mean(1 for pred, gold in zip(predictions_tool_free, golds) if pred == gold)
acc_w  = mean(1 for pred, gold in zip(predictions_tool_available, golds) if pred == gold)
# Compute tool-induced gap and decomposition
G = acc_w - acc_wo
# Factorize into Mass, Policy, Quality for specific subsets (fail/succ)
# Alignment: intersection of human and AI judgments on Call Gain samples

Common pitfalls

• Using Acc_schema (schema provided but execution forbidden) as the baseline instead of Acc_wo artificially inflates tool gains by treating recovery from an artificial constraint as improvement.
• Evaluating call quality P(✓|c, D_fail) on the dynamic failure set at each checkpoint confuses execution quality improvements with the shifting difficulty of the remaining failures.
• Assuming tool-use RL primarily optimizes tool handling; the protocol shows intrinsic capability drift dominates performance gains.

Evidence (verbatim from paper)

We evaluate models under both tool-available and tool-free protocols on six benchmarks (VStar, HR-Bench 4k/8k, VisualProbe Easy/Medium/Harm), with greedy decoding (temperature = 0) at regular checkpoints throughout training. The absolute accuracy for both Acc_w (solid line) and Acc_wo (dashed line) increases monotonically.

Citation

@misc{ma2026visiontoolusereinforcement,
  title={What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom},
  author={Yan Ma et al. (2026)},
  year={2026},
  note={arXiv:2602.01334}
}

• arXiv: 2602.01334