Multimodal Tool Use Eval

multimodal-tool-use-eval

Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Models — Yan et al. (2026) (arXiv:2604.08545, 2026)

What this evaluates

Evaluates the ability of agentic multimodal models to perform visual perception, document understanding, and mathematical reasoning. It specifically probes whether models can strategically decide when to invoke external tools (e.g., image cropping, web search, Python code execution) versus answering directly, balancing task accuracy with tool efficiency.

Datasets

• V-Bench — total ?; splits: test (-1)
• HRBench-4K/8K — total ?; splits: test (-1)
• TreeBench — total ?; splits: test (-1)
• MME-RealWorld — total ?; splits: test (-1)
• SEEDBench2-Plus — total ?; splits: test (-1)
• CharXiv — total ?; splits: test (-1)
• MathVista_mini — total ?; splits: test (-1)
• MathVerse_mini — total ?; splits: test (-1)
• WeMath — total ?; splits: test (-1)
• DynaMath — total ?; splits: test (-1)
• LogicVista — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions, computed as (number of correct predictions / total number of prompts) * 100.
• tool_invocation_count — range: other
  • Total number of tool calls (e.g., image cropping, web search, Python code execution) made per prompt during inference.

Input / output format

Input: Multimodal prompts containing an image and a text question, requiring the model to decide whether to answer directly or invoke tools.

Output: Textual response, optionally interleaved with tool execution blocks (e.g., Python code, search queries, or crop coordinates) before the final answer.

Scoring recipe

def compute_metrics(predictions, golds, tool_calls):
    correct = sum(1 for pred, gold in zip(predictions, golds) if normalize_answer(pred) == normalize_answer(gold))
    accuracy = (correct / len(golds)) * 100
    avg_tools = sum(len(tc) for tc in tool_calls) / len(tool_calls)
    return {'accuracy': accuracy, 'avg_tool_invocations': avg_tools}

Common pitfalls

• The paper evaluates on 'mini' splits of MathVista and MathVerse, not the full benchmarks, which significantly reduces sample size.
• CharXiv is split into descriptive questions (DQ) and reasoning questions (RQ); reporting only DQ or averaging without distinction misrepresents performance on complex tasks.
• Tool efficiency is measured by invocation count reduction, but the paper does not provide a standardized threshold for what constitutes a 'necessary' vs 'redundant' tool call, making cross-model efficiency comparisons sensitive to the baseline's verbosity.

Evidence (verbatim from paper)

These results empirically validate our core hypothesis: task accuracy and tool efficiency are not inherently conflicting. By decoupling the two objectives and eliminating gradient entanglement, HDPO successfully suppresses noisy, redundant tool invocations, which in turn consistently elevates the final reasoning accuracy.

Citation

@misc{yan2026actwisely,
  title={Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Models},
  author={Yan et al. (2026)},
  year={2026},
  note={arXiv:2604.08545}
}

• arXiv: 2604.08545