Vtc Bench Eval

vtc-bench-eval

VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining — Zhu et al. (2026) (arXiv:2603.15030, 2026)

What this evaluates

Evaluates multimodal large language models' ability to perform agentic visual reasoning by composing multiple OpenCV-based tool calls. It probes long-horizon planning, precise tool selection, and the capacity to chain coarse- to fine-grained visual operations to solve complex multi-step problems.

Datasets

• VTC-Bench — total 680; splits: test (680)

Metrics

• Average Pass Rate (APR) (primary) — range: percent
  • The percentage of benchmark problems for which the model produces the correct final answer within the maximum allowed interaction rounds. Calculated as (number of correctly solved problems / total problems) × 100.
• Tool Call Rate (TCR) — range: percent
  • The percentage of problems where the model successfully invokes at least one tool.
• Tool Usage Efficiency (Eff.) — range: percent
  • The ratio of effective toolchain steps to total attempted tool steps, measuring how closely the model's execution path matches the ground-truth trajectory.

Input / output format

Input: A visual input (image) paired with a natural language task prompt describing the required visual reasoning or operation.

Output: A final textual or numerical answer generated after an iterative loop of tool calls and execution results, constrained by a maximum round limit.

Scoring recipe

def compute_apr(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if deterministic_match(pred, gold):
            correct += 1
        elif llm_judge_match(pred, gold, model='gpt-4o'):
            correct += 1
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Models often achieve high pass rates but use excessive redundant tool calls, failing to align with optimal ground-truth execution paths.
• Agents frequently over-rely on unverified intermediate tool outputs, bypassing critical cross-verification with the original visual input.
• Interface-based and code-based invocation yield different performance profiles, making direct comparison without controlling for the execution paradigm misleading.

Evidence (verbatim from paper)

The left panel of Fig.[5] reveals a positive correlation between Tool Call Rate (TCR) and Average Pass Rate (APR), validating the utility of active tool invocation. Conversely, the efficiency analysis in the right panel indicates that general-purpose models achieve superior efficiency even with lower TCR.

Citation

@misc{zhu2026vtcbench,
  title={VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining},
  author={Zhu et al. (2026)},
  year={2026},
  note={arXiv:2603.15030}
}

• arXiv: 2603.15030