Tir Bench Eval

tir-bench-eval

TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning — Ming Li et al. (2025) (arXiv:2511.01833, 2025)

What this evaluates

Evaluates multimodal language models' ability to perform agentic reasoning with images, specifically requiring dynamic visual manipulation and tool-use to solve complex tasks like rotation, jigsaw assembly, and instrument reading. It probes whether models can iteratively process, crop, or transform visual inputs to extract information or solve spatial problems.

Datasets

• TIR-Bench — total ?; splits: test (-1); repo https://github.com/agents-x-project/TIR-Bench

Metrics

• accuracy (primary) — range: percent
  • For multiple-choice and short-form answers, the final answer is extracted via GPT-4o and compared directly against the ground-truth value. For grounding-type problems (e.g., Jigsaw Game, Spot the Difference), intersection over union (IoU) is calculated. Results are reported as percentage accuracy across tasks.

Input / output format

Input: A single image and a text prompt/question. Models are evaluated in a zero-shot setting without task-specific fine-tuning or in-context examples.

Output: Text response containing reasoning steps and/or a final answer. For grounding tasks, models output coordinates or list-type answers. Final answers are extracted by GPT-4o before scoring.

Scoring recipe

def compute_accuracy(predictions, golds, task_types):
    correct = 0
    for pred, gold, task in zip(predictions, golds, task_types):
        if task in ['grounding', 'jigsaw', 'spot_difference']:
            iou = calculate_iou(pred, gold)
            correct += (1.0 if iou > 0.5 else 0.0)
        else:
            extracted = gpt4o_extract_answer(pred)
            correct += (1.0 if extracted == gold else 0.0)
    return (correct / len(predictions)) * 100

Common pitfalls

• Answer extraction relies on GPT-4o, which may introduce parsing inconsistencies or model-specific biases compared to deterministic string matching.
• IoU calculation for list-type grounding answers requires explicit handling of coordinate formats and list ordering, which is not detailed in the paper.
• Agentic models require external tool-use environments (e.g., code interpreters, sandboxes) to function, making zero-shot API evaluation setup-dependent and potentially unfair to non-agentic baselines.

Evidence (verbatim from paper)

We follow previous works*(lu2023mathvista; li2025eee)* to first generate answers from models and subsequently using GPT-4o to extract the final answer from the answer content. For multiple-choice and short-form answers, we compare the extracted value directly against the ground-truth to calculate accuracy; for grounding type problems such as Jigsaw Game and Spot the Difference with list type answer, we calculate the intersection over union (IoU).

Citation

@misc{li2025tirbench,
  title={TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning},
  author={Ming Li et al. (2025)},
  year={2025},
  note={arXiv:2511.01833}
}

• arXiv: 2511.01833