Treebench Eval

treebench-eval

Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology — Wang et al. (2025) (arXiv:2507.07999, 2025)

What this evaluates

Evaluates visual grounded reasoning by requiring models to precisely localize target objects in cluttered scenes and perform second-order reasoning about their interactions. It measures both the correctness of the final answer and the spatial accuracy of the traceable bounding box evidence.

Datasets

• TreeBench — total ?; splits: Perception (-1), Reasoning (-1); repo https://github.com/Haochen-Wang409/TreeVGR

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions.
• mIoU — range: [0, 1]
  • Mean Intersection over Union between predicted and ground-truth bounding boxes across all questions, measuring localization precision.

Input / output format

Input: A single image paired with a natural language question requiring object identification, spatial reasoning, or interaction analysis.

Output: A sequence of bounding boxes (traceable evidence) followed by a final answer or reasoning text.

Scoring recipe

def evaluate(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p['answer'] == g['answer'])
    acc = correct / len(gold)
    ious = []
    for p, g in zip(predictions, gold):
        pred_boxes = p['boxes']
        gt_boxes = g['boxes']
        iou_scores = compute_pairwise_iou(pred_boxes, gt_boxes)
        ious.append(mean(iou_scores))
    mIoU = mean(ious)
    return {'Accuracy': acc, 'mIoU': mIoU}

Common pitfalls

• Models may output excessive candidate bounding boxes to maximize IoU recall, leading to repetition and failure to produce a final answer.
• High localization precision (mIoU) does not automatically translate to high reasoning accuracy, as complex reasoning requires second-order cognitive capabilities beyond spatial grounding.

Evidence (verbatim from paper)

Importantly, for visual grounded reasoning models, our traceable evaluation demonstrates a positive correlation between localization precision and the overall performance, as illustrated in Table[2]. This positive correlation between precise localization (mIoU) and overall performance is evident in the progressive improvement from DeepEyes-7B to Pixel-Reasoner-7B to our final TreeVGR-7B. As mIoU increases, the overall scores rise correspondingly, with TreeVGR-7B achieving the highest mIoU and strongest overall performance at the same time.

Citation

@misc{wang2025traceable,
  title={Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2507.07999}
}

• arXiv: 2507.07999