Gar Bench Eval

gar-bench-eval

Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs — Wang et al. (2025) (arXiv:2510.18876, 2025)

What this evaluates

Evaluates a multimodal LLM's ability to perform precise, context-aware visual understanding at the region level. It probes fine-grained perception, compositional reasoning across multiple visual prompts, and detailed localized captioning for both images and videos.

Datasets

• GAR-Bench-VQA — total ?; splits: test (-1)
• GAR-Bench-Cap — total 204; splits: test (204)
• DLC-Bench — total ?; splits: test (-1)
• Ferret-Bench — total ?; splits: test (-1)
• MDVP-Bench — total ?; splits: test (-1)
• LVIS — total ?; splits: test (-1)
• PACO — total ?; splits: test (-1)
• VideoRefer-Bench — total ?; splits: test (-1)

Metrics

• Overall score (primary) — range: percent
  • Percentage accuracy or LLM-judge score aggregated across subtasks (Perception, Reasoning, Color, Texture). For captioning tasks, GPT-4o is used as an automated judge to score generated descriptions against ground truth.
• Semantic Similarity & IoU — range: percent
  • Semantic similarity (Sim.) measures textual overlap/embedding similarity between predicted and ground-truth categories. IoU measures the intersection-over-union between predicted and ground-truth bounding boxes.

Input / output format

Input: An image (or sampled video frames) with a specified region prompt (bounding box or mask), accompanied by a question or captioning instruction.

Output: Text response containing either a direct answer to the VQA question or a detailed localized caption describing the specified region.

Scoring recipe

# For VQA/Captioning
if judge == "GPT-4o":
    score = gpt4o_judge(model_output, ground_truth, cropped_image, mask)
else:
    score = exact_match(model_output, ground_truth)
overall = mean(scores)

# For LVIS/PACO
sim = semantic_similarity(predicted_cat, ground_truth_cat)
iou = compute_iou(predicted_box, ground_truth_box)

Common pitfalls

• Captioning scores heavily depend on the LLM judge (GPT-4o) configuration; using cropped images with masks significantly changes scores compared to text-only judging.
• Video evaluation uses uniform 16-frame sampling and disables AnyRes for multi-frame tasks, which may underperform temporally dense videos.
• Zero-shot video transfer shows low scores on temporal/future prediction tasks because models are trained only on images.

Evidence (verbatim from paper)

Table 5: Results of category-level image recognition on LVIS [[15]] and PACO [[36]] following Osprey [[60]]. Our GAR-8B demonstrates a significant leap in performance, establishing a new state-of-the-art. It consistently outperforms all prior methods across every metric, achieving top scores of 93.6 semantic similarity and 88.7 semantic IoU on LVIS *[[15]], and 95.5 semantic similarity and 91.8 semantic IoU on PACO *[[36]].

Citation

@misc{wang2025graspanyregion,
  title={Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.18876}
}

• arXiv: 2510.18876