Vision R1 Eval

vision-r1-eval

Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning — Zhan et al. (2025) (arXiv:2503.18013, 2025)

What this evaluates

Evaluates Large Vision-Language Models on their ability to detect, localize, and ground objects in images across diverse and challenging scenarios, including in-domain dense detection, out-of-domain real-world settings, and generalization to unseen categories or scenes.

Datasets

• MSCOCO Val2017 — total ?; splits: val (-1)
• ODINW-13 — total ?; splits: test (-1)

Metrics

• mAP (primary) — range: [0, 100]
  • Mean Average Precision across all object categories, computed by averaging the Area Under the Precision-Recall curve at IoU thresholds of 0.50 to 0.95 (standard COCO protocol).
• AP50 — range: [0, 100]
  • Average Precision at Intersection over Union (IoU) threshold of 0.50.
• AP75 — range: [0, 100]
  • Average Precision at Intersection over Union (IoU) threshold of 0.75.
• AR100 — range: [0, 100]
  • Average Recall at a maximum of 100 detected objects per image.

Input / output format

Input: RGB image paired with a text prompt specifying the localization task (e.g., object detection query, visual grounding instruction, or referring expression).

Output: A list of predicted bounding boxes (coordinates) and deterministic category labels for each detected object.

Scoring recipe

def compute_coco_metrics(predictions, ground_truths):
    # 1. Group predictions and ground truths by category
    # 2. For each category, compute IoU between predicted and GT boxes
    # 3. Sort predictions by deterministic match and compute PR curve
    # 4. Calculate AP at IoU thresholds 0.50, 0.75, and 0.50:0.95
    # 5. Average AP across all categories to get mAP
    # 6. Compute AR@100 by counting true positives in top-100 predictions per image
    return mAP, AP50, AP75, AR100

Common pitfalls

• ODINW evaluation follows a visual grounding setting rather than standard object detection, requiring careful prompt formatting.
• Out-of-domain evaluation relaxes the strict 'unseen category + unseen scene' constraint; only one of the two needs to be absent during post-training.
• LVLMs output deterministic category labels instead of class probabilities, so box matching relies primarily on spatial accuracy rather than Hungarian matching with confidence scores.

Evidence (verbatim from paper)

The results in Tab. [1] demonstrate the broad effectiveness of the Vision-R1 in object localization tasks. When applied to the Griffon-G model, which excels in object detection, Vision-R1 further improves its performance by 1.8 on COCO and achieves an average mAP increase of 2.5 on ODINW-13.

Citation

@misc{zhan2025visionr1,
  title={Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning},
  author={Zhan et al. (2025)},
  year={2025},
  note={arXiv:2503.18013}
}

• arXiv: 2503.18013