Robo Refer Spatial Eval

robo-refer-spatial-eval

RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics — Zhou et al. (2025) (arXiv:2506.04308, 2025)

What this evaluates

Evaluates vision-language models' ability to perform single-step and multi-step spatial understanding and referring tasks in robotics contexts. It probes capabilities like 2D/3D relation reasoning, depth perception, and complex compositional spatial constraints in cluttered scenes.

Datasets

• CV-Bench — total ?; splits: (unstated)
• BLINK — total ?; splits: val (-1)
• RoboSpatial — total ?; splits: (unstated)
• RefSpatial-Bench — total 200; splits: Location (100), Placement (100)
• RefCOCO — total ?; splits: val (-1), testA (-1), testB (-1)

Metrics

• Top-1 accuracy (primary) — range: [0, 1]
  • Calculated as the proportion of correctly predicted instances out of the total. For referring tasks, a prediction is correct if the predicted point lies within the ground-truth mask or bounding box.
• Success rate — range: percent
  • Average success rate of predicted points within the ground-truth mask or bounding box, reported as a percentage.

Input / output format

Input: RGB or RGB-D image paired with a natural language spatial query or instruction.

Output: Predicted 2D point coordinates (x, y) or bounding box, or a multiple-choice label.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if isinstance(gold, dict) and 'mask' in gold:
            if point_in_mask(pred, gold['mask']): correct += 1
        elif isinstance(gold, dict) and 'bbox' in gold:
            if point_in_bbox(pred, gold['bbox']): correct += 1
        else:
            if pred == gold: correct += 1
    return correct / len(predictions)

Common pitfalls

• Evaluating point predictions against bounding boxes using IoU instead of point-in-box containment.
• Ignoring the impact of RGB vs. RGB-D inputs, which drastically changes 3D spatial benchmark performance.
• Assuming single-step accuracy generalizes to multi-step reasoning, which requires sequential refinement.

Evidence (verbatim from paper)

For metrics, we report the average success rate of predicted points within the mask. ... Since our model predicts a single point, we deem a prediction correct if the point lies within the ground-truth bounding box.

Citation

@misc{zhou2025roborefer,
  title={RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics},
  author={Zhou et al. (2025)},
  year={2025},
  note={arXiv:2506.04308}
}

• arXiv: 2506.04308