3d 2d Vl Grounding Eval

3d-2d-vl-grounding-eval

Unifying 2D and 3D Vision-Language Understanding — Ayush Jain et al. (2025) (arXiv:2503.10745, 2025)

What this evaluates

Evaluates a unified vision-language model's ability to ground natural language instructions to 3D objects and 2D regions, as well as answer 3D visual questions. It probes spatial reasoning, cross-modal alignment, and robustness to different 3D input representations (mesh-sampled vs. sensor RGB-D point clouds).

Datasets

• SR3D — total ?; splits: val (-1)
• NR3D — total ?; splits: val (-1)
• ScanRefer — total ?; splits: val (-1)
• RefCOCO — total ?; splits: val (-1)
• RefCOCO+ — total ?; splits: val (-1)
• RefCOCOg — total ?; splits: val (-1)
• ScanQA — total ?; splits: val (-1)
• SQA3D — total ?; splits: val (-1)

Metrics

• top-1 accuracy (Acc@25/50/75) (primary) — range: percent
  • Top-1 accuracy computed at IoU thresholds of 0.25, 0.5, and 0.75. A prediction is correct if the IoU between the predicted bounding box (derived from mask extents) and the ground-truth box exceeds the threshold. For the GT setup, correctness is determined by selecting the correct pooled feature token.
• exact-match accuracy (EM@1) — range: percent
  • Measures whether the generated text answer exactly matches one of the provided ground-truth answer candidates (1 for SQA3D, 2 for ScanQA).

Input / output format

Input: 3D point clouds (either mesh-sampled or sensor RGB-D unprojected) paired with natural language instructions/queries; or 2D images paired with language instructions.

Output: Predicted segmentation masks (converted to bounding boxes via extents) for grounding tasks; or generated text answers for VQA tasks.

Scoring recipe

def compute_accuracy(preds, golds, iou_threshold=0.25):
    correct = 0
    for pred, gold in zip(preds, golds):
        pred_box = get_bbox_from_mask(pred)
        iou = intersection_over_union(pred_box, gold)
        if iou >= iou_threshold:
            correct += 1
    return correct / len(golds)

def compute_em(preds, golds):
    matches = sum(1 for p, g in zip(preds, golds) if p in g)
    return matches / len(golds)

Common pitfalls

• Confusing the GT setup (uses ground-truth object proposals) with the Det setup (requires object detection without GT boxes), which drastically changes the difficulty and reported accuracy.
• Mesh-sampled point clouds are standard in benchmarks but misalign with real sensor data, unfairly penalizing sensor-based methods; evaluations should explicitly note input type.
• Mask decoding relies on dot-products with visual tokens, while box decoding regresses coordinates; comparing them directly without accounting for this architectural difference is misleading.

Evidence (verbatim from paper)

We use the standard top-1 accuracy metric. For the Det setup, a predicted bounding box is considered correct if its intersection over union (IoU) with the ground truth box is higher than a predetermined threshold (we use the standard 0.25, 0.5 and 0.75). As UniVLG predicts masks (instead of axis-aligned bounding boxes), we obtain a bounding box by taking the extents of the mask.

Citation

@misc{jain2025unifying,
  title={Unifying 2D and 3D Vision-Language Understanding},
  author={Ayush Jain et al. (2025)},
  year={2025},
  note={arXiv:2503.10745}
}

• arXiv: 2503.10745