qhjqhj00 multi3drefer-eval
Multi3DRefer: Grounding Text Description to Multiple 3D Objects — Zhang et al. (2023) (arXiv:2309.05251, 2023)
What this evaluates
Evaluates 3D visual grounding models on their ability to localize zero, single, or multiple objects in 3D scenes based on natural language descriptions. It tests whether models can correctly identify target objects while handling ambiguous references, distractors, and zero-target cases.
Datasets
- Multi3DRefer — total ?; splits: train (3826), val (2862), test (13178)
Metrics
Acc@0.5(primary) — range: percent- Percentage of correctly grounded instances where the Intersection over Union (IoU) between the predicted and ground-truth bounding box exceeds 0.5.
F1@0.5— range: percent- Harmonic mean of precision and recall computed at an IoU threshold of 0.5, evaluating both correct localization and correct rejection of zero-target cases.
Input / output format
Input: 3D point cloud scene (coordinates, normals, multi-view features), natural language description, and optionally ground-truth or predicted bounding boxes for object proposals.
Output: Predicted 3D bounding box (or set of boxes) corresponding to the text description.
Scoring recipe
def compute_acc_at_05(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if iou(p, g) >= 0.5)
return (correct / len(gold)) * 100
def compute_f1_at_05(predictions, gold):
tp = sum(1 for p, g in zip(predictions, gold) if iou(p, g) >= 0.5)
fp = len(predictions) - tp
fn = len(gold) - tp
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
Common pitfalls
- Must explicitly state whether ground-truth or predicted bounding boxes are used, as performance drops significantly with predicted boxes.
- Zero-target cases require models to correctly reject all proposals; failing to do so heavily penalizes recall and F1.
- Distractors (objects of the same class as the target) are explicitly included in some splits and drastically lower performance if not handled.
Evidence (verbatim from paper)
We report F1 scores with GT boxes on Multi3DRefer val set.
Citation
@misc{zhang2023multi3drefer,
title={Multi3DRefer: Grounding Text Description to Multiple 3D Objects},
author={Zhang et al. (2023)},
year={2023},
note={arXiv:2309.05251}
}
- arXiv: 2309.05251