Descrip3d 3d Scene Eval

descrip3d-3d-scene-eval

Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions — Xue et al. (2025) (arXiv:2507.14555, 2025)

What this evaluates

Evaluates large language models' ability to perform 3D scene understanding tasks, including single and multi-object visual grounding, 3D scene captioning, and contextual question answering, using object-level text descriptions for relational reasoning.

Datasets

• ScanRefer — total ?; splits: train (-1), val (-1), test (-1)
• Multi3DRefer — total ?; splits: train (-1), val (-1), test (-1)
• Scan2Cap — total ?; splits: train (-1), val (-1), test (-1)
• ScanQA — total ?; splits: train (-1), val (-1), test (-1)
• SQA3D — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Acc@0.25 / Acc@0.5 (primary) — range: percent
  • Thresholded accuracy for single-object grounding. Predicted bounding box IoU with ground truth must exceed 0.25 or 0.5 to count as correct.
• F1@0.25 / F1@0.5 (primary) — range: percent
  • F1 score for multi-object grounding at IoU thresholds of 0.25 and 0.5, balancing precision and recall across multiple target objects.
• CIDEr@0.5 / CIDEr (primary) — range: percent
  • Consensus-based Image Description Evaluation metric. CIDEr@0.5 integrates captioning quality with spatial alignment via an IoU threshold of 0.5.
• BLEU-4@0.5 / BLEU-4 — range: percent
  • Bilingual Evaluation Understudy metric measuring 4-gram overlap between predicted and ground truth text. @0.5 applies an IoU threshold for spatial alignment in captioning.
• EM / EM-R (primary) — range: percent
  • Exact Match accuracy; EM-R is a refined variant that allows for minor paraphrasing or formatting variations while preserving semantic correctness.

Input / output format

Input: 3D point clouds, RGB images, camera poses, and object proposals per scene, formatted as instruction-following prompts with object-level text descriptions.

Output: Textual answers for QA/captioning tasks; predicted object bounding boxes or object IDs for grounding tasks.

Scoring recipe

def score_grounding(pred_boxes, gt_boxes, iou_thresh):
    ious = compute_iou(pred_boxes, gt_boxes)
    correct = sum(1 for iou in ious if iou >= iou_thresh)
    return correct / len(gt_boxes)

def score_multi_grounding(pred_boxes, gt_boxes, iou_thresh):
    prec = sum(1 for p in pred_boxes if max(compute_iou(p, gt)) >= iou_thresh)
    rec = sum(1 for g in gt_boxes if max(compute_iou(g, pred)) >= iou_thresh)
    f1 = 2 * prec * rec / (prec + rec)
    return f1

def score_text(pred_text, gt_text, metric):
    if metric == 'CIDEr': return compute_cider(pred_text, gt_text)
    if metric == 'BLEU-4': return compute_bleu(pred_text, gt_text, n=4)
    if metric == 'EM': return 1.0 if pred_text == gt_text else 0.0
    if metric == 'EM-R': return 1.0 if is_semantically_equivalent(pred_text, gt_text) else 0.0

Common pitfalls

• Using raw object names instead of unique IDs for reference causes ambiguity in scenes with multiple instances of the same category.
• BLEU-4 penalizes valid paraphrasing and diverse phrasing, making CIDEr a more reliable metric for captioning quality.
• IoU thresholds (0.25 vs 0.5) drastically change grounding scores and must be reported separately to avoid misleading comparisons.

Evidence (verbatim from paper)

For ScanRefer, we report thresholded accuracies Acc@0.25 and Acc@0.5, which assess whether the predicted object bounding box has an IoU with the ground truth exceeding 0.25 or 0.5. For Multi3DRefer, which involves grounding multiple targets, we use the F1 score at IoU thresholds of 0.25 and 0.5. For the captioning task Scan2Cap, we adopt CIDEr@0.5 and BLEU-4@0.5, integrating captioning quality with spatial alignment via IoU. For visual question answering, ScanQA is evaluated using CIDEr and BLEU-4, while SQA3D is evaluated using Exact Match (EM) and its refined variant EM-R as proposed in LEO.

Citation

@misc{xue2025descrip3d,
  title={Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions},
  author={Xue et al. (2025)},
  year={2025},
  note={arXiv:2507.14555}
}

• arXiv: 2507.14555