Pts3d LLM Eval

pts3d-llm-eval

Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models — Thomas et al. (2025) (arXiv:2506.05689, 2025)

What this evaluates

Evaluates multimodal large language models on 3D scene understanding tasks, including visual grounding, dense captioning, and spatial/situated question answering. It specifically probes how different 3D token structures (point-based vs. video-based) and feature fusion strategies impact performance on indoor RGB-D scans.

Datasets

• ScanRefer — total ?; splits: val (-1)
• Multi3DRefer — total ?; splits: val (-1)
• Scan2Cap — total ?; splits: val (-1)
• ScanQA — total ?; splits: val (-1)
• SQA3D — total ?; splits: test (-1)

Metrics

• Ac25 / Ac50 — range: percent
  • Accuracy at IoU thresholds of 0.25 and 0.5 for single-target 3D visual grounding. A prediction is correct if its IoU with the ground truth exceeds the threshold.
• F125 / F150 — range: percent
  • F1 scores at IoU thresholds of 0.25 and 0.5 for multi-target 3D visual grounding, accounting for variable numbers of target objects.
• C50 / B450 — range: percent
  • CIDEr@0.5IoU (C50) and BLEU-4@0.5IoU (B450) for 3D dense captioning, combining captioning metrics with IoU-based bounding box alignment.
• C / EM — range: percent
  • CIDEr (C) and exact match accuracy (EM) for 3D question answering (ScanQA).
• EM — range: percent
  • Exact match accuracy for situated question answering (SQA3D).
• NS (primary) — range: percent
  • Normalized Score: average of multiple metrics relative to a state-of-the-art baseline. Computed as s_NS = (1/|M|) * sum(100 * s_m / s^sota_m) for m in M.

Input / output format

Input: RGB-D video frames sampled at 3 FPS with corresponding camera intrinsics/extrinsics, 3D point clouds, and Mask3D-generated object proposals. Text inputs include object queries, caption prompts, or questions.

Output: Predicted 3D bounding boxes or point coordinates for grounding, generated captions for dense captioning, and natural language answers for QA tasks.

Scoring recipe

def compute_iou_accuracy(pred_boxes, gt_boxes, iou_thresh=0.25):
    ious = [compute_iou(p, g) for p, g in zip(pred_boxes, gt_boxes)]
    return sum(1 for i in ious if i >= iou_thresh) / len(gt_boxes) * 100

def compute_normalized_score(model_scores, sota_scores, metrics):
    ns = 0.0
    for m in metrics:
        ns += 100 * (model_scores[m] / sota_scores[m])
    return ns / len(metrics)

Common pitfalls

• Evaluating on the test set for ScanRefer, Multi3DRefer, Scan2Cap, and ScanQA instead of their official validation sets.
• Reporting the Normalized Score (NS) as an absolute metric; it is relative to specific SOTA baselines and varies depending on which baseline scores are used as the denominator.
• Averaging results over only one seed; the paper requires averaging over 10 seeds for main comparisons and 5 seeds for ablations to report mean ± std.

Evidence (verbatim from paper)

We evaluate our model on five widely-used 3D scene understanding benchmarks, all derived from the ScanNet dataset [[26]], which includes 1,513 richly annotated RGB-D video scans of indoor scenes. ... Evaluations are conducted on the validation sets for ScanRefer, Multi3DRefer, Scan2Cap, and ScanQA, and the test set for SQA3D... For ScanRefer, we report accuracy at IoU thresholds of 0.25 and 0.5 (Ac25, Ac50)... For Multi3DRefer, we use F1 scores at IoU thresholds of 0.25 and 0.5 (F125, F150)... For Scan2Cap, we compute CIDEr@0.5IoU (C50) and BLEU-4@0.5IoU (B450)... For ScanQA, we report CIDEr (C) and exact match accuracy (EM)... For SQA3D, we use exact match accuracy (EM)... Finally, we also use a normalized score (NS) metric... We compute it as an average of multiple metrics relative to the state of the art.

Citation

@misc{thomas2025pts3dllm,
  title={Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models},
  author={Thomas et al. (2025)},
  year={2025},
  note={arXiv:2506.05689}
}

• arXiv: 2506.05689