Mmscan Visual Grounding Eval

mmscan-visual-grounding-eval

MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations — Lyu et al. (2024) (arXiv:2406.09401, 2024)

What this evaluates

Evaluates a model's ability to localize specific 3D objects or regions within a large-scale scene based on complex natural language prompts. It probes spatial reasoning, attribute understanding, and multi-target grounding capabilities in 3D point cloud environments.

Datasets

• MMScan (3D Visual Grounding) — total 1275586; splits: train (848867), val (217002), test (209717); repo https://github.com/OpenRobotLab/EmbodiedScan

Metrics

• gTop-k (primary) — range: [0, 1]
  • Generalized Top-k score for multi-target grounding: gTop(k) = (1/N) * sum_{i=1}^{N} [Hit(min(i*k, M)) >= i]. N is the number of GT boxes, M is the number of predicted boxes, and Hit(X) counts how many of the top-X scored predictions match any GT box at a given IoU threshold.
• AP_sample — range: [0, 1]
  • Average Precision computed across all prompt samples, treating each sample as an independent detection task.
• AP_box — range: [0, 1]
  • Average Precision computed across all predicted bounding boxes, aggregating confidence scores across samples within the same subclass.
• AR — range: [0, 1]
  • Average Recall measuring the proportion of GT boxes successfully matched by predictions at a given IoU threshold.

Input / output format

Input: 3D scene representation (reconstructed point clouds or RGB-D features) paired with a natural language prompt describing one or more target objects/regions (e.g., single-target attribute, object-object attribute, object-region).

Output: A ranked list of predicted 3D bounding boxes with associated confidence scores for each input prompt.

Scoring recipe

def compute_gTop_k(pred_boxes, gt_boxes, k, iou_thresh):
    N = len(gt_boxes)
    M = len(pred_boxes)
    score = 0.0
    for i in range(1, N + 1):
        top_x = min(i * k, M)
        top_preds = pred_boxes[:top_x]
        matches = sum(1 for p in top_preds if any(box_iou(p, g) >= iou_thresh for g in gt_boxes))
        if matches >= i:
            score += 1.0
    return score / N

Common pitfalls

• AP_box aggregates confidence scores across all samples in a subclass, which can unfairly penalize models whose confidence distributions are not well-calibrated across different prompts, whereas AP_sample evaluates each prompt independently.
• Traditional Top-k metrics assume exactly one ground truth target; they fail when prompts require grounding multiple objects, necessitating the use of gTop-k.
• Evaluations in the paper report results on the validation set after training on only 20% of the training data to save compute, which may not reflect full-data training performance.

Evidence (verbatim from paper)

Therefore, we propose gTop-k (generalized Top-k score), which extends the Top-k metric to multi-target cases. Specifically, we define it as: | | $gTop(k)=\frac{1}{N}\sum_{i=1}^{N}[Hit(min(ik,M))>=i],$ | | (1) | where $N$ is the number of ground truth (GT) boxes, $M$ is the number of predicted boxes, and Hit(X) represents the number of boxes with top-X scores that can match GT boxes (with IoU below a specific threshold).

Citation

@misc{lyu2024mmscan,
  title={MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations},
  author={Lyu et al. (2024)},
  year={2024},
  note={arXiv:2406.09401}
}

• arXiv: 2406.09401