Scanreason Eval

scanreason-eval

ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilities — Zhu et al. (2024) (arXiv:2407.01525, 2024)

What this evaluates

Evaluates a model's ability to perform 3D visual grounding by jointly reasoning about implicit human instructions and localizing target objects in 3D scenes. It probes spatial, functional, logical, emotional, and safety-related reasoning capabilities alongside precise 3D bounding box localization.

Datasets

• ScanReason — total 10000; splits: test (-1)

Metrics

• matching score (primary) — range: [0, 1]
  • Not explicitly defined in the provided Method section. The prediction head outputs a matching score used to evaluate localization quality and filter successful object locations in the Chain-of-Grounding mechanism.

Input / output format

Input: 3D scene representation (point cloud features or multi-view 2D images back-projected to 3D) combined with a natural language instruction or question.

Output: Text answer and/or 3D bounding box prediction(s). For grounding tasks, the model outputs a special token followed by the predicted 3D box coordinates.

Scoring recipe

def evaluate(predictions, gold):
    # Compute IoU between predicted and ground-truth 3D boxes
    iou = compute_iou(predictions.bbox, gold.bbox)
    # Use matching score or confidence threshold for filtering
    if predictions.confidence > threshold:
        recall_at_k = calculate_recall(iou, k=10)
    return recall_at_k

Common pitfalls

• Models may struggle with implicit instructions that require multi-step spatial or functional reasoning rather than direct object matching.
• The Chain-of-Grounding mechanism requires careful threshold tuning for confidence scores to avoid error propagation across interleaved reasoning and grounding steps.

Evidence (verbatim from paper)

Finally, the prediction head takes the updated object queries as input and predicts the final 3D locations and matching score.

Citation

@misc{zhu2024scanreason,
  title={ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilities},
  author={Zhu et al. (2024)},
  year={2024},
  note={arXiv:2407.01525}
}

• arXiv: 2407.01525