C2rope 3d Vqa Eval

c2rope-3d-vqa-eval

C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoning — Ye et al. (2026) (arXiv:2602.10551, 2026)

What this evaluates

Evaluates a 3D large multimodal model's ability to perform spatial reasoning and visual question answering on multi-view 3D scene data. It probes the model's capacity to retain early visual context, understand spatial relationships, and generate accurate text responses to complex 3D scene queries.

Datasets

• ScanQA — total 33400; splits: test (-1), val (-1)
• SQA3D — total 19000; splits: test (-1), val (-1)

Metrics

• EM@1 (primary) — range: [0, 1]
  • Exact Match at 1: 1 if the generated response exactly matches the ground truth answer (case-insensitive, stripped), else 0. Averaged over the dataset.
• EM@R — range: [0, 1]
  • Exact Match Recall: Measures the recall of exact matches, likely across multiple generated candidates or a relaxed matching criterion as used in the SQA3D benchmark.
• B-4 — range: [0, 1]
  • BLEU-4: N-gram precision up to 4-grams, typically with brevity penalty, comparing generated text to reference answers.
• MET — range: [0, 1]
  • METEOR: Metric for Evaluation of Translation with Explicit ORdering, considering synonymy, stemming, and alignment between generated and reference text.
• RGE — range: [0, 1]
  • ROUGE-L: Recall-Oriented Understudy for Gisting Evaluation based on the longest common subsequence between generated and reference text.
• CIDEr — range: [0, 1]
  • Consensus-based Image Description Evaluation: TF-IDF weighted n-gram similarity that measures consensus between generated captions and multiple references.

Input / output format

Input: Multi-view 3D scene images (16 views per scene, each encoded into 576 2D patches, totaling 9,216 visual tokens) paired with a natural language question/instruction.

Output: A single natural language text response answering the question.

Scoring recipe

def compute_metrics(predictions, golds):
    em1 = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower()) / len(golds)
    # EM@R, BLEU-4, METEOR, ROUGE-L, CIDEr computed via standard NLG evaluation libraries (e.g., sacrebleu, rouge-score, cider)
    return {'EM@1': em1, 'EM@R': 'computed via standard NLG libraries', 'B-4': 'computed via standard NLG libraries', 'MET': 'computed via standard NLG libraries', 'RGE': 'computed via standard NLG libraries', 'CIDEr': 'computed via standard NLG libraries'}

Common pitfalls

• Models often hallucinate spatial relationships when visual tokens are long (9,216 patches), leading to incorrect answers despite high BLEU/CIDEr scores.
• EM@R is not standard exact match; it likely measures recall of exact matches across multiple generated candidates or a relaxed matching criterion, which can be ambiguous without code.
• Evaluation relies on human-annotated (ScanQA) vs GPT-4-generated (SQA3D) questions, which may differ in difficulty and phrasing, affecting cross-dataset comparison.

Evidence (verbatim from paper)

As reported in Tab.[I], compared to the baseline LLaVA-3D, our method achieves consistent improvements across all five evaluation metrics on ScanQA, with gains of +4.3 on EM@1, +8.5 on B-4, +13.4 on MET, +2.5 on RGE, and +18.1 on CIDEr.

Citation

@misc{ye2026c2rope,
  title={C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoning},
  author={Ye et al. (2026)},
  year={2026},
  note={arXiv:2602.10551}
}

• arXiv: 2602.10551