Perla 3d Eval

perla-3d-eval

PerLA: Perceptive 3D Language Assistant — Mei et al. (2024) (arXiv:2411.19774, 2024)

What this evaluates

Evaluates a 3D vision-language model's ability to answer questions about indoor scenes and generate dense captions for 3D instances. It probes fine-grained spatial reasoning, object attribute recognition, and scene understanding by comparing generated text against ground-truth annotations using standard natural language generation metrics.

Datasets

• ScanNet — total 1513; splits: train (1201), val (312)

Metrics

• CiDEr (primary) — range: percent
  • TF-IDF weighted n-gram overlap between predicted and ground-truth text. Scores are scaled to [0, 100] following standard NLG evaluation conventions.
• BLEU-4 — range: percent
  • 4-gram precision with brevity penalty, scaled to [0, 100].
• METEOR — range: percent
  • Unigram precision/recall with synonymy and stemming matching, scaled to [0, 100].
• Rouge-L — range: percent
  • Longest common subsequence F1 score between predictions and references, scaled to [0, 100].

Input / output format

Input: 3D point cloud scene (40,000 randomly sampled points) paired with a natural language question (for QA) or used alone (for dense captioning).

Output: Natural language text response (for QA) or localized 3D instance descriptions with bounding box references (for dense captioning).

Scoring recipe

def evaluate(predictions, golds, task='qa'):
    scores = {}
    scores['CiDEr'] = cider(predictions, golds)
    scores['BLEU-4'] = bleu(predictions, golds, n=4)
    scores['METEOR'] = meteor(predictions, golds)
    scores['Rouge-L'] = rouge(predictions, golds, 'L')
    if task == 'dense_captioning':
        for thresh in [0.25, 0.5]:
            for m in ['CiDEr', 'BLEU-4', 'METEOR', 'Rouge-L']:
                scores[f'{m}@{thresh}'] = apply_iou_threshold(predictions, golds, m, thresh)
    return scores

Common pitfalls

• Classification-based (CLS) methods select from a fixed answer set while generation-based (GEN) methods produce free text; direct metric comparison requires careful handling of answer vocabularies and tokenization.
• Dense captioning metrics use m@$k$IoU thresholds that vary by dataset (0.25 for ScanRefer, 0.5 for Nr3D); failing to specify the threshold leads to incomparable results.
• Point clouds are downsampled to 40,000 random points per scene, which may underrepresent fine-grained geometry compared to full-resolution scans and bias performance evaluations.

Evidence (verbatim from paper)

We follow LL3DA’s evaluation protocol*[[9]]* to evaluate the quality of output responses. We use the abbreviations C, B4, M and R for CiDEr*[[55]], BLEU-4[[43]], METEOR[[3]], and Rouge-L[[34]]*, respectively.

Citation

@misc{mei2024perla,
  title={PerLA: Perceptive 3D Language Assistant},
  author={Mei et al. (2024)},
  year={2024},
  note={arXiv:2411.19774}
}

• arXiv: 2411.19774