Gaussianvlm Eval

gaussianvlm-eval

GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond — Halacheva et al. (2025) (arXiv:2507.00886, 2025)

What this evaluates

Evaluates a 3D vision-language model's ability to perform object-centric and scene-centric reasoning tasks, including captioning, question answering, embodied planning, and dialogue. It probes spatial grounding, semantic abstraction, and robust generalization to out-of-domain real-world scene representations.

Datasets

• ScanRefer — total ?; splits: test (-1)
• ScanQA — total ?; splits: test (-1)
• Nr3D — total ?; splits: test (-1)
• SQA3D — total ?; splits: test (-1)
• 3D-LLM ScanNet subset — total ?; splits: test (-1)
• ScanNet++ (OOD object counting) — total 1000; splits: val (1000)

Metrics

• CIDEr — range: percent
  • Consensus-based Image Description Evaluation metric that weights n-grams by their IDF scores to penalize common words and reward distinctive phrases.
• BLEU-4 — range: percent
  • Precision-based metric computing the geometric mean of modified n-gram precisions up to order 4, with a brevity penalty.
• METEOR — range: percent
  • Metric that aligns predictions and references at word/phrase level, incorporating synonym matching, stemming, and exact matches to compute a harmonic mean of precision and recall.
• ROUGE — range: percent
  • Recall-Oriented Understudy for Gisting Evaluation; measures overlap of n-grams, longest common subsequence, or word sequences between prediction and reference.
• Sentence-BERT similarity (Sim) — range: [0, 1]
  • Computes cosine similarity between dense vector embeddings of the predicted and reference texts using a pre-trained Sentence-BERT model.
• Exact-match accuracy (EM1) (primary) — range: percent
  • Percentage of instances where the model's generated answer exactly matches the ground truth answer string.
• Accuracy — range: percent
  • Proportion of correctly answered questions, specifically used for the out-of-domain object counting task.

Input / output format

Input: 3D scene representation encoded as 40k randomly sampled Gaussian splats (or point clouds for baselines) paired with a natural language prompt or question.

Output: Free-form text generation (captions, answers, plans, or dialogue turns).

Scoring recipe

def compute_metrics(predictions, references):
    scores = {}
    scores['EM'] = sum(1 for p, r in zip(predictions, references) if p.strip() == r.strip()) / len(predictions)
    scores['Accuracy'] = scores['EM']
    scores['CIDEr'] = compute_cider(predictions, references)
    scores['BLEU-4'] = compute_bleu(predictions, references, n=4)
    scores['METEOR'] = compute_meteor(predictions, references)
    scores['ROUGE'] = compute_rouge(predictions, references)
    scores['Sim'] = compute_sbert_similarity(predictions, references)
    return scores

Common pitfalls

• BLEU-4 and CIDEr are excluded for object-centric tasks because they reward superficial n-gram overlap and can assign misleadingly high scores to captions that describe context correctly but identify the wrong object.
• Exact-match accuracy is used for SQA3D and object counting, but semantic similarity metrics (Sentence-BERT, METEOR, ROUGE) are preferred for scene-centric tasks to capture paraphrasing and partial matches without penalizing valid alternative phrasings.

Evidence (verbatim from paper)

For scene-centric tasks, where captions and answers typically encompass diverse and richly descriptive content, we report standard metrics including CIDEr, BLEU-4, METEOR, ROUGE, exact-match accuracy, and Sentence-BERT similarity. For object-centric tasks, we exclude BLEU-4 and CIDEr. BLEU, a precision-based metric, and CIDEr are overly sensitive to superficial n-gram overlap, rendering them unsuitable for evaluating long-form object captions.

Citation

@misc{halacheva2025gaussianvlm,
  title={GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond},
  author={Halacheva et al. (2025)},
  year={2025},
  note={arXiv:2507.00886}
}

• arXiv: 2507.00886