Xr Scene Eval

xr-scene-eval

LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences — Zhi et al. (2024) (arXiv:2412.01292, 2024)

What this evaluates

Probes large-scale 3D scene understanding by evaluating a model's ability to reason across multiple rooms, locate specific objects, generate embodied task plans, and produce detailed captions in complex, high-density environments. It specifically tests spatial awareness, contextual inference, and fine-grained detail retention beyond single-room benchmarks.

Datasets

• XR-Scene — total 1000; splits: test (-1); repo https://github.com/Hoyyyaard/LSceneLLM

Metrics

• ROUGE — range: [0, 1]
  • n-gram overlap between generated and reference text, typically computed as ROUGE-L or ROUGE-1/2/3. Measures lexical similarity.
• METEOR — range: [0, 1]
  • Alignment-based metric that considers synonyms, stemming, and word order to measure semantic similarity between generated and reference text.
• CIDEr (primary) — range: [0, 1]
  • Consensus-based Image Description Evaluation; computes TF-IDF weighted n-gram similarity between generated and multiple reference captions to penalize common words and reward consensus.

Input / output format

Input: 3D point cloud data of multi-room scenes, top-down spatial views, and room-level annotations. Inputs are paired with task-specific prompts: questions for XR-QA, high-level goals for XR-EmbodiedPlanning, and room identifiers for XR-SceneCaption.

Output: Natural language text: direct answers to spatial queries, step-by-step subtask plans for navigation/planning, or descriptive captions for specific rooms.

Scoring recipe

def compute_metrics(predictions, references):
    # predictions and references are lists of strings
    rouge_scores = rouge_score(predictions, references, use_stemmer=True)
    meteor_scores = [meteor_score([ref], pred) for pred, ref in zip(predictions, references)]
    cider_scores = compute_cider(predictions, references)
    return {
        "ROUGE": np.mean(rouge_scores),
        "METEOR": np.mean(meteor_scores),
        "CIDEr": np.mean(cider_scores)
    }

Common pitfalls

• Down-sampling large point clouds to fit model context windows often discards fine-grained details, causing severe performance drops on small objects (addressed by the XR-QA-S subset with a 0.05 m² bounding box threshold).
• Single-room benchmarks heavily bias models toward common furniture (chairs/tables); cross-room evaluation requires balanced object diversity and explicit spatial reasoning across room boundaries.
• Models may rely on shortcut heuristics or room-level priors rather than true cross-room attention, failing to localize objects accurately in dense environments.

Evidence (verbatim from paper)

XR-SceneCaption challenges 3D-VLM to generate comprehensive scene descriptions and captions for specific rooms while inferring attributes based on present objects. Table 5: Ablation studies. ATR: the activate token ratio of sparse vision tokens. #: do not use the scene magnifier module.

| Parameter | | ROUGE | METEOR | CIDEr |

Citation

@misc{zhi2024lscenellm,
  title={LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences},
  author={Zhi et al. (2024)},
  year={2024},
  note={arXiv:2412.01292}
}

• arXiv: 2412.01292