Ego3d Bench Eval

ego3d-bench-eval

Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes — Gholami et al. (2025) (arXiv:2509.06266, 2025)

What this evaluates

Evaluates 3D spatial reasoning and multi-view understanding in Vision-Language Models, specifically testing ego-centric distance estimation, object localization, motion tracking, travel time estimation, and relative location reasoning across multiple camera views.

Datasets

• Ego3D-Bench — total ?; splits: test (-1); repo https://github.com/vbdi/Ego3D-Bench

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated per task category (Ego Dist, Obj Dist, Loc, Ego Mot, Obj Mot, Travel Time, Ego Rel, Obj Rel) and averaged.
• RMSE (primary) — range: other
  • Root Mean Square Error between predicted and ground-truth distances in meters. Calculated per task category (Ego Dist, Obj Dist) and averaged.

Input / output format

Input: Multi-view ego-centric images, camera parameters (or estimated), and a natural language question/query about spatial relationships, distances, or locations.

Output: For QA: a selected option from multiple choices. For distance: a numerical value in meters.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

def compute_rmse(predictions, gold):
    sq_errors = [(p - g) ** 2 for p, g in zip(predictions, gold)]
    return (sum(sq_errors) / len(gold)) ** 0.5

Common pitfalls

• Chance level varies significantly by task type (e.g., ~25% for distance vs ~50% for motion), so baselines must be evaluated against task-specific random selection.
• Human performance degrades substantially on exact distance estimation compared to relative location, highlighting task difficulty rather than model failure alone.
• Models often struggle with localization and travel time even with cognitive map support, indicating these require intricate spatial reasoning.

Evidence (verbatim from paper)

Table 1 presents the results of this analysis. While humans can accurately answer the questions that require reasoning about relative location of the objects in space, their performance degrades in questions that require estimation of the exact distance between objects. This highlights the challenging nature of accurate distance estimation.

Citation

@misc{gholami2025spatial,
  title={Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes},
  author={Gholami et al. (2025)},
  year={2025},
  note={arXiv:2509.06266}
}

• arXiv: 2509.06266