qhjqhj00 msnn-eval
Multi-modal Situated Reasoning in 3D Scenes — Linghu et al. (2024) (arXiv:2409.02389, 2024)
What this evaluates
Evaluates a model's ability to predict the immediate next navigation step in a 3D scene given a multi-modal situation description and a textual goal.
Datasets
- MSNN — total 34000; splits: test (-1)
Metrics
Next-step Action Accuracy(primary) — range: percent- Percentage of correctly predicted immediate next-step actions compared to the ground truth action derived from an A* optimal trajectory.
Input / output format
Input: Interleaved multi-modal description of the agent's current situation (location, orientation, text), a textual goal description, and the overall scene.
Output: Textual description of the immediate next navigation action.
Scoring recipe
correct = 0
for situation, goal, gt_action in test_set:
pred_action = model.predict(situation, goal)
if pred_action == gt_action:
correct += 1
accuracy = correct / len(test_set) * 100
Common pitfalls
- Ground truth actions are derived from A* shortest-path planning on floor plans, which may not reflect real-world navigation constraints or agent capabilities.
- Evaluation focuses only on the immediate next step, potentially ignoring long-horizon planning failures.
Evidence (verbatim from paper)
we instruct models to answer the immediate next action for navigating to the goal in a textual form. For evaluation, we generate MSNN... The optimal trajectory is sampled by running an A* algorithm planning the shortest path from the starting location to the goal on the floor plan and the immediate next-step action is determined by following the direction of optimal trajectory relative to the starting situation.
Citation
@misc{linghu2024multimodal,
title={Multi-modal Situated Reasoning in 3D Scenes},
author={Linghu et al. (2024)},
year={2024},
note={arXiv:2409.02389}
}
- arXiv: 2409.02389