# Msnn Eval

> Evaluates a model's ability to predict the immediate next navigation step in a 3D scene given a multi-modal situation description and a textual goal. Use when the user wants to benchmark on MSNN, or asks about evaluating this task. Reports Next-step Action Accuracy.

- Skill: `qhjqhj00/msnn-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/msnn-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/msnn-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/msnn-eval

---


# msnn-eval

> Multi-modal Situated Reasoning in 3D Scenes — Linghu et al. (2024) (arXiv:2409.02389, 2024)

## What this evaluates

Evaluates a model's ability to predict the immediate next navigation step in a 3D scene given a multi-modal situation description and a textual goal.

## Datasets

- **MSNN** — total 34000; splits: test (-1)

## Metrics

- `Next-step Action Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted immediate next-step actions compared to the ground truth action derived from an A* optimal trajectory.

## Input / output format

**Input**: Interleaved multi-modal description of the agent's current situation (location, orientation, text), a textual goal description, and the overall scene.

**Output**: Textual description of the immediate next navigation action.

## Scoring recipe

```python
correct = 0
for situation, goal, gt_action in test_set:
    pred_action = model.predict(situation, goal)
    if pred_action == gt_action:
        correct += 1
accuracy = correct / len(test_set) * 100
```

## Common pitfalls

- Ground truth actions are derived from A* shortest-path planning on floor plans, which may not reflect real-world navigation constraints or agent capabilities.
- Evaluation focuses only on the immediate next step, potentially ignoring long-horizon planning failures.

## Evidence (verbatim from paper)

> we instruct models to answer the immediate next action for navigating to the goal in a textual form. For evaluation, we generate MSNN... The optimal trajectory is sampled by running an A* algorithm planning the shortest path from the starting location to the goal on the floor plan and the immediate next-step action is determined by following the direction of optimal trajectory relative to the starting situation.

## Citation

```bibtex
@misc{linghu2024multimodal,
  title={Multi-modal Situated Reasoning in 3D Scenes},
  author={Linghu et al. (2024)},
  year={2024},
  note={arXiv:2409.02389}
}
```

- arXiv: 2409.02389