qhjqhj00 bsc-nav-eval
From reactive to cognitive: brain-inspired spatial intelligence for embodied agents — Ruan et al. (2025) (arXiv:2508.17198, 2025)
What this evaluates
Evaluates embodied agents' spatial cognition and navigation capabilities across category-level, instance-level, and long-horizon instruction-following tasks, as well as active embodied question answering and real-world mobile manipulation.
Datasets
- MP3D & HM3D (Habitat Simulator) — total 8195; splits: seen (-1), unseen (-1)
- VLN-CE R2R — total 1000; splits: test (-1)
- OpenEQA (A-EQA subset) — total 184; splits: test (-1)
- Real-world Indoor Environment — total 75; splits: test (-1)
Metrics
Success Rate (SR)(primary) — range: [0, 1]- Binary metric: 1 if the agent executes the 'stop' action within 1.0 m of the goal location, 0 otherwise. Averaged over all episodes.
Success weighted by Path Length (SPL)— range: [0, 1]- Standard SPL metric: SR multiplied by the ratio of the shortest possible path length to the actual path length taken by the agent. Averages efficiency and success.
LLM-Match— range: other- Semantic similarity score between the agent's generated answer and the reference answer, computed using an LLM-based evaluator.
Input / output format
Input: Egocentric visual observations, text or image navigation instructions, and spatial context from structured memory modules.
Output: Discrete navigation actions (forward 25 cm, turn left 30°, turn right 30°, stop) or natural language answers for QA tasks.
Scoring recipe
def compute_sr(predictions, gold):
success = 0
for pred, goal in zip(predictions, gold):
dist = calculate_distance(pred.stop_pos, goal.pos)
if dist <= 1.0:
success += 1
return success / len(predictions)
def compute_spl(predictions, gold):
sr = compute_sr(predictions, gold)
path_ratios = []
for pred, goal in zip(predictions, gold):
shortest = goal.shortest_path_len
actual = pred.actual_path_len
if shortest > 0:
path_ratios.append(actual / shortest)
else:
path_ratios.append(0)
return sr * (sum(path_ratios) / len(path_ratios))
Common pitfalls
- Success is strictly defined as stopping within 1.0 m of the goal, not just reaching the vicinity.
- LIN/VLN-CE evaluation is zero-shot; do not assume instruction-level supervised training.
- LLM-Match requires careful prompt design for semantic similarity, as it replaces exact string matching.
Evidence (verbatim from paper)
Following standard protocols, navigation succeeds only when agents execute stop within 1.0 m of goals. Evaluation metrics include the Success Rate (SR) for efficacy and the Success weighted by Path Length (SPL) for efficiency (detailed in Methods).
Citation
@misc{ruan2025bscnav,
title={From reactive to cognitive: brain-inspired spatial intelligence for embodied agents},
author={Ruan et al. (2025)},
year={2025},
note={arXiv:2508.17198}
}
- arXiv: 2508.17198