Goal Oriented Embodied Navigation Eval

goal-oriented-embodied-navigation-eval

How Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspace — Zhao et al. (2026) (arXiv:2604.07973, 2026)

What this evaluates

Evaluates large multimodal models' ability to perform goal-oriented embodied navigation in complex urban 3D airspace. It probes geometric perception, cross-view understanding, spatial imagination, and long-term memory by requiring models to navigate from a start point to a semantic goal using visual observations and historical context.

Datasets

• Embodied Navigation Benchmark — total 5037; splits: short (-1), middle (-1), long (-1); repo https://github.com/serenditipy-AC/Embodied-Navigation-Bench

Metrics

• SR (primary) — range: percent
  • Success Rate: the percentage of navigation trajectories that successfully reach the goal location within the allowed steps or distance threshold.
• SPL — range: [0, 1]
  • Success weighted by Path Length: SR multiplied by the ratio of the optimal path length to the actual path length taken. Measures both success and path efficiency.
• accuracy — range: percent
  • Reported as the percentage of correctly completed navigation tasks across different trajectory length groups (short, middle, long).

Input / output format

Input: Current RGB observation combined with a memory buffer containing previous observations, actions, and rationales. For VLA baselines, inputs are aligned to match the goal-oriented navigation task format.

Output: Discrete spatial action commands expressed in natural language with an accompanying rationale, high-level reasoning/planning steps, or discrete action tokens, depending on the evaluation paradigm.

Scoring recipe

def evaluate_navigation(traj, goal, initial_pos, optimal_len):
    reached = distance(traj[-1], goal) < threshold
    progress = [distance(step, goal) / distance(initial_pos, goal) for step in traj]
    sr = 1.0 if reached else 0.0
    spl = sr * (1.0 if optimal_len == 0 else optimal_len / len(traj))
    return sr, spl

# Aggregate over dataset
SR = mean([sr for sr, _ in results]) * 100
SPL = mean([spl for _, spl in results])

Common pitfalls

• Models are evaluated across three distinct paradigms (action-as-language, action-as-reasoning, action-as-token), so comparing raw scores across paradigms without accounting for output format differences is misleading.
• Navigation completion progress is non-linear; errors often trigger irreversible 'Critical Decision Bifurcations' (CDBs) where distance to goal increases monotonically, making standard linear error accumulation metrics insufficient.
• Trajectory length significantly impacts difficulty; short, middle, and long groups must be evaluated separately as performance gaps widen non-linearly with distance.

Evidence (verbatim from paper)

Both the random and action sample methods exhibit SR and SPL scores close to 0 in middle-distance and long-distance groups. This indicates that the task encompasses a vast action space.

Citation

@misc{zhao2026embodiednavigationbench,
  title={How Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspace},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2604.07973}
}

• arXiv: 2604.07973