Wanderland Eval

wanderland-eval

Wanderland: Geometrically Grounded Simulation for Open-World Embodied AI — Liu et al. (2025) (arXiv:2511.20620, 2025)

What this evaluates

Evaluates the fidelity of geometrically grounded 3D reconstruction and novel view synthesis for open-world urban environments, and benchmarks the performance of embodied navigation policies trained and evaluated in these simulated environments.

Datasets

• Wanderland — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Navigation Error (NE) (primary) — range: meters
  • Euclidean distance in meters between the agent's final predicted position and the target goal location.
• Success Rate (SR) — range: [0, 1]
  • Binary indicator (1 if final distance to goal < 0.2m, else 0) averaged over episodes.
• Success Path Length (SPL) — range: [0, 1]
  • SR multiplied by the ratio of the shortest path length to the actual path length taken by the agent.
• Intervention Rate (IR) — range: [0, 1]
  • Fraction of episodes where the agent fails to reach the goal without timing out, capturing non-timeout failures.

Input / output format

Input: RGB images, camera poses, and 3D scene geometry (point clouds/meshes) for reconstruction; RGB observations and navigation goals (text/point/image) for embodied navigation tasks.

Output: Predicted camera poses and reconstructed meshes for reconstruction tasks; predicted navigation actions/trajectories for embodied navigation tasks.

Scoring recipe

def score_navigation(pred_traj, gold_traj, max_steps=100, threshold=0.2):
    ne = np.linalg.norm(pred_traj[-1] - gold_traj[-1])
    sr = 1.0 if ne < threshold else 0.0
    spl = sr * (len(gold_traj) / max(len(gold_traj), len(pred_traj)))
    ir = 1.0 if (ne >= threshold and len(pred_traj) < max_steps) else 0.0
    return {'NE': ne, 'SR': sr, 'SPL': spl, 'IR': ir}

Common pitfalls

• Evaluating navigation policies in geometrically inaccurate simulations yields artificially inflated success rates but fails to generalize to real-world or metrically grounded environments.
• Outdoor navigation tasks are significantly harder than indoor ones due to longer trajectories, complex topology, and elevation changes, requiring careful split balancing.
• Intervention Rate (IR) specifically captures non-timeout failures, which standard SR misses; ignoring IR can mask policy brittleness in open-world settings.

Evidence (verbatim from paper)

We use standard navigation error (NE), success rate (SR), and success path length (SPL) to evaluate embodied navigation performance. We additionally defined the intervention rate (IR) to evaluate none-timeout failure cases.

Citation

@misc{liu2025wanderland,
  title={Wanderland: Geometrically Grounded Simulation for Open-World Embodied AI},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2511.20620}
}

• arXiv: 2511.20620