qhjqhj00 vln-ce-eval
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments — Krantz et al. (2020) (arXiv:2004.02857, 2020)
What this evaluates
Evaluates an agent's ability to follow natural language instructions to navigate to a target location in a continuous 3D environment. It probes low-level action control, obstacle avoidance, and spatial reasoning without relying on a pre-defined graph topology or oracle localization.
Datasets
- VLN-CE — total ?; splits: val-seen (-1), val-unseen (-1), test (-1); repo https://github.com/jacobkrantz/VLN-CE
Metrics
TL— range: meters- Total distance traveled by the agent along its trajectory.
NE— range: meters- Euclidean distance between the agent's final position and the goal location at termination.
OS— range: [0, 1]- Fraction of episodes where the agent reaches within 1 meter of the goal at any point during the trajectory.
SR(primary) — range: [0, 1]- Fraction of episodes where the agent reaches within 1 meter of the goal at termination.
SPL(primary) — range: [0, 1]- Success rate weighted by the inverse ratio of the agent's path length to the optimal path length.
nDTW— range: [0, 1]- Normalized dynamic-time warping score measuring the similarity between the agent's trajectory and the ground truth path.
Input / output format
Input: Egocentric RGB-D images, natural language navigation instructions, and history of previous actions/observations.
Output: Discrete low-level actions: forward, turn-left, turn-right, or stop.
Scoring recipe
def compute_spl(predictions, gold_paths):
spl_scores = []
for pred, gold in zip(predictions, gold_paths):
final_pos = pred[-1]
goal_pos = gold[-1]
dist = euclidean_distance(final_pos, goal_pos)
if dist <= 1.0:
optimal_len = len(gold)
actual_len = len(pred)
spl_scores.append(optimal_len / actual_len)
else:
spl_scores.append(0.0)
return sum(spl_scores) / len(predictions) if predictions else 0.0
Common pitfalls
- Directly comparing VLN-CE performance to graph-based VLN benchmarks without accounting for the structural prior provided by the nav-graph.
- Assuming all VLN trajectories are navigable in continuous environments; ~20% are non-navigable and excluded in VLN-CE.
Evidence (verbatim from paper)
We report standard metrics for visual navigation tasks defined in [[4], [2], [18]] – trajectory length in meters (TL), navigation error in meters from goal at termination (NE), oracle success rate (OS), success rate (SR), success weighted by inverse path length (SPL), and normalized dynamic-time warping (nDTW). For our discussion, we will examine success rate and SPL as the primary metrics for performance and use NDTW to describe how paths differ in shape from ground truth trajectories.
Citation
@misc{krantz2020beyond,
title={Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments},
author={Krantz et al. (2020)},
year={2020},
note={arXiv:2004.02857}
}
- arXiv: 2004.02857