Rxr Eval

rxr-eval

Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding — Ku et al. (2020) (arXiv:2010.07954, 2020)

What this evaluates

Evaluates the ability of embodied agents to follow natural language instructions for navigation in photo-realistic 3D environments. It probes multilingual understanding, spatial reasoning, and dense spatiotemporal grounding by measuring how accurately an agent navigates from a start to a target location.

Datasets

• Room-Across-Room (RxR) — total 126000; splits: val-seen (-1), val-unseen (-1), test-standard (-1), test-challenge (-1); repo https://github.com/google-research-datasets/RxR

Metrics

• NE — range: meters
  • Navigation Error: Euclidean distance (in meters) between the agent's final position and the target location. Lower is better.
• SR (primary) — range: [0, 1]
  • Success Rate: Fraction of trajectories that end within 3 meters of the target location. Higher is better.
• NDTW (primary) — range: [0, 1]
  • Normalized Dynamic Time Warping: Measures the similarity between the predicted path and the gold path, normalized by the gold path length. Higher is better.
• SDTW — range: [0, 1]
  • Soft Dynamic Time Warping: A differentiable approximation of DTW measuring path similarity. Higher is better.
• SPL — range: [0, 1]
  • Success weighted by Path Length: SR multiplied by the ratio of the gold path length to the predicted path length. Higher is better.

Input / output format

Input: A sequence of panoramic images (36 views per step) with navigable direction encodings, paired with a natural language instruction in English, Hindi, or Telugu.

Output: A sequence of discrete navigation actions (e.g., move forward, turn left, turn right) until the agent stops at a target location or reaches a maximum step limit.

Scoring recipe

def compute_metrics(predicted_path, gold_path, target_pos):
    ne = euclidean_distance(predicted_path[-1], target_pos)
    sr = 1.0 if ne <= 3.0 else 0.0
    ndtw = normalized_dtw(predicted_path, gold_path)
    sdtw = soft_dtw(predicted_path, gold_path)
    spl = sr * (len(gold_path) / max(len(predicted_path), 1))
    return {'NE': ne, 'SR': sr, 'NDTW': ndtw, 'SDTW': sdtw, 'SPL': spl}

Common pitfalls

• Val-unseen and test-standard splits are used for reporting, but test-challenge remains sequestered for a public leaderboard.
• Multilingual training on all three languages simultaneously degrades performance on high-resource English compared to monolingual training.
• Guide paths are algorithmically generated, while Follower paths are human demonstrations; mixing them changes the training distribution and evaluation dynamics.

Evidence (verbatim from paper)

Table 5 provides results on the val-unseen split for several training settings, as well as human performance from Follower annotations. ... NE ↓, SR ↑, SDTW ↑, NDTW ↑.

Citation

@misc{ku2020roomacrossroom,
  title={Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding},
  author={Ku et al. (2020)},
  year={2020},
  note={arXiv:2010.07954}
}

• arXiv: 2010.07954