Molmospaces Bench Eval

molmospaces-bench-eval

MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation — Kim et al. (2026) (arXiv:2602.11337, 2026)

What this evaluates

Evaluates zero-shot generalization of vision-language-action and navigation policies across diverse indoor scenes. Probes robustness to environmental perturbations, language prompt variations, and sim-to-real transferability for long-horizon manipulation and semantic navigation tasks.

Datasets

• MolmoSpaces-Bench — total ?; splits: navigate-to (2000), manipulation (-1); repo https://github.com/allenai/molmospaces

Metrics

• success rate (primary) — range: [0, 1]
  • Fraction of episodes where the robot successfully completes the specified task within the allowed action horizon.
• Pearson correlation coefficient — range: [-1, 1]
  • Linear correlation between simulation success rates and real-world success rates across tasks to measure predictive validity.
• Spearman rank correlation coefficient — range: [-1, 1]
  • Rank-based correlation between simulation and real-world success rates to assess monotonic relationship and robustness to outliers.

Input / output format

Input: RGB-D visual observations (from wrist and third-person cameras) paired with natural language instructions (e.g., semantic navigation commands like 'go to [object]' or manipulation prompts).

Output: Robot action commands (joint positions, gripper states) or binary task completion status.

Scoring recipe

def compute_metrics(predictions, gold, sim_rates, real_rates):
    # Success rate
    successes = sum(1 for p, g in zip(predictions, gold) if p == g)
    success_rate = successes / len(gold)
    
    # Correlations (sim vs real)
    pearson_r = pearsonr(sim_rates, real_rates)
    spearman_rho = spearmanr(sim_rates, real_rates)
    
    return success_rate, pearson_r, spearman_rho

Common pitfalls

• Task horizon must be set to 300 for π models and 50 for CAP to avoid premature termination bias and misrepresenting retry behavior.
• Prompt phrasing significantly impacts performance due to training data distribution shifts; using non-DROID frequent verbs degrades results by up to 14%.
• Camera occlusion drastically reduces success rates, especially for wrist-mounted cameras (drops to 2%), which must be accounted for in robustness evaluations.

Evidence (verbatim from paper)

We evaluate the correlation for the pick, open, and close tasks individually. Results for are shown in [Fig. 11]. For the pick task, we observe a strong linear correlation between our MolmoSpaces-Bench results and the results from 752 RoboArena pick tasks, with Pearson and Spearman rank correlation coefficients of 0.96 and 0.98, respectively.

Citation

@misc{kim2026molmospaces,
  title={MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation},
  author={Kim et al. (2026)},
  year={2026},
  note={arXiv:2602.11337}
}

• arXiv: 2602.11337