qhjqhj00 canvas-eval
D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI — Suhwan Choi et al. (2025) (arXiv:2510.05684, 2025)
What this evaluates
Evaluates robotic navigation policies under diverse simulated environments, testing robustness to both precise and misleading natural language instructions.
Datasets
- CANVAS — total ?; splits: test (-1)
Metrics
success rate (%)(primary) — range: percent- Percentage of successful navigation rollouts out of total attempts. Calculated as (number of successful episodes / total episodes) * 100.
Input / output format
Input: Visual observations and natural language navigation instructions.
Output: Robot navigation commands (e.g., velocity or waypoint).
Scoring recipe
success_count = 0
for rollout in rollouts:
if rollout.success:
success_count += 1
success_rate = (success_count / len(rollouts)) * 100
Common pitfalls
- Success rates are highly sensitive to instruction type (misleading vs. precise), so reporting a single aggregate score hides instruction-following robustness.
- The benchmark uses simulated environments, which may not capture real-world sensor noise or navigation obstacles.
Evidence (verbatim from paper)
Table 9: Results on CANVAS tasks (success rates, %)
Citation
@misc{choi2025d2e,
title={D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI},
author={Suhwan Choi et al. (2025)},
year={2025},
note={arXiv:2510.05684}
}
- arXiv: 2510.05684