qhjqhj00 meta-world-eval
D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI — Suhwan Choi et al. (2025) (arXiv:2510.05684, 2025)
What this evaluates
Assesses multi-task robotic manipulation capabilities across varying difficulty levels (Easy, Medium, Hard, Very Hard) in simulation to evaluate robustness and generalization.
Datasets
- Meta-World — total ?; splits: test (-1)
Metrics
success rate (%)(primary) — range: percent- Percentage of successful rollouts out of total attempts per task category. Calculated as (number of successful episodes / total episodes) * 100.
Input / output format
Input: Visual observations and task instructions.
Output: Robot action commands.
Scoring recipe
success_count = 0
for rollout in rollouts:
if rollout.success:
success_count += 1
success_rate = (success_count / len(rollouts)) * 100
Common pitfalls
- Performance drops sharply on Hard/Very Hard tasks, so averaging across all difficulties can overstate general capability.
- Simulated physics parameters and task initializations can cause high variance in success rates.
Evidence (verbatim from paper)
Next, we evaluate our VAPT models on Meta-World, a standard benchmark for multi-task robotic manipulation... Even without robotics-specific pretraining or extensive hyperparameter tuning, VAPT consistently outperforms the baseline, showing an average success rate improvement of roughly 5% (a ~25% relative gain).
Citation
@misc{choi2025d2e,
title={D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI},
author={Suhwan Choi et al. (2025)},
year={2025},
note={arXiv:2510.05684}
}
- arXiv: 2510.05684