# Meta World Eval

> Assesses multi-task robotic manipulation capabilities across varying difficulty levels (Easy, Medium, Hard, Very Hard) in simulation to evaluate robustness and generalization. Use when the user wants to benchmark on Meta-World, or asks about evaluating this task. Reports success rate (%).

- Skill: `qhjqhj00/meta-world-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/meta-world-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/meta-world-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/meta-world-eval

---


# meta-world-eval

> D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI — Suhwan Choi et al. (2025) (arXiv:2510.05684, 2025)

## What this evaluates

Assesses multi-task robotic manipulation capabilities across varying difficulty levels (Easy, Medium, Hard, Very Hard) in simulation to evaluate robustness and generalization.

## Datasets

- **Meta-World** — total ?; splits: test (-1)

## Metrics

- `success rate (%)` **(primary)** — range: percent
  - Percentage of successful rollouts out of total attempts per task category. Calculated as (number of successful episodes / total episodes) * 100.

## Input / output format

**Input**: Visual observations and task instructions.

**Output**: Robot action commands.

## Scoring recipe

```python
success_count = 0
for rollout in rollouts:
    if rollout.success:
        success_count += 1
success_rate = (success_count / len(rollouts)) * 100
```

## Common pitfalls

- Performance drops sharply on Hard/Very Hard tasks, so averaging across all difficulties can overstate general capability.
- Simulated physics parameters and task initializations can cause high variance in success rates.

## Evidence (verbatim from paper)

> Next, we evaluate our VAPT models on Meta-World, a standard benchmark for multi-task robotic manipulation... Even without robotics-specific pretraining or extensive hyperparameter tuning, VAPT consistently outperforms the baseline, showing an average success rate improvement of roughly 5% (a ~25% relative gain).

## Citation

```bibtex
@misc{choi2025d2e,
  title={D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI},
  author={Suhwan Choi et al. (2025)},
  year={2025},
  note={arXiv:2510.05684}
}
```

- arXiv: 2510.05684