Roboverse Imitation Eval

roboverse-imitation-eval

RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning — Geng et al. (2025) (arXiv:2504.18904, 2025)

What this evaluates

Evaluates robot manipulation policies on a unified set of contact-rich pick-and-place and articulation tasks across multiple simulators. It probes both specialist and generalist vision-language-action models on success rates under standard and progressively challenging generalization levels.

Datasets

• ROBOVERSE Imitation Learning Benchmark — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• success rate (primary) — range: percent
  • Binary success per episode averaged across task settings and random seeds. Reported as a percentage.

Input / output format

Input: 256×256×3 RGB images and a short language description of the task.

Output: For specialist models: 9-dim robot joint state. For generalist models: delta end-effector position and binary gripper action {0, +1}, transformed to joint space via cuRobo IK solver.

Scoring recipe

total_success = 0.0
num_evals = len(task_settings) * 3
for setting in task_settings:
    for seed in [seed1, seed2, seed3]:
        success = run_episode(model, setting, seed)
        total_success += success
return (total_success / num_evals) * 100

Common pitfalls

• Results are constrained to single-task fine-tuning due to resource limits, not multi-task or zero-shot generalization.
• OpenVLA baseline only evaluated on 20 uniformly sampled testing scenarios instead of the full set.
• Success metric definition may vary across source simulators (ManiSkill, RLBench, etc.), requiring careful alignment when comparing cross-benchmark results.

Evidence (verbatim from paper)

During evaluations, we randomly select ten task settings from training sets and another ten from the validation sets. The reported success rates are computed as the averages over three random seeds. For each step, the inputs are $256 imes 256 imes 3$ RGB images and a short language description depending on the task settings.

Citation

@misc{geng2025roboverse,
  title={RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning},
  author={Geng et al. (2025)},
  year={2025},
  note={arXiv:2504.18904}
}

• arXiv: 2504.18904