Vima Bench Eval

vima-bench-eval

Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks — Parekh et al. (2024) (arXiv:2407.03967, 2024)

What this evaluates

This evaluation probes a model's ability to generalise to novel robotic manipulation tasks by testing robustness to instruction variations and increased task difficulty. It specifically measures compositional generalisation capabilities across four systematicity levels, ranging from object pose sensitivity to entirely novel objects and tasks.

Datasets

• VIMABench — total ?; splits: test (-1); repo https://github.com/amitkparekh/CoGeLoT

Metrics

• compositional generalisation capabilities (primary) — range: [0, 1]
  • Not explicitly defined in the provided text; typically measured as the proportion of successfully completed task episodes, aggregated across four systematicity levels (L1-L4).

Input / output format

Input: Multimodal prompts with interleaved visual and linguistic features. Visual features are implicitly encoded by embedding image frames per observation, while linguistic features are processed via a frozen pretrained T5 language model.

Output: An action defining a linear movement between two end-effector poses in SE(3), representing 3D position and rotation.

Scoring recipe

success_count = 0
total_episodes = 0
for task in VIMABench.tasks:
    obs = get_observation(task)
    action = model.predict(obs)  # SE(3) pose movement
    if task_environment.check_success(action):
        success_count += 1
    total_episodes += 1
return success_count / total_episodes

Common pitfalls

• The original VIMA-Bench benchmark was not reproducible (Jiang et al. 2023 did not release it), so evaluations must use the authors' remediated version.
• Results must be reported separately for four distinct systematicity levels (L1-L4) to properly assess compositional generalisation rather than as a single aggregate score.

Evidence (verbatim from paper)

We use VIMABench to compare model performance across various skills, tasks, and levels of systematicity, as it is best suited for evaluating the role instructions play in generalising from multimodal prompts. Specifically, we assess the compositional generalisation capabilities at four distinct levels of systematicity (L1, L2, L3, L4).

Citation

@misc{parekh2024investigating,
  title={Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks},
  author={Parekh et al. (2024)},
  year={2024},
  note={arXiv:2407.03967}
}

• arXiv: 2407.03967