Simplerenv Eval

simplerenv_eval

InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation — Yang et al. (2025) (arXiv:2507.17520, 2025)

What this evaluates

Evaluates a robot's ability to execute manipulation tasks from visual inputs and textual instructions, testing both atomic skill execution and high-level instruction generalization in simulated and real-world settings.

Datasets

• SimplerEnv — total ?; splits: test (-1)
• SimplerEnv-Instruct — total ?; splits: test (-1)

Metrics

• visual matching (VM) (primary) — range: percent
  • Success rate measured by visual matching of the final state to the goal.
• variance aggregation (VA) — range: percent
  • Success rate measured by variance aggregation across multiple trials or states.

Input / output format

Input: Robot camera images, robot state observations, and textual instructions.

Output: Action tokens (low-level control commands) and optionally textual reasoning.

Scoring recipe

predictions, gold = load_data(simplerenv)
success = sum(1 for p in predictions if p == gold) / len(predictions) * 100
return success

Common pitfalls

• VM and VA are two distinct evaluation settings; results are reported separately and should not be averaged without clarification.
• SimplerEnv-Instruct evaluates high-level instruction generalization, which requires reasoning over novel objects/tasks not seen during training.

Evidence (verbatim from paper)

SimplerEnv [37] features real-to-sim evaluation on large-scale manipulation datasets [34, 33] with visual matching and variance aggregation settings to evaluate generalization ability.

Citation

@misc{yang2025instructvla,
  title={InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2507.17520}
}

• arXiv: 2507.17520