# Simplerenv Eval

> Evaluates a robot's ability to execute manipulation tasks from visual inputs and textual instructions, testing both atomic skill execution and high-level instruction generalization in simulated and real-world settings. Use when the user wants to benchmark on SimplerEnv, SimplerEnv-Instruct, or asks about evaluating this task. Reports visual matching (VM).

- Skill: `qhjqhj00/simplerenv-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/simplerenv-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/simplerenv-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/simplerenv-eval

---


# simplerenv_eval

> InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation — Yang et al. (2025) (arXiv:2507.17520, 2025)

## What this evaluates

Evaluates a robot's ability to execute manipulation tasks from visual inputs and textual instructions, testing both atomic skill execution and high-level instruction generalization in simulated and real-world settings.

## Datasets

- **SimplerEnv** — total ?; splits: test (-1)
- **SimplerEnv-Instruct** — total ?; splits: test (-1)

## Metrics

- `visual matching (VM)` **(primary)** — range: percent
  - Success rate measured by visual matching of the final state to the goal.
- `variance aggregation (VA)` — range: percent
  - Success rate measured by variance aggregation across multiple trials or states.

## Input / output format

**Input**: Robot camera images, robot state observations, and textual instructions.

**Output**: Action tokens (low-level control commands) and optionally textual reasoning.

## Scoring recipe

```python
predictions, gold = load_data(simplerenv)
success = sum(1 for p in predictions if p == gold) / len(predictions) * 100
return success
```

## Common pitfalls

- VM and VA are two distinct evaluation settings; results are reported separately and should not be averaged without clarification.
- SimplerEnv-Instruct evaluates high-level instruction generalization, which requires reasoning over novel objects/tasks not seen during training.

## Evidence (verbatim from paper)

> SimplerEnv [37] features real-to-sim evaluation on large-scale manipulation datasets [34, 33] with visual matching and variance aggregation settings to evaluate generalization ability.

## Citation

```bibtex
@misc{yang2025instructvla,
  title={InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2507.17520}
}
```

- arXiv: 2507.17520