Eo Bench Eval

eo-bench-eval

EO-1: Interleaved Vision-Text-Action Pretraining for General Robot Control — Qu et al. (2025) (arXiv:2508.21112, 2025)

What this evaluates

Evaluates a model's ability to reason about embodied interactions, including spatial understanding, physical commonsense, task planning, and state estimation from robot vision and text inputs.

Datasets

• EO-Bench — total 648; splits: test (648)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard question-answering accuracy. Predicted text is compared against the ground-truth answer using exact match or normalized string matching for free-form QA, and exact match for multiple-choice options.

Input / output format

Input: Robot manipulation video frames or image tokens paired with natural language questions covering spatial, physical, and task reasoning.

Output: Natural language text answers (free-form or multiple-choice selections).

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        pred_norm = normalize(pred)
        gold_norm = normalize(gold)
        if pred_norm == gold_norm:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Existing benchmarks often conflate multiple reasoning aspects (e.g., mixing spatial trajectories with extensive commonsense), leading to ambiguous evaluations. EO-Bench explicitly disentangles these to attribute performance accurately.
• The benchmark relies on manually labeled QA pairs from robot control videos, so evaluation quality is tightly coupled to annotation consistency and template design.

Evidence (verbatim from paper)

Embodied Onevision Benchmark ( EO-Bench) aims to construct a comprehensive and balanced evaluation suite for open-world embodied reasoning, covering both challenging and accessible tasks. ... In total, the benchmark comprises 648 QA pairs manually labeled on diverse robot control data, distributed across categories as follows: 370 for spatial understanding, 140 for task reasoning, 84 for physical dynamic reasoning, and 48 for physical commonsense. Together, these four categories provide a structured yet broad evaluation of a model’s ability to reason about space, physics, tasks, and states in embodied intelligence.

Citation

@misc{qu2025eo1,
  title={EO-1: Interleaved Vision-Text-Action Pretraining for General Robot Control},
  author={Qu et al. (2025)},
  year={2025},
  note={arXiv:2508.21112}
}

• arXiv: 2508.21112