Vlaser Embodied Reasoning Eval

vlaser-embodied-reasoning-eval

Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoning — Yang et al. (2025) (arXiv:2510.11027, 2025)

What this evaluates

This evaluation probes a model's embodied reasoning capabilities, including spatial understanding, visual grounding, task planning, and closed-loop robotic control. It measures how well vision-language models transfer general multimodal knowledge to robot-specific manipulation tasks and identifies the domain gap between internet-scale pretraining and real-world embodiment.

Datasets

• ERQA — total ?; splits: test (-1)
• Ego-Plan2 — total ?; splits: test (-1)
• Where2place — total ?; splits: test (-1)
• Pointarena — total ?; splits: test (-1)
• Paco-Lavis — total ?; splits: test (-1)
• Pixmo-Points — total ?; splits: test (-1)
• VSI-Bench — total ?; splits: test (-1)
• RefSpatial-Bench — total ?; splits: test (-1)
• MMSI-Bench — total ?; splits: test (-1)
• VLABench — total ?; splits: test (-1)
• EmbodiedBench — total ?; splits: test (-1)
• SimplerEnv — total ?; splits: test (-1)
• Robotwin — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions or correctly grounded points across all instances. Computed as (correct predictions / total instances) * 100.
• success rate — range: percent
  • Percentage of successfully completed robotic manipulation episodes. Computed as (successful episodes / total episodes) * 100.
• normalized average accuracy — range: percent
  • Arithmetic mean of the accuracy/success rates across all 12 embodied reasoning benchmarks and simulation environments.

Input / output format

Input: RGB images (single or multi-view) paired with natural language questions, spatial queries, or task instructions for manipulation.

Output: Natural language answers (e.g., object names, coordinates, step-by-step plans) or discrete action tokens for robot control.

Scoring recipe

def compute_score(predictions, golds, task_type):
    if task_type in ['reasoning', 'grounding', 'planning']:
        correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
        return (correct / len(golds)) * 100
    elif task_type == 'robot_control':
        success = sum(1 for p, g in zip(predictions, golds) if p == g)
        return (success / len(golds)) * 100
    return 0.0

Common pitfalls

• Normalizing averages across benchmarks with different scales can obscure performance drops on specific sub-tasks like spatial intelligence.
• High scores on internet-sourced reasoning benchmarks do not guarantee high success rates on closed-loop robot control due to domain shift between pretraining data and real embodiments.
• Model scaling is non-monotonic; smaller models (2B) can outperform larger ones (8B) on simple grounding tasks requiring direct answers.

Evidence (verbatim from paper)

Table 2: SimplerEnv Evaluation on WidowX Robot Tasks. Avg indicates the average success rate among the four tasks. Model sizes are indicated within parentheses. The result of RT-1-X, Octo-Base, OpenVLA, RoboVLM and SpatialVLA are from Qu et al. while the results of π0 is from Zren.

Citation

@misc{yang2025vlaser,
  title={Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoning},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2510.11027}
}

• arXiv: 2510.11027