Gemini Robotics 15 Eval

gemini-robotics-15-eval

Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer — Gemini Robotics Team et al. (2025) (arXiv:2510.03342, 2025)

What this evaluates

Evaluates a robot's ability to execute short-horizon and multi-step manipulation tasks across diverse embodiments, environments, and visual/instructional variations. It specifically probes zero-shot cross-embodiment skill transfer and the impact of explicit 'thinking' traces on task progress and success.

Datasets

• Gemini Robotics 1.5 Benchmark — total 230; splits: test (230)

Metrics

• progress score (primary) — range: [0, 1]
  • A continuous, finer-grained measure of how far a robot has progressed toward completing a task. Exact calculation details are provided in the paper's appendices.
• success rate — range: percent
  • Binary metric indicating whether a task was fully completed successfully.

Input / output format

Input: Open-vocabulary natural language instructions, visual observations from robot cameras, and environmental feedback.

Output: Low-level robot actions (joint commands or primitive motions), optionally preceded by natural language 'thinking' traces.

Scoring recipe

def evaluate_episode(model_output, gold_trajectory):
    progress = compute_progress_score(model_output, gold_trajectory)
    success = 1.0 if is_task_completed(model_output) else 0.0
    return {'progress_score': progress, 'success_rate': success}

Common pitfalls

• Simulation evaluations show strong rank consistency with real robots but do not replace real-world validation for absolute quality.
• Progress score provides finer-grained feedback for multi-step tasks compared to binary success rate, so relying solely on success rate may mask partial improvements.
• Cross-embodiment transfer is tested zero-shot; models trained on single embodiments will fail this specific benchmark subset.

Evidence (verbatim from paper)

We generally report mean and standard error of the mean of progress score (definitions in Appendix B.2 - Appendix B.4), as it provides a continuous and finer-grained measure of model performance and, as such, is especially useful for complex multi-step tasks. For completeness, we also include the corresponding plots of success rate in the Appendix B.5.

Citation

@misc{geminiroboticsteam2025geminirobotics15,
  title={Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer},
  author={Gemini Robotics Team et al. (2025)},
  year={2025},
  note={arXiv:2510.03342}
}

• arXiv: 2510.03342