qhjqhj00 robot-manipulation-eval
EmbodiedMAE: A Unified 3D Multi-Modal Representation for Robot Manipulation — Dong et al. (2025) (arXiv:2505.10105, 2025)
What this evaluates
Evaluates how well vision foundation models support robot manipulation policies in simulation and real-world environments. It probes cross-modal spatial reasoning, task generalization across diverse manipulation suites, and robustness to sensor noise and platform differences.
Datasets
- LIBERO — total 40; splits: test (40)
- MetaWorld — total 30; splits: test (30)
Metrics
success rate(primary) — range: percent- Calculated as the number of successful task completions divided by the total number of evaluation trials, multiplied by 100. Reported as an average across tasks or difficulty levels.
Input / output format
Input: RGB and/or depth images (RGBD), point clouds, and robot proprioceptive state observations fed into a compact RDT policy network.
Output: Continuous action vectors for robot joint control.
Scoring recipe
def compute_success_rate(predictions, gold, trials_per_task=150):
total_trials = len(gold) * trials_per_task
successes = sum(1 for task in gold for i in range(trials_per_task) if predictions[task][i] == gold[task])
return (successes / total_trials) * 100
Common pitfalls
- Point cloud inputs often underperform RGB-only inputs in practice due to sensor noise from object reflectivity and lighting variations, contrary to prior assumptions about their compactness.
- Naively adding a trainable depth branch to RGB-only vision models (e.g., DINOv2) can degrade performance compared to using RGB-only inputs.
Evidence (verbatim from paper)
We report learning curves on LIBERO in [Figure 6] and success rate on MetaWorld in [Table 1]. Unless otherwise specified, “EmbodiedMAE” refers to the Large-scale, RGB-only variant. Each task is evaluated across 150 trials.
Citation
@misc{dong2025embodiedmae,
title={EmbodiedMAE: A Unified 3D Multi-Modal Representation for Robot Manipulation},
author={Dong et al. (2025)},
year={2025},
note={arXiv:2505.10105}
}
- arXiv: 2505.10105