success-rate-3d-policy
R3D: Revisiting 3D Policy Learning — Hong et al. (2026) (arXiv:2604.15281, 2026)
What this evaluates
Evaluates the robustness and generalization of 3D policy learning models for robotic manipulation across varying environmental conditions, temporal horizons, and real-world interference. It probes spatial understanding, fine-grained pose control, and resilience to domain randomization and lighting changes.
Datasets
- RoboTwin 2.0 — total ?; splits: Easy (-1), Hard (-1)
- ManiSkill2 — total ?; splits: PegInsertionSide (-1), PickCube (-1), StackCube (-1)
- Real-World Manipulation — total ?; splits: Place Kettle (-1), Open Drawer (-1), Fold Towel (-1)
Metrics
Success Rate (%) (primary) — range: percent
- Calculated as the percentage of successful trials out of the total number of trials per task. The paper specifies evaluating each data point on 50 trials for both simulation and real-world settings.
Input / output format
Input: Cropped 3D point clouds (1024 points for simulation, 8192 for real-world) representing the environment, robot state, and goal positions. Real-world inputs may use single or dual camera views.
Output: Continuous action commands for robotic manipulation, decoded via a diffusion-based action decoder.
Scoring recipe
def compute_success_rate(trial_outcomes, total_trials=50):
successes = sum(1 for outcome in trial_outcomes if outcome)
return (successes / total_trials) * 100
Common pitfalls
- Point cloud downsampling resolution differs between simulation (1024 points) and real-world (8192 points), which can artificially inflate or deflate performance if not standardized during evaluation.
- The PegInsertion task is evaluated across three sequential subtasks (Grasp, Align, Insert) rather than a single end-to-end metric, requiring separate tracking of each stage's success rate.
- Real-world evaluation includes a dynamic 'disco light' interference condition that specifically tests color robustness, which many baselines fail due to lighting-dependent point cloud coloring.
Evidence (verbatim from paper)
We report the success rate as the evaluation metric for these two benchmarks. Each data point is evaluated on 50 trials.
Citation
@misc{hong2026r3d,
title={R3D: Revisiting 3D Policy Learning},
author={Hong et al. (2026)},
year={2026},
note={arXiv:2604.15281}
}
1---2name: success-rate-3d-policy3description: Evaluates the robustness and generalization of 3D policy learning models for robotic manipulation across varying environmental conditions, temporal horizons, and real-world interference. It probes spatial understanding, fine-grained pose control, and resilience to domain randomization and lighting changes. Use when the user wants to benchmark on RoboTwin 2.0, ManiSkill2, Real-World Manipulation, or asks about evaluating this task. Reports Success Rate (%).4---56# success-rate-3d-policy78> R3D: Revisiting 3D Policy Learning — Hong et al. (2026) (arXiv:2604.15281, 2026)910## What this evaluates1112Evaluates the robustness and generalization of 3D policy learning models for robotic manipulation across varying environmental conditions, temporal horizons, and real-world interference. It probes spatial understanding, fine-grained pose control, and resilience to domain randomization and lighting changes.1314## Datasets1516- **RoboTwin 2.0** — total ?; splits: Easy (-1), Hard (-1)17- **ManiSkill2** — total ?; splits: PegInsertionSide (-1), PickCube (-1), StackCube (-1)18- **Real-World Manipulation** — total ?; splits: Place Kettle (-1), Open Drawer (-1), Fold Towel (-1)1920## Metrics2122- `Success Rate (%)` **(primary)** — range: percent23 - Calculated as the percentage of successful trials out of the total number of trials per task. The paper specifies evaluating each data point on 50 trials for both simulation and real-world settings.2425## Input / output format2627**Input**: Cropped 3D point clouds (1024 points for simulation, 8192 for real-world) representing the environment, robot state, and goal positions. Real-world inputs may use single or dual camera views.2829**Output**: Continuous action commands for robotic manipulation, decoded via a diffusion-based action decoder.3031## Scoring recipe3233```python34def compute_success_rate(trial_outcomes, total_trials=50):35 successes = sum(1 for outcome in trial_outcomes if outcome)36 return (successes / total_trials) * 10037```3839## Common pitfalls4041- Point cloud downsampling resolution differs between simulation (1024 points) and real-world (8192 points), which can artificially inflate or deflate performance if not standardized during evaluation.42- The PegInsertion task is evaluated across three sequential subtasks (Grasp, Align, Insert) rather than a single end-to-end metric, requiring separate tracking of each stage's success rate.43- Real-world evaluation includes a dynamic 'disco light' interference condition that specifically tests color robustness, which many baselines fail due to lighting-dependent point cloud coloring.4445## Evidence (verbatim from paper)4647> We report the success rate as the evaluation metric for these two benchmarks. Each data point is evaluated on 50 trials.4849## Citation5051```bibtex52@misc{hong2026r3d,53 title={R3D: Revisiting 3D Policy Learning},54 author={Hong et al. (2026)},55 year={2026},56 note={arXiv:2604.15281}57}58```5960- arXiv: 2604.15281
qhjqhj00