qhjqhj00 robotic-manipulation-eval
The Ingredients for Robotic Diffusion Transformers — Dasari et al. (2024) (arXiv:2410.10088, 2024)
What this evaluates
Evaluates the ability of diffusion transformer policies to perform long-horizon robotic manipulation tasks across bi-manual, single-arm, and simulated environments. It probes stable training, observation tokenization, and generalization across different robot morphologies and action spaces.
Datasets
- Robotic Manipulation Task Suite — total ?; splits: test (-1)
Metrics
success_rate(primary) — range: percent- Percentage of episodes where the robot successfully completes the specified manipulation task within the allotted time/steps.
Input / output format
Input: Multi-view camera images, goal conditioning vectors, and language instructions.
Output: Continuous robot action vectors (e.g., joint velocities or end-effector poses).
Scoring recipe
def compute_success_rate(predictions, gold, task_config):
success_count = 0
for pred, gold_step in zip(predictions, gold):
if check_task_completion(pred, task_config):
success_count += 1
return (success_count / len(gold)) * 100
Common pitfalls
- Baselines like D.P. Transformer often fail to train stably without extensive hyperparameter tuning, leading to unsafe/noisy action predictions.
- Sim-to-real evaluation gap means high simulation success rates do not guarantee real-world performance.
Evidence (verbatim from paper)
We evaluate DiT-Block Policy on a set of 3 Bi-Manual and 2 Single-Arm manipulation tasks... We find that DiT-Block Policy is able to outperform the strongest by roughly 20% when trained with BiPlay, and by 10% when trained on task data alone.
Citation
@misc{dasari2024ingredients,
title={The Ingredients for Robotic Diffusion Transformers},
author={Dasari et al. (2024)},
year={2024},
note={arXiv:2410.10088}
}
- arXiv: 2410.10088