Droid Robot Manipulation Eval

droid-robot-manipulation-eval

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — Khazatsky et al. (2024) (arXiv:2403.12945, 2024)

What this evaluates

Evaluates the robustness and generalization of robot manipulation policies when co-trained with diverse, in-the-wild data. It probes the model's ability to handle distractors, novel objects, and scene variations across short, medium, and long-horizon tasks.

Datasets

• DROID Evaluation Tasks — total ?; splits: test (6)

Metrics

• success rate (primary) — range: [0, 1]
  • Fraction of successful rollouts out of 10 attempts per task and setting, averaged across all 6 tasks. Evaluated separately for in-distribution (ID) and out-of-distribution (OOD) settings.

Input / output format

Input: Two RGB camera streams (128x128), robot proprioceptive state, and a language instruction.

Output: 16-step sequence of absolute end-effector translation, rotation, and gripper actions.

Scoring recipe

def compute_success_rate(task_rollouts):
    return sum(1 for r in task_rollouts if r.success) / len(task_rollouts)

task_rates = [compute_success_rate(rollouts) for rollouts in all_task_rollouts]
overall_success_rate = sum(task_rates) / len(task_rates)

Common pitfalls

• Confusing in-distribution (ID) with out-of-distribution (OOD) settings; OOD specifically introduces distractors, novel objects, or camera shifts.
• Forgetting that the 50/50 co-training mix explicitly excludes trajectories marked as 'not successful' from DROID.
• Assuming the metric is per-episode accuracy; it is averaged across 10 rollouts per task and then across all 6 tasks.

Evidence (verbatim from paper)

To test how DROID and existing datasets affect policy robustness, we evaluate each task and method in two settings: “in-distribution,” which reflects the distribution of tasks in the in-domain demonstrations with noise added to the initial robot and object positions, and “out-of-distribution” (OOD), which tests policy robustness e.g., by introducing distractor objects or switching the manipulated object. We compare all policies head-to-head in A/B evaluations using 10 rollouts for each task setting and method. ... We compare success rate averaged across all tasks with standard error, and find DROID outperforms the next best method by 22% absolute success rate in-distribution and by 17% out of distribution.

Citation

@misc{khazatsky2024droid,
  title={DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset},
  author={Khazatsky et al. (2024)},
  year={2024},
  note={arXiv:2403.12945}
}

• arXiv: 2403.12945