Dexcanvas Success Eval

dexcanvas-success-eval

DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation — Xu et al. (2025) (arXiv:2510.15786, 2025)

What this evaluates

This evaluation probes a robot policy's ability to successfully reproduce human-demonstrated dexterous manipulation trajectories in physics simulation. It measures robustness by testing performance under nominal conditions and under controlled initial pose perturbations.

Datasets

• DexCanvas — total 7000; splits: test (32); repo https://github.com/dexrobot/dexcanvas

Metrics

• success rate (primary) — range: percent
  • Calculated as the number of successful rollouts divided by the total number of rollouts (100 per policy), expressed as a percentage. A rollout is successful if the policy completes the demonstration trajectory without triggering termination.

Input / output format

Input: Object-manipulation pair configuration, initial object pose (nominal or perturbed by up to 20% of object size in length/width), and trained policy parameters.

Output: Boolean success/failure flag per rollout, determined by whether the positional error exceeds 5cm during execution.

Scoring recipe

success_count = 0
total_rollouts = 0
for policy in policies:
    for _ in range(100):
        trajectory = run_simulation(policy, initial_pose)
        total_rollouts += 1
        if max_positional_error(trajectory) <= 5.0:
            success_count += 1
success_rate = (success_count / total_rollouts) * 100

Common pitfalls

• Termination is triggered specifically when positional error exceeds 5cm, not by time limits or other constraints.
• Perturbations are defined as a 20% shift in object size along length and width axes, not uniform random noise.
• Success rate is averaged across 32 representative object-manipulation pairs, not reported per-pair in the headline metric.

Evidence (verbatim from paper)

Each policy was rolled out 100 times in simulation, with success defined as completing the demonstration trajectory without termination (triggered when positional error exceeded 5cm). Policies achieved an 80.15% success rate under nominal conditions, demonstrating effective reproduction of human demonstrations in physics simulation. When initial object poses were perturbed by up to 20% of object size, the success rate decreased moderately to 62.54%—a drop of only 17.61 percentage points.

Citation

@misc{xu2025dexcanvas,
  title={DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2510.15786}
}

• arXiv: 2510.15786