Real Robot Challenge 2022 Eval

real-robot-challenge-2022-eval

Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World — Gurtler et al. (2023) (arXiv:2308.07741, 2023)

What this evaluates

Evaluates offline reinforcement learning and imitation learning algorithms on real-world dexterous manipulation tasks. It probes the ability to learn precise in-hand orientation and stable grasping from pre-collected robot data without online interaction, and measures transfer performance to physical hardware.

Datasets

• Real Robot Challenge 2022 TriFinger Datasets — total ?; splits: Expert (-1), Mixed (-1)

Metrics

• overall score (primary) — range: other
  • Average of the mean returns across four task/dataset combinations: Push/Expert, Push/Mixed, Lift/Expert, and Lift/Mixed. Failed execution runs are assigned a return of 0.
• average return — range: other
  • Mean reward accumulated per task/dataset combination over multiple evaluation runs. Standard error of the mean is also reported.

Input / output format

Input: Pre-collected TriFinger robot datasets containing observations (e.g., images, proprioception) and actions for Push and Lift tasks.

Output: A learned behavior policy or offline RL algorithm executed on real hardware via remote access to generate actions for the robot.

Scoring recipe

def compute_overall_score(task_returns):
    # task_returns: dict mapping task/dataset combo to mean return
    # failed runs are already counted as 0 in the mean return
    overall = sum(task_returns.values()) / len(task_returns)
    return overall

Common pitfalls

• Hyperparameters for offline RL algorithms are often tuned on simulated versions of the tasks, which can inflate real-world performance compared to purely real-world tuned methods.
• Failed physical execution runs are treated as a return of 0, heavily penalizing unstable policies.
• Imitation learning (Behavioral Cloning) sometimes outperforms complex offline RL algorithms due to easier tuning and robustness to noisy real-world observations.

Evidence (verbatim from paper)

Table 1 shows the average returns the teams achieved on all task/dataset combinations in the real-robot phase. The last column contains the overall score which is obtained by averaging these returns. (failed runs correspond to a return of 0).

Citation

@misc{gurtler2023realrobotchallenge,
  title={Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World},
  author={Gurtler et al. (2023)},
  year={2023},
  note={arXiv:2308.07741}
}

• arXiv: 2308.07741