The Colosseum Eval

the-colosseum-eval

THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation — Pumacay et al. (2024) (arXiv:2402.08191, 2024)

What this evaluates

Evaluates the generalization and robustness of robotic behavior cloning models under various environmental perturbations. It probes how well models trained on clean demonstrations can complete manipulation tasks when faced with changes in lighting, color, distractors, camera pose, and object properties.

Datasets

• The Colosseum — total ?; splits: test (-1)

Metrics

• task-averaged success rate (primary) — range: percent
  • Percentage of episodes where the model successfully completes the manipulation task. Calculated as the average success rate across all 20 tasks for a given perturbation condition.

Input / output format

Input: Four camera RGB or RGBD views, a language instruction, and robot proprioception (arm pose).

Output: Continuous 7-DoF keypoint pose, discretized rotation (Euler angles), and binary gripper open/close state.

Scoring recipe

success_count = 0
for episode in episodes:
    if model_completes_task(episode):
        success_count += 1
success_rate = (success_count / len(episodes)) * 100
task_averaged_success_rate = np.mean([success_rate_per_task for task in tasks])

Common pitfalls

• Confusing RLBench default task variations (e.g., drawer location) with The Colosseum's environmental perturbations.
• Averaging success rates across tasks without reporting per-task performance, which masks task-specific failure modes.
• Using the final training checkpoint rather than validation-based selection, as the protocol fixes evaluation to the last checkpoint.

Evidence (verbatim from paper)

A test episode is successful if the model completes the task fully. We report the average success rate for each test set, further averaged across tasks, referred to as task-averaged success rate hereon.

Citation

@misc{pumacay2024colosseum,
  title={THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation},
  author={Pumacay et al. (2024)},
  year={2024},
  note={arXiv:2402.08191}
}

• arXiv: 2402.08191