Peg Insertion Eval

peg-insertion-eval

Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks — Lee et al. (2018) (arXiv:1810.10191, 2018)

What this evaluates

Evaluates a robot policy's ability to perform contact-rich manipulation by jointly reasoning over visual and haptic feedback. It measures how well a learned representation improves sample efficiency, generalizes across peg geometries, and recovers from perturbations during peg insertion tasks.

Datasets

• Custom Peg Insertion Environment — total ?; splits: train (-1), eval (-1)

Metrics

• sum of rewards achieved in an episode, normalized by the highest attainable reward (primary) — range: [0, 1]
  • Sum of rewards achieved in an episode divided by the highest attainable reward (10).
• percentage of evaluation episodes — range: percent
  • Percentage of evaluation episodes falling into four categories: completed insertion (peg reaches bottom), inserted into hole (peg enters but not bottom), touched the box (contact only), or failed (no contact).

Input / output format

Input: 128x128 RGB image, end-effector pose, linear and angular velocity, and 6-axis force/torque vector.

Output: Action commands at 20Hz control frequency.

Scoring recipe

def compute_metrics(rewards_per_episode, episode_outcomes):
    max_reward = 10.0
    normalized_rewards = [sum(r) / max_reward for r in rewards_per_episode]
    stage_counts = {'completed': 0, 'inserted': 0, 'touched': 0, 'failed': 0}
    for outcome in episode_outcomes:
        stage_counts[outcome] += 1
    stage_rates = {k: v / len(episode_outcomes) * 100 for k, v in stage_counts.items()}
    return normalized_rewards, stage_rates

Common pitfalls

• Confusing the internal staged reward phases (reaching, alignment) with the final evaluation categories (completed, inserted, touched, failed).
• Reporting raw cumulative reward instead of normalizing by the theoretical maximum of 10.
• Assuming simulation and real-robot sensor inputs are identical without accounting for Kinect vs. fixed camera and OptoForce vs. simulated contact models.

Evidence (verbatim from paper)

We report the quantitative performance of the policies using the sum of rewards achieved in an episode, normalized by the highest attainable reward. We also provide the statistics of the stages of the peg insertion task that each policy can achieve, and report the percentage of evaluation episodes in the following four categories: 1) completed insertion: the peg reaches bottom of the hole; 2) inserted into hole: the peg goes into the hole but has not reached the bottom; 3) touched the box: the peg only makes contact with the box; 4) failed: the peg fails to reach the box.

Citation

@misc{lee2018makingsense,
  title={Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks},
  author={Lee et al. (2018)},
  year={2018},
  note={arXiv:1810.10191}
}

• arXiv: 1810.10191