Contact Rich Manipulation Eval

contact-rich-manipulation-eval

FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation — Wu et al. (2025) (arXiv:2506.01941, 2025)

What this evaluates

Evaluates the success rate of imitation learning policies in contact-rich manipulation tasks requiring precise force control, slip detection, and in-hand pose estimation. It compares vision-only baselines against visuo-tactile policies with and without temporal-aware contrastive pretraining.

Datasets

• Contact-Rich Manipulation Tasks — total ?; splits: test (-1)

Metrics

• success rate (primary) — range: percent
  • Calculated as the number of successful task completions divided by the total number of evaluation trials (20 per task), expressed as a percentage. Success is determined by whether the task objective is met without damage or failure.
• completion_rate — range: percent
  • Fully completed tasks as a percentage of those initiated during the user study data collection phase.
• CPUT — range: other
  • Completion per Unit Time, defined as completion_rate multiplied by collection efficiency (inverse of data collection time).

Input / output format

Input: RGB images from wrist camera (vision-only) or synchronized RGB images and tactile observations from wearable fingertip sensors (visuo-tactile).

Output: Robot joint control commands to execute the manipulation task.

Scoring recipe

def compute_success_rate(predictions, gold, total_trials=20):
    successes = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (successes / total_trials) * 100

Common pitfalls

• The evaluation uses a fixed number of 20 trials per task rather than standard train/val/test splits.
• Confusing the data collection system metrics (CPUT, completion rate) with the policy evaluation metric (success rate).
• The tactile encoder pretraining uses a specific multi-positive contrastive objective, not standard supervised or self-supervised methods.

Evidence (verbatim from paper)

We collect data and train policies with tasks in Fig. 3 and evaluate each task over 20 trials. ... The vision-only baseline ACT achieves low performance across all tasks, with an average success rate of 21%. ... When tactile feedback is incorporated naively, i.e., without pre-training, performance improves significantly, with the average success rate increasing to 55%. ... Incorporating time-aware visual-tactile pairs in pretraining leads to a notable performance boost, with the average success rate increasing to 71%.

Citation

@misc{wu2025freetacman,
  title={FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2506.01941}
}

• arXiv: 2506.01941