Tamen Contact Rich Eval

tamen-contact-rich-eval

TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks — Longyan Wu et al. (2026) (arXiv:2604.07335, 2026)

What this evaluates

Probes a robot policy's ability to execute contact-rich bimanual manipulation tasks using visuo-tactile feedback. It evaluates robustness to visual disturbances, generalization to unseen object appearances, and the effectiveness of tactile pretraining and recovery data in imitation learning.

Datasets

• TAMEn Contact-Rich Manipulation Tasks — total 80; splits: test (80)

Metrics

• success rate (%) (primary) — range: percent
  • Calculated as the number of successful task executions divided by the total number of trials (20 per task), multiplied by 100. Success is defined by task-specific completion criteria (e.g., herbs successfully poured, cable fully seated, clip detached, stain removed).

Input / output format

Input: RGB images from wrist-mounted fisheye cameras and tactile sensor readings from four fingertip visuo-tactile sensors.

Output: Continuous 16-DoF action vector controlling two 7-DoF arms and two DH grippers.

Scoring recipe

def compute_success_rate(predictions, gold, task_name):
    total_trials = 20
    successful = 0
    for trial in range(total_trials):
        if task_completed(task_name, predictions[trial]):
            successful += 1
    return (successful / total_trials) * 100

Common pitfalls

• Confusing the large-scale pretraining dataset (FreeTacMan, 3M+ pairs) with the evaluation protocol, which uses only 20 trials per task.
• Overlooking the two distinct visual disturbance settings (full episode vs. post-grasp), which yield different success rates and should be reported separately.
• Assuming vision-only baselines are evaluated identically; tactile baselines require synchronized visuo-tactile input, and success often hinges on contact-rich stages where vision fails.

Evidence (verbatim from paper)

Each task in[Fig.˜6] is trained and evaluated over 20 trials. As shown in[Tab.˜III], incorporating tactile sensing improves the average success rate from 34% to 55%.

Citation

@misc{wu2026tamen,
  title={TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks},
  author={Longyan Wu et al. (2026)},
  year={2026},
  note={arXiv:2604.07335}
}

• arXiv: 2604.07335