Fus Multimodal Robot Eval

fus-multimodal-robot-eval

Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding — Jones et al. (2025) (arXiv:2501.04693, 2025)

What this evaluates

Evaluates a robot policy's ability to ground heterogeneous sensor modalities (vision, touch, sound) into language instructions for zero-shot task execution in partially observable environments. It probes multimodal prompting, compositional reasoning, and the necessity of auxiliary contrastive and language grounding losses.

Datasets

• WidowX Multimodal Teleoperation Dataset — total 26866; splits: train (26866), test (-1)

Metrics

• task success (primary) — range: [0, 1]
  • Fraction of successful rollouts out of 5 per scenario. Success is binary for reach/grasp/press, with intermediate success defined as 'object touched but not fully grasped'.

Input / output format

Input: Multimodal observations: third-person RGB (640x480), wrist RGB, two DIGIT tactile images (320x240), 1-second audio clip (44.1kHz), IMU data, plus a natural language instruction.

Output: Delta end-effector position commands at 5 Hz frequency.

Scoring recipe

def compute_task_success(predictions, gold, num_rollouts=5):
    successes = 0
    for i in range(num_rollouts):
        if is_successful_trajectory(predictions[i], gold):
            successes += 1
    return successes / num_rollouts

Common pitfalls

• Partial visual occlusion in the shopping bag task makes vision-only baselines fail; tactile/sound are required.
• Training from scratch on the 26K trajectory dataset without pre-training leads to poor performance; pre-training on OXE is necessary.
• Both contrastive and language generative losses are required; ablating either significantly drops performance on unseen objects.

Evidence (verbatim from paper)

We evaluate each model on several different scenarios (e.g., different objects and distractors) for each of the tasks, by running the same scenario for 5 different rollouts. Lighter shades of color represent intermediate task success, i.e., object touched but not fully grasped.

Citation

@misc{jones2025beyondsight,
  title={Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding},
  author={Jones et al. (2025)},
  year={2025},
  note={arXiv:2501.04693}
}

• arXiv: 2501.04693