Robointer Vqa Eval

robointer-vqa-eval

RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation — Hao Li et al. (2026) (arXiv:2602.09973, 2026)

What this evaluates

Evaluates embodied reasoning capabilities of vision-language models on robotic manipulation tasks. It probes spatial understanding and generation (e.g., object grounding, grasp pose prediction) and temporal understanding and generation (e.g., motion trace reconstruction, multi-step planning) across diverse indoor and tabletop scenarios.

Datasets

• RoboInter-VQA — total ?; splits: train (-1), val (-1), test (7246); repo https://github.com/InternRobotics/RoboInter

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered selection, judgment, and generation tasks. For spatial/temporal generation tasks, correctness is determined by IoU thresholds for bounding boxes or exact match for categorical outputs.

Input / output format

Input: Video frames or image sequences paired with text prompts conditioning on varying prior information (e.g., past subtasks, overall instructions, or initial waypoints).

Output: Text responses for selection/judgment tasks; 2D bounding boxes, grasp poses, placement proposals, or gripper coordinates for spatial generation; motion traces or multi-step action plans for temporal generation.

Scoring recipe

def score(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if gold_item['type'] in ['selection', 'judgment']:
            if pred.strip().lower() == gold_item['answer'].strip().lower():
                correct += 1
        elif gold_item['type'] in ['spatial_gen', 'temporal_gen']:
            if compute_iou(pred['coords'], gold_item['coords']) >= 0.5:
                correct += 1
    return correct / len(gold)

Common pitfalls

• Information leakage between training and validation sets if the 7,246-video evaluation pool is not strictly held out.
• Generation tasks vary in difficulty based on contextual completeness (e.g., with/without initial waypoints), requiring careful stratified reporting.
• Raw data lacks reliable camera parameters, so end-effector traces and 3D-to-2D projections rely on estimated calibration matrices that may introduce noise.

Evidence (verbatim from paper)

To prevent information leakage between training and validation, we carefully designate 7,246 videos as the evaluation pool with the remaining data used for training, and sample validation sets for each question category from this pool. Tasks are organized along two axes: (i) intermediate representation type (spatial vs. temporal) and (ii) target capability (understanding vs. generation).

Citation

@misc{li2026robointer,
  title={RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation},
  author={Hao Li et al. (2026)},
  year={2026},
  note={arXiv:2602.09973}
}

• arXiv: 2602.09973