qhjqhj00 rovi-manipulation-eval
Robotic Visual Instruction — Yanbang Li et al. (2025) (arXiv:2505.00693, 2025)
What this evaluates
This benchmark probes a model's ability to comprehend hand-drawn, object-centric visual instructions (arrows, circles, colors) and translate them into precise spatiotemporal action plans for robotic manipulation. It evaluates both high-level task reasoning and low-level execution accuracy in cluttered, unseen environments.
Datasets
- RoVI Book dataset — total 15000; splits: train (-1)
- SIMPLER — total ?; splits: test (3)
Metrics
action success rate(primary) — range: percent- Percentage of tasks that successfully meet the defined semantic goals over 10 trials per task.
spatiotemporal alignment— range: other- A 6-point Likert scale rating the consistency of movement trajectories and the alignment of an object’s final spatial state with semantic goals.
task and planning success rate— range: percent- Human-annotated success rate measuring the accuracy of VLM language responses in comprehending task definitions and decomposing complex tasks into sequential sub-goals.
Euclidean distance error— range: other- Mean pixel distance between predicted keypoint locations and ground-truth locations.
mAP— range: [0, 1]- Mean Average Precision calculated at a 50-pixel localization threshold for keypoint extraction.
Input / output format
Input: RGB-D camera observations of the environment paired with hand-drawn RoVI sketches containing symbolic elements (arrows, circles, colors, numbers) to encode spatial and temporal constraints.
Output: Language responses describing task comprehension and sequential sub-goal planning, or Python functions for low-level robotic control.
Scoring recipe
def score_action_success(predictions, gold, trials=10):
success_count = 0
for trial in range(trials):
if predictions[trial].meets_goal(gold.semantic_goal):
success_count += 1
return (success_count / trials) * 100
def score_alignment(predictions, gold):
return likert_scale(predictions.trajectory, predictions.final_state, gold)
def score_task_planning(predictions, gold):
return human_eval(predictions.language_response, gold.plan)
Common pitfalls
- Assuming standard language instructions are sufficient; RoVI requires pixel-level precision via symbolic drawing, which open-vocabulary detectors often miss.
- Evaluating only end-state accuracy without considering trajectory consistency; the protocol explicitly requires spatiotemporal alignment scoring via Likert scale.
- Using models without fine-tuning on the RoVI Book dataset for multi-step tasks; small models fail to comprehend RoVI without LoRA fine-tuning.
Evidence (verbatim from paper)
We report two metrics for assessing manipulation execution: action success rate, measuring the percentage of tasks that meet defined goals, and spatiotemporal alignment, evaluating the consistency of movement trajectories and the alignment of an object’s final spatial state with semantic goals. A 6-point Likert scale is used for assessment (details in the appendix). Each task is evaluated over 10 trials.
Citation
@misc{li2025rovi,
title={Robotic Visual Instruction},
author={Yanbang Li et al. (2025)},
year={2025},
note={arXiv:2505.00693}
}
- arXiv: 2505.00693