Vlmbench Eval

vlmbench-eval

VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation — Zheng et al. (2022) (arXiv:2206.08522, 2022)

What this evaluates

This benchmark evaluates a robot agent's ability to execute 6D manipulation tasks guided by natural language instructions and visual observations. It probes compositional reasoning, object localization, and precise pose estimation in both seen and unseen object settings.

Datasets

• VLMbench — total ?; splits: test (-1)

Metrics

• success rate (primary) — range: percent
  • Calculated as the number of success conditions satisfied divided by the total number of tests. Success is determined by an object detector (target objects moved inside a predefined space) or a joint detector (joint angles reach a predefined range).

Input / output format

Input: Multi-view RGBD observations, natural language instructions encoded via CLIP, and optionally pre-generated waypoints or gripper states for sub-goal decomposition.

Output: 6DoF action parameters (x, y, z position and 3D orientation) for each step in the task sequence.

Scoring recipe

def compute_success_rate(predictions, gold):
    successes = 0
    for task in gold:
        # Check success conditions defined in task config
        if object_detector(task.target_objects, task.predefined_space) or \
           joint_detector(task.joint_angles, task.predefined_range):
            successes += 1
    return (successes / len(gold)) * 100

Common pitfalls

• Language-only agents fail completely because they cannot localize objects in 3D space without visual input.
• Vision-only agents struggle significantly on tasks requiring strict pose constraints (e.g., Drop, Pour, Shape Sorter) without language guidance.
• Position estimation is typically the primary bottleneck for task success, as object localization is harder than orientation estimation in this setup.

Evidence (verbatim from paper)

Success rate is used as the primary evaluation metric, calculated by dividing the number of success conditions satisfied by the number of tests. We use the average success rate of all variations for each task category. The success conditions are mainly determined by an object detector or a joint detector. The object detector returns true if particular objects have moved inside the predefined space, and the joint detector returns true when the joint angle reaches the predefined range.

Citation

@misc{zheng2022vlmbench,
  title={VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation},
  author={Zheng et al. (2022)},
  year={2022},
  note={arXiv:2206.08522}
}

• arXiv: 2206.08522