Vlabench Eval

vlabench-eval

VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks — Zhang et al. (2024) (arXiv:2412.18194, 2024)

What this evaluates

Evaluates the generalization, long-horizon reasoning, and language-conditioned manipulation capabilities of Vision-Language-Action (VLA) models, workflow frameworks, and Vision-Language Models (VLMs) in simulated robotic environments. It probes performance across seen/unseen objects, semantic instruction understanding, and composite task decomposition.

Datasets

• VLABench — total 1600; splits: seen (-1), unseen (-1)

Metrics

• task_progress_score (primary) — range: percent
  • A percentage-based score (0-100) measuring task completion rate or success rate across primitive and composite tasks. For VLMs, it is a weighted aggregation of four DAG-matching metrics (skill, parameter, dependency, and completeness) comparing predicted skill sequences to reference ones.

Input / output format

Input: For VLAs: multi-view visual observations and natural language instructions. For VLMs: two four-view images (one annotated with masks/labels, one reference) paired with a natural language instruction.

Output: For VLAs: continuous or discretized robot action sequences/trajectories. For VLMs: a Domain-Specific Language (DSL) output consisting of a sequence of skills, each with a name and parameters, conforming to predefined patterns.

Scoring recipe

def score_vlabench(predictions, gold, mode="interactive"):
    if mode == "interactive":
        env = VLABench_Env()
        return env.run_and_track_progress(predictions)
    else:
        pred_dag = build_dag(predictions)
        gold_dag = build_dag(gold)
        m1, m2, m3, m4 = compute_dag_metrics(pred_dag, gold_dag)
        return weighted_aggregate([m1, m2, m3, m4])

Common pitfalls

• Confusing interactive evaluation (environment simulation tracking task progress) with non-interactive evaluation (DAG matching of DSL outputs).
• Assuming the 'four metrics' for VLM scoring are standard NLP metrics; they specifically measure structural alignment between predicted and reference skill DAGs.
• Overlooking that 'seen' vs 'unseen' splits are category-level, meaning models may memorize specific object textures rather than generalize to new instances.

Evidence (verbatim from paper)

Interactive evaluation computes a task progress score based on the interaction with the environment. VLABench provides a controller that parses the DSL action sequences output by the VLM into executable actions, which are then applied in a simulation environment to interact with real-world objects.

Citation

@misc{zhang2024vlabench,
  title={VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2412.18194}
}

• arXiv: 2412.18194