Cebench Eval

cebench-eval

Rethinking the Practicality of Vision-language-action Model: A Comprehensive Benchmark and An Improved Baseline — Song et al. (2026) (arXiv:2602.22663, 2026)

What this evaluates

Evaluates vision-language-action (VLA) models on cross-embodiment robotic manipulation tasks, including single-arm, bimanual, and mobile manipulation. It probes spatial reasoning, visual generalization under domain randomization, and the ability to unify navigation and manipulation in a single policy.

Datasets

• CEBench — total ?; splits: test (-1), seen (-1), domain_randomization (-1)

Metrics

• success_rate (primary) — range: percent
  • Binary metric indicating whether a task episode is completed successfully. Reported as a percentage across multiple trials (e.g., 100 or 1000 runs per task).
• avg_completed_trajectory_length — range: other
  • Average number of steps or subtasks successfully completed per trajectory across all trials.

Input / output format

Input: Multi-view camera images, language instructions, and proprioceptive state observations.

Output: Unified action commands for navigation and manipulation (e.g., joint velocities or end-effector poses).

Scoring recipe

def compute_metrics(predictions, gold, episodes):
    success_count = 0
    total_steps = 0
    for ep in episodes:
        if check_success(ep, predictions[ep]):
            success_count += 1
            total_steps += len(ep.completed_trajectory)
    success_rate = (success_count / len(episodes)) * 100
    avg_len = total_steps / success_count if success_count > 0 else 0
    return {'success_rate': success_rate, 'avg_completed_trajectory_length': avg_len}

Common pitfalls

• Domain randomization (DR) settings cause dramatic success rate drops for baseline models, so evaluating only on 'seen' tasks overestimates generalization.
• Mobile manipulation success depends heavily on navigation accuracy; poor navigation causes manipulation success to drop to near zero regardless of the manipulation policy.
• Robotics evaluations are highly stochastic; the protocol requires 100 to 1000 trials per task to ensure statistical reliability.

Evidence (verbatim from paper)

We report the average completed trajectory length (Avg. Len.) across all five subtasks as well as success rates on each subtask. Following the official ABC→D settings, the evaluation is conducted in an unseen scene. To ensure reliable evaluation, we test each method 1000 times.

Citation

@misc{song2026cebench,
  title={Rethinking the Practicality of Vision-language-action Model: A Comprehensive Benchmark and An Improved Baseline},
  author={Song et al. (2026)},
  year={2026},
  note={arXiv:2602.22663}
}

• arXiv: 2602.22663