Libero Plus Eval

libero-plus-eval

LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models — Fei et al. (2025) (arXiv:2510.13626, 2025)

What this evaluates

Evaluates the robustness of vision-language-action (VLA) models under realistic perturbations across seven dimensions (camera, robot, language, light, background, noise, layout). It probes visual shift tolerance, kinematic reasoning, and linguistic robustness by measuring success rates on a curated set of non-trivial tasks.

Datasets

• LIBERO-Plus — total 10030; splits: test (10030)

Metrics

• success rate (primary) — range: [0, 1]
  • Calculated as the fraction of tasks successfully completed by the model. Tasks are stratified into five difficulty levels (L1–L5) based on how many of four baseline models solved them.

Input / output format

Input: Perturbed visual observations (varying camera viewpoint, lighting, background, sensor noise, and layout), language instructions, and robot initial state.

Output: Robot action sequence or policy output.

Scoring recipe

def compute_success_rate(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) if len(gold) > 0 else 0.0

Common pitfalls

• Tasks are pre-filtered to remove ceiling effects, so performance reflects only non-trivial difficulty.
• Difficulty levels (L1-L5) are relative to four specific baseline models, not absolute ground truth.
• Language perturbations yield negligible performance drops, suggesting models rely on visual cues rather than true language understanding.

Evidence (verbatim from paper)

Figure [8] shows the success rates of each model across the five difficulty levels.

Citation

@misc{fei2025liberoplus,
  title={LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models},
  author={Fei et al. (2025)},
  year={2025},
  note={arXiv:2510.13626}
}

• arXiv: 2510.13626