qhjqhj00 vla-cross-embodiment-eval
X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model — Zheng et al. (2025) (arXiv:2510.10274, 2025)
What this evaluates
Evaluates a vision-language-action model's ability to generalize across diverse robotic embodiments, simulation environments, and real-world platforms. It probes cross-embodiment adaptation, parameter-efficient fine-tuning capabilities, and dexterous manipulation performance.
Datasets
- Libero — total ?; splits: test (-1)
- Simpler — total ?; splits: test (-1)
- Calvin — total ?; splits: test (-1)
- VLABench — total ?; splits: test (-1)
- RoboTwin-2.0 — total ?; splits: test (-1)
- NAVSIM — total ?; splits: test (-1)
- BridgeData-v2 — total ?; splits: test (-1)
- Soft-Fold — total 1200; splits: test (1200)
Metrics
success_rate(primary) — range: percent- Percentage of episodes where the robot successfully completes the specified task within the environment's constraints or step limit.
l1_action_error— range: other- Mean absolute error between predicted actions (after flow-matching denoising) and ground-truth actions on held-out validation sets.
Input / output format
Input: Multi-view RGB images concatenated with natural language task instructions.
Output: Continuous action vectors (e.g., end-effector poses/velocities) generated via flow-matching denoising.
Scoring recipe
def compute_success_rate(predictions, gold, env_config):
successes = 0
for pred, gold_inst in zip(predictions, gold):
if env_config.check_success(pred, gold_inst):
successes += 1
return (successes / len(gold)) * 100
Common pitfalls
- Different benchmarks report success rates on different sub-tasks (e.g., Libero-Spatial vs. Libero-Long); results must specify the exact variant.
- Real-world success rates depend heavily on contact-based or goal-reaching criteria that vary per environment; mixing simulation and real-world metrics without clarification causes confusion.
- PEFT results use only 1% of parameters (9M); comparing them directly to fully fine-tuned baselines without noting parameter count is misleading.
Evidence (verbatim from paper)
achieving 93% and 54% success rates on Libero and Simpler-WidowX benchmarks, respectively.
Citation
@misc{zheng2025xvla,
title={X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model},
author={Zheng et al. (2025)},
year={2025},
note={arXiv:2510.10274}
}
- arXiv: 2510.10274