Starvla Alpha Generalist Eval

starvla-alpha-generalist-eval

StarVLA-$\alpha$: Reducing Complexity in Vision-Language-Action Systems — Ye et al. (2026) (arXiv:2604.11757, 2026)

What this evaluates

Evaluates whether a single Vision-Language-Action model can generalize across diverse robotic manipulation benchmarks without task-specific fine-tuning. It probes the model's cross-embodiment generalization and robustness to varying action spaces and task distributions.

Datasets

• LIBERO — total ?; splits: train (-1), test (-1)
• SimplerEnv — total ?; splits: train (-1), test (-1)
• RoboTwin 2.0 — total ?; splits: train (-1), test (-1)
• RoboCasa-GR1 — total ?; splits: train (-1), test (-1)
• RoboChallenge — total ?; splits: test (-1)

Metrics

• success_rate (primary) — range: percent
  • Calculated as the percentage of successful task completions out of the total number of evaluation trials.
• progress_score — range: other
  • A continuous progress metric averaged across trials for each task, reported alongside success rate.

Input / output format

Input: Visual observations, natural language task instructions, and robot proprioceptive/state observations.

Output: 32-dimensional action vector (uniformly padded across robots with varying degrees of freedom).

Scoring recipe

def compute_metrics(predictions, gold, total_trials):
    # gold: binary flags for task completion per trial
    success_rate = sum(gold) / total_trials * 100
    # progress_score: average of continuous progress values per trial
    progress_score = sum(progress_values) / total_trials
    return success_rate, progress_score

Common pitfalls

• Fine-tuning the model on individual benchmark datasets before evaluation, which violates the generalist protocol.
• Using complex, robot-specific action parameterizations (e.g., RDT Action, Multi-Action Head) instead of the specified simple padding strategy.
• Ignoring the uniform 32-dimensional action space requirement, leading to incompatible evaluation across different robot embodiments.

Evidence (verbatim from paper)

In this setting, we utilize all datasets to train a single model jointly and directly evaluate it on multiple benchmarks, without any additional fine-tuning on benchmark-specific datasets. Specifically, we select LIBERO, SimplerEnv, RoboTwin 2.0, and RoboCasa-GR1 as the unified benchmark suite and train the model on the combined training sets of these benchmarks. SR represents success rate, and score represents progress score.

Citation

@misc{ye2026starvla,
  title={StarVLA-$\alpha$: Reducing Complexity in Vision-Language-Action Systems},
  author={Ye et al. (2026)},
  year={2026},
  note={arXiv:2604.11757}
}

• arXiv: 2604.11757