Smolvla Robotics Eval

smolvla-robotics-eval

SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics — Mustafa Shukor et al. (2025) (arXiv:2506.01844, 2025)

What this evaluates

Evaluates a vision-language-action model's ability to perform robotic manipulation tasks in both simulated and real-world environments. It probes visuomotor policy generalization, fine-grained task decomposition handling, and the impact of pretraining and inference modes on success rates.

Datasets

• LIBERO — total 1693; splits: eval (-1)
• Meta-World — total 2500; splits: eval (-1)
• SO100 Real-World Tasks — total 150; splits: eval (-1)
• SO101 Real-World Tasks — total 50; splits: eval (-1)

Metrics

• Success Rate (SR) (primary) — range: percent
  • Average score over 10 trials per task. In simulation, binary: 1 if fully completed, 0 otherwise. In real-world, fine-grained subtask scoring (e.g., 0.5 for grasp + 0.5 for place; max 1.0). Reported as percentage.

Input / output format

Input: RGB images (resized to 512×512), robot sensorimotor states, and natural language instructions.

Output: Action chunks of n=50 continuous actions predicted via flow matching.

Scoring recipe

def compute_sr(trials, mode):
    scores = []
    for trial in trials:
        if mode == 'simulation':
            scores.append(1.0 if trial.completed else 0.0)
        else:
            score = 0.0
            if trial.grasp: score += 0.5
            if trial.place: score += 0.5
            # Sorting uses 0.25 per subtask (grasp, match1, match2, grasp2)
            scores.append(min(score, 1.0))
    return sum(scores) / len(scores) * 100

Common pitfalls

• Assuming real-world success rates use binary completion; they actually use fine-grained subtask scoring that must be summed to a maximum of 1.0.
• Overlooking that the SO101 benchmark is strictly out-of-distribution for pretraining, as the model was explicitly not pretrained on any SO101 data.
• Confusing synchronous inference (standard robotics eval) with asynchronous inference, which trades some success rate for significantly lower latency and higher task throughput.

Evidence (verbatim from paper)

Evaluation metrics. We report success rate (SR) as the primary metric across all benchmarks. For simulation-based evaluations, SR is binary-set to 1 if the task is successfully completed, and 0 otherwise. For real-world evaluations, we adopt a more fine-grained scoring approach by decomposing each task into subtasks. For example, in the Pick-and-Place task, we assign a score of 0.5 for successfully picking the cube and an additional 0.5 for correctly placing it into the target container.

Citation

@misc{shukor2025smolvla,
  title={SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics},
  author={Mustafa Shukor et al. (2025)},
  year={2025},
  note={arXiv:2506.01844}
}

• arXiv: 2506.01844