Vla Generality Benchmark Eval

vla-generality-benchmark-eval

Benchmarking the Generality of Vision-Language-Action Models — Guruprasad et al. (2025) (arXiv:2512.11315, 2025)

What this evaluates

Evaluates the cross-domain generality of vision-language-action models across vision-language understanding, discrete multi-agent control, and continuous robot manipulation. Probes strict format compliance, semantic alignment, action prediction accuracy, and failure modes such as output collapse or modality misalignment.

Datasets

• PIQA — total ?; splits: test (-1)
• SQA3D — total ?; splits: test (-1)
• RoboVQA — total ?; splits: test (-1)
• ODINW — total ?; splits: test (-1)
• BFCL — total ?; splits: test (-1)
• Overcooked — total ?; splits: test (-1)
• Open-X — total ?; splits: test (-1)

Metrics

• EMR (primary) — range: percent
  • Exact-Match Rate: percentage of predictions that exactly match the ground truth label or answer string after strict output parsing.
• Semantic Similarity — range: [0, 1]
  • Embedding-based cosine similarity between model output and ground truth, used as a forgiving measure for open-ended natural language answers.
• Macro F1 — range: [0, 1]
  • Unweighted mean of recall per class, computed over discrete action labels or object detection categories.
• BNAMAE — range: [0, 1]
  • Baseline Normalized Average Mean Absolute Error: normalized L1 distance between predicted and ground truth continuous action vectors, averaged across tasks.
• BNAMSE — range: [0, 1]
  • Baseline Normalized Average Mean Squared Error: normalized L2 distance between predicted and ground truth continuous action vectors, averaged across tasks.

Input / output format

Input: Vision-language tasks: image(s) and text prompts. Discrete control: multi-agent observation states. Continuous control: robot proprioceptive/visual observations; GPT-5 additionally receives action probabilities in the prompt. All evaluations use zero-shot task-specific prompting.

Output: VL tasks: short textual answers or class labels. Discrete control: discrete action labels. Continuous control: 12-dimensional action values or natural language tokens representing actions. Strict output parsing is applied to all responses.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type == 'vl':
        emr = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
        sim = cosine_similarity(embeddings(predictions), embeddings(golds))
        return {'EMR': emr, 'Semantic_Similarity': sim.mean()}
    elif task_type == 'discrete':
        return {'Macro_F1': macro_f1_score(golds, predictions)}
    elif task_type == 'continuous':
        errors = [np.linalg.norm(p - g) for p, g in zip(predictions, golds)]
        baseline = compute_baseline_error(golds)
        return {'BNAMAE': np.mean(errors) / baseline, 'BNAMSE': np.mean([e**2 for e in errors]) / baseline}

Common pitfalls

• Models trained on set-of-marks or trace-of-marks tasks (e.g., Magma) often output spatial coordinates instead of text labels, causing artificially low EMR/F1 despite semantically relevant content.
• Action-tuned models (e.g., π0) suffer catastrophic language collapse, repeatedly emitting token fragments (e.g., 'increa') and failing strict format compliance on VL tasks.
• High-DoF continuous control tasks (e.g., Open-X Quadrupedal) require explicit action-space transformations; evaluating them as standard VL tasks without transformation yields misleadingly high errors.

Evidence (verbatim from paper)

As shown in Fig. 2, GPT-5 attains the highest exact-match rate (EMR) across PIQA, SQA3D, RoboVQA, ODINW, and BFCL, reflecting strong language grounding and format compliance.

Citation

@misc{guruprasad2025benchmarking,
  title={Benchmarking the Generality of Vision-Language-Action Models},
  author={Guruprasad et al. (2025)},
  year={2025},
  note={arXiv:2512.11315}
}

• arXiv: 2512.11315