# Vla Generality Benchmark Eval

> Evaluates the cross-domain generality of vision-language-action models across vision-language understanding, discrete multi-agent control, and continuous robot manipulation. Probes strict format compliance, semantic alignment, action prediction accuracy, and failure modes such as output collapse or modality misalignment. Use when the user wants to benchmark on PIQA, SQA3D, RoboVQA, ODINW, BFCL, Overcooked, Open-X, or asks about evaluating this task. Reports EMR.

- Skill: `qhjqhj00/vla-generality-benchmark-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/vla-generality-benchmark-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/vla-generality-benchmark-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/vla-generality-benchmark-eval

---


# vla-generality-benchmark-eval

> Benchmarking the Generality of Vision-Language-Action Models — Guruprasad et al. (2025) (arXiv:2512.11315, 2025)

## What this evaluates

Evaluates the cross-domain generality of vision-language-action models across vision-language understanding, discrete multi-agent control, and continuous robot manipulation. Probes strict format compliance, semantic alignment, action prediction accuracy, and failure modes such as output collapse or modality misalignment.

## Datasets

- **PIQA** — total ?; splits: test (-1)
- **SQA3D** — total ?; splits: test (-1)
- **RoboVQA** — total ?; splits: test (-1)
- **ODINW** — total ?; splits: test (-1)
- **BFCL** — total ?; splits: test (-1)
- **Overcooked** — total ?; splits: test (-1)
- **Open-X** — total ?; splits: test (-1)

## Metrics

- `EMR` **(primary)** — range: percent
  - Exact-Match Rate: percentage of predictions that exactly match the ground truth label or answer string after strict output parsing.
- `Semantic Similarity` — range: [0, 1]
  - Embedding-based cosine similarity between model output and ground truth, used as a forgiving measure for open-ended natural language answers.
- `Macro F1` — range: [0, 1]
  - Unweighted mean of recall per class, computed over discrete action labels or object detection categories.
- `BNAMAE` — range: [0, 1]
  - Baseline Normalized Average Mean Absolute Error: normalized L1 distance between predicted and ground truth continuous action vectors, averaged across tasks.
- `BNAMSE` — range: [0, 1]
  - Baseline Normalized Average Mean Squared Error: normalized L2 distance between predicted and ground truth continuous action vectors, averaged across tasks.

## Input / output format

**Input**: Vision-language tasks: image(s) and text prompts. Discrete control: multi-agent observation states. Continuous control: robot proprioceptive/visual observations; GPT-5 additionally receives action probabilities in the prompt. All evaluations use zero-shot task-specific prompting.

**Output**: VL tasks: short textual answers or class labels. Discrete control: discrete action labels. Continuous control: 12-dimensional action values or natural language tokens representing actions. Strict output parsing is applied to all responses.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_type):
    if task_type == 'vl':
        emr = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
        sim = cosine_similarity(embeddings(predictions), embeddings(golds))
        return {'EMR': emr, 'Semantic_Similarity': sim.mean()}
    elif task_type == 'discrete':
        return {'Macro_F1': macro_f1_score(golds, predictions)}
    elif task_type == 'continuous':
        errors = [np.linalg.norm(p - g) for p, g in zip(predictions, golds)]
        baseline = compute_baseline_error(golds)
        return {'BNAMAE': np.mean(errors) / baseline, 'BNAMSE': np.mean([e**2 for e in errors]) / baseline}
```

## Common pitfalls

- Models trained on set-of-marks or trace-of-marks tasks (e.g., Magma) often output spatial coordinates instead of text labels, causing artificially low EMR/F1 despite semantically relevant content.
- Action-tuned models (e.g., π0) suffer catastrophic language collapse, repeatedly emitting token fragments (e.g., 'increa') and failing strict format compliance on VL tasks.
- High-DoF continuous control tasks (e.g., Open-X Quadrupedal) require explicit action-space transformations; evaluating them as standard VL tasks without transformation yields misleadingly high errors.

## Evidence (verbatim from paper)

> As shown in Fig. 2, GPT-5 attains the highest exact-match rate (EMR) across PIQA, SQA3D, RoboVQA, ODINW, and BFCL, reflecting strong language grounding and format compliance.

## Citation

```bibtex
@misc{guruprasad2025benchmarking,
  title={Benchmarking the Generality of Vision-Language-Action Models},
  author={Guruprasad et al. (2025)},
  year={2025},
  note={arXiv:2512.11315}
}
```

- arXiv: 2512.11315