# Embodied Arena Eval

> This benchmark suite evaluates embodied AI models across perception, spatial reasoning, navigation, and task planning. It aggregates 22 diverse benchmarks to measure capabilities like 2D/3D question answering, instruction following in navigation, and complex task decomposition. Use when the user wants to benchmark on Embodied Arena, or asks about evaluating this task. Reports Exact Matching Accuracy.

- Skill: `qhjqhj00/embodied-arena-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/embodied-arena-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/embodied-arena-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/embodied-arena-eval

---


# embodied-arena-eval

> Embodied Arena: A Comprehensive, Unified, and Evolving Evaluation Platform for Embodied AI — Ni et al. (2025) (arXiv:2509.15273, 2025)

## What this evaluates

This benchmark suite evaluates embodied AI models across perception, spatial reasoning, navigation, and task planning. It aggregates 22 diverse benchmarks to measure capabilities like 2D/3D question answering, instruction following in navigation, and complex task decomposition.

## Datasets

- **Embodied Arena** — total 64000; splits: test (-1)

## Metrics

- `Exact Matching Accuracy` **(primary)** — range: percent
  - Percentage of instances where the model's categorical output exactly matches the ground truth label.
- `Success Rate` — range: percent
  - Percentage of navigation or planning episodes that successfully reach the target or complete the task sequence.
- `Path Length Weighted Success Rate (SPL)` — range: percent
  - Success Rate weighted by the ratio of the optimal path length to the actual path length, evaluating navigation efficiency.
- `Fuzzy Matching Accuracy` — range: percent
  - Accuracy measured using rule-based metrics (CIDEr, BLEU, ROUGE, MRA) or LLM-based semantic evaluation for open-ended responses.

## Input / output format

**Input**: Visual context (2D image, 3D scene/point cloud, or video) combined with a natural language prompt specifying a question, navigation instruction, or task planning goal.

**Output**: Natural language text (categorical answer or descriptive caption) for QA tasks; action sequence or task completion status for navigation and planning tasks.

## Scoring recipe

```python
def compute_leaderboard_score(predictions, golds, benchmarks, dimensions):
    total_score = 0.0
    for n, bench in enumerate(benchmarks):
        dim_scores = []
        for m, dim in enumerate(bench.dimensions):
            k = len(dim.questions)
            c = sum(1 for pred, gold in zip(dim.predictions, dim.golds) if matches(pred, gold))
            s = (c / k) * 100 if k > 0 else 0
            dim_scores.append(s)
        bench_total = sum(dim_scores) / len(dim_scores)
        total_score += bench_total
    return total_score / len(benchmarks)
```

## Common pitfalls

- Aggregating scores across benchmarks with different numbers of capability dimensions without normalizing per dimension can bias the leaderboard.
- Fuzzy matching relies on rule-based n-gram overlap or LLM judges, which may not align with human judgment for spatial reasoning or open-ended descriptions.
- Static benchmark data limits evaluation of long-horizon planning and dynamic environment adaptation, as noted by the authors for future work.

## Evidence (verbatim from paper)

> During the evaluation phase, we select the corresponding evaluation metric based on the characteristics of the benchmark itself, which generally include the following types: Embodied Question Answering: Exact Matching Accuracy: Applied to benchmarks requiring precise categorical responses... Embodied Navigation Evaluation: Success Rate: Primary metric for navigation benchmarks... Path Length Weighted Success Rate (SPL): Evaluates navigation efficiency... Embodied Task Planning Evaluation: Task Completion Success Rate: Applied to benchmarks such as EB-ALFRED...

## Citation

```bibtex
@misc{ni2025embodiedarena,
  title={Embodied Arena: A Comprehensive, Unified, and Evolving Evaluation Platform for Embodied AI},
  author={Ni et al. (2025)},
  year={2025},
  note={arXiv:2509.15273}
}
```

- arXiv: 2509.15273