Embodied Arena Eval

embodied-arena-eval

Embodied Arena: A Comprehensive, Unified, and Evolving Evaluation Platform for Embodied AI — Ni et al. (2025) (arXiv:2509.15273, 2025)

What this evaluates

This benchmark suite evaluates embodied AI models across perception, spatial reasoning, navigation, and task planning. It aggregates 22 diverse benchmarks to measure capabilities like 2D/3D question answering, instruction following in navigation, and complex task decomposition.

Datasets

• Embodied Arena — total 64000; splits: test (-1)

Metrics

• Exact Matching Accuracy (primary) — range: percent
  • Percentage of instances where the model's categorical output exactly matches the ground truth label.
• Success Rate — range: percent
  • Percentage of navigation or planning episodes that successfully reach the target or complete the task sequence.
• Path Length Weighted Success Rate (SPL) — range: percent
  • Success Rate weighted by the ratio of the optimal path length to the actual path length, evaluating navigation efficiency.
• Fuzzy Matching Accuracy — range: percent
  • Accuracy measured using rule-based metrics (CIDEr, BLEU, ROUGE, MRA) or LLM-based semantic evaluation for open-ended responses.

Input / output format

Input: Visual context (2D image, 3D scene/point cloud, or video) combined with a natural language prompt specifying a question, navigation instruction, or task planning goal.

Output: Natural language text (categorical answer or descriptive caption) for QA tasks; action sequence or task completion status for navigation and planning tasks.

Scoring recipe

def compute_leaderboard_score(predictions, golds, benchmarks, dimensions):
    total_score = 0.0
    for n, bench in enumerate(benchmarks):
        dim_scores = []
        for m, dim in enumerate(bench.dimensions):
            k = len(dim.questions)
            c = sum(1 for pred, gold in zip(dim.predictions, dim.golds) if matches(pred, gold))
            s = (c / k) * 100 if k > 0 else 0
            dim_scores.append(s)
        bench_total = sum(dim_scores) / len(dim_scores)
        total_score += bench_total
    return total_score / len(benchmarks)

Common pitfalls

• Aggregating scores across benchmarks with different numbers of capability dimensions without normalizing per dimension can bias the leaderboard.
• Fuzzy matching relies on rule-based n-gram overlap or LLM judges, which may not align with human judgment for spatial reasoning or open-ended descriptions.
• Static benchmark data limits evaluation of long-horizon planning and dynamic environment adaptation, as noted by the authors for future work.

Evidence (verbatim from paper)

During the evaluation phase, we select the corresponding evaluation metric based on the characteristics of the benchmark itself, which generally include the following types: Embodied Question Answering: Exact Matching Accuracy: Applied to benchmarks requiring precise categorical responses... Embodied Navigation Evaluation: Success Rate: Primary metric for navigation benchmarks... Path Length Weighted Success Rate (SPL): Evaluates navigation efficiency... Embodied Task Planning Evaluation: Task Completion Success Rate: Applied to benchmarks such as EB-ALFRED...

Citation

@misc{ni2025embodiedarena,
  title={Embodied Arena: A Comprehensive, Unified, and Evolving Evaluation Platform for Embodied AI},
  author={Ni et al. (2025)},
  year={2025},
  note={arXiv:2509.15273}
}

• arXiv: 2509.15273