Roboarena Eval

roboarena-eval

RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies — Atreya et al. (2025) (arXiv:2506.18123, 2025)

What this evaluates

This benchmark evaluates the ranking accuracy and sample efficiency of generalist robot policies in real-world environments. It probes whether a distributed, pairwise comparison framework can reliably approximate an exhaustive oracle ranking across diverse scenes and tasks.

Datasets

• RoboArena Real-World Policy Evaluation — total 4284; splits: test (4284)

Metrics

• Pearson correlation r (primary) — range: [-1, 1]
  • Measures the linear correlation between the predicted policy ranking and the oracle ranking derived from exhaustive evaluation. Ranges from -1 to 1, where 1 indicates perfect rank agreement.
• Mean Maximum Rank Violation (MMRV) — range: other
  • A ranking metric that quantifies the maximum discrepancy in rank positions between the predicted and oracle rankings, explicitly accounting for performance differences between policies rather than just ordinal swaps.

Input / output format

Input: Pairwise video observations of two robot policies executing the same task instruction in a real-world scene, accompanied by the task prompt.

Output: Per comparison: progress scores for each policy, a preference/winner label, and a task category. Aggregated output: a global ranked list of policies.

Scoring recipe

def compute_metrics(predicted_ranking, oracle_ranking):
    policies = list(predicted_ranking.keys())
    pred_vals = [predicted_ranking[p] for p in policies]
    oracle_vals = [oracle_ranking[p] for p in policies]
    r = pearsonr(pred_vals, oracle_vals)
    mmrv = max(abs(predicted_ranking[p] - oracle_ranking[p]) for p in policies)
    return r, mmrv

Common pitfalls

• Progress-based rankings alone can miss nuanced policy behaviors (e.g., speed or confidence) when evaluators assign identical progress scores to both policies in a pair.
• Conventional fixed-task evaluations are insufficient for generalist policies because they lack the environmental and task diversity needed to capture robust performance.
• MMRV accounts for performance differences between policies, not just ordinal rank swaps, so interpreting it as a simple rank-distance metric is incorrect.

Evidence (verbatim from paper)

We follow Li et al. [29] and report Pearson correlation $r$ as well as Mean Maximum Rank Violation (MMRV), a ranking metric that takes the performance difference between policies into account.

Citation

@misc{atreya2025roboarena,
  title={RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies},
  author={Atreya et al. (2025)},
  year={2025},
  note={arXiv:2506.18123}
}

• arXiv: 2506.18123