Omniear Eval

omniear-eval

OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks — Wang et al. (2025) (arXiv:2508.05614, 2025)

What this evaluates

Evaluates embodied agent reasoning by testing how well models infer capability gaps, dynamic tool acquisition, and coordination needs from environmental constraints. Probes the ability to ground abstract reasoning in physical reality under partial observability.

Datasets

• EAR-Bench — total 2800; splits: test (2800); repo https://github.com/ZJU-REAL/OmniEmbodied

Metrics

• Success Rate (SR) (primary) — range: percent
  • Percentage of tasks completed successfully out of the total number of test scenarios.
• Step Count — range: other
  • Average number of actions executed for tasks that were completed successfully.
• Relative Step Ratio (RSR) — range: other
  • Ratio of expert trajectory length to model trajectory length (L_expert / L_model), measuring execution efficiency.

Input / output format

Input: Standardized prompts, environment descriptions, and task contexts under partial observability. Agents must explore to discover object locations and properties, with tool-dependent actions dynamically enabled based on context.

Output: A sequence of actions or tool-use commands selected from a standardized action vocabulary.

Scoring recipe

def compute_metrics(predictions, gold_trajectories):
    success_count = 0
    total_steps_successful = 0
    successful_count = 0
    for pred, gold in zip(predictions, gold_trajectories):
        if is_task_successful(pred, gold):
            success_count += 1
            total_steps_successful += len(pred)
            successful_count += 1
    sr = (success_count / len(predictions)) * 100
    avg_steps = total_steps_successful / successful_count if successful_count > 0 else 0
    return sr, avg_steps

Common pitfalls

• Assuming full environmental observability: The protocol explicitly implements partial observability, requiring agents to explore to discover object locations and properties.
• Equating model scale with reasoning capability: Performance on physical constraint tasks plateaus early regardless of parameter count, showing that scale improves execution/planning but not physical grounding.
• Ignoring the distinction between explicit and implicit collaboration: The benchmark isolates pure multi-agent reasoning by using centralized coordination, meaning models must infer coordination needs rather than rely on communication protocols.

Evidence (verbatim from paper)

Success Rate (SR) measures task completion percentage, Step Count indicates average actions for successful completion. Each model completes 2,800 test scenarios across seven task categories with three independent runs for statistical reliability.

Citation

@misc{wang2025omniear,
  title={OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2508.05614}
}

• arXiv: 2508.05614