Embodied Reasoner Eval

embodied-reasoner-eval

Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks — Wenqi Zhang et al. (2025) (arXiv:2503.21696, 2025)

What this evaluates

Evaluates an agent's ability to perform long-horizon embodied interactive tasks by synergizing visual search, reasoning, and action. It probes spatial reasoning, self-reflection, and planning capabilities in both simulated and real-world environments.

Datasets

• Unspecified (Simulated & Real-world tasks) — total ?; splits: simulated (-1), real-world (30)

Metrics

• success_rate (primary) — range: percent
  • The percentage of tasks successfully completed out of the total number of evaluated tasks.
• search_efficiency — range: percent
  • A relative measure of how quickly or effectively the model locates target objects compared to baselines, reported as percentage improvement.
• task_completeness — range: percent
  • An assessment of whether all required sub-goals and final objectives within a multi-step task are fully fulfilled.
• repetitive_exploration_rate — range: percent
  • Measures how often the model navigates to the same area within its trajectory, quantifying redundant search behavior.

Input / output format

Input: Real-time visual input (images or camera feed) paired with natural language task instructions.

Output: Action command to be executed by the environment or human operator.

Scoring recipe

def compute_metrics(predictions, gold_labels, trajectories):
    success_rate = sum(1 for p, g in zip(predictions, gold_labels) if p == g) / len(predictions) * 100
    rer_scores = []
    for traj in trajectories:
        repeated = count_repeated_area_visits(traj)
        rer_scores.append(repeated / len(traj) * 100)
    return {'success_rate': success_rate, 'repetitive_exploration_rate': sum(rer_scores)/len(rer_scores)}

Common pitfalls

• Models may over-explore on simpler tasks, leading to missed detections of nearby objects.
• Baseline models frequently exhibit repetitive searching behaviors due to weak temporal reasoning and context awareness.
• Performance drops significantly on long-horizon tasks exceeding five key actions for baseline models.

Evidence (verbatim from paper)

To quantify this, we define a repetitive exploration rate (RER), which measures how often the model navigates to the same area within its trajectory. As shown in [Fig. 6], our models (Embodied-Reasoner / Explorer) consistently exhibit a significantly lower RER (-50%) compared to baseline models across all four tasks.

Citation

@misc{zhang2025embodiedreasoner,
  title={Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks},
  author={Wenqi Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.21696}
}

• arXiv: 2503.21696