Emmo Eval

emmo-eval

EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environments — Dongping Li et al. (2025) (arXiv:2503.08604, 2025)

What this evaluates

This benchmark evaluates embodied mobile manipulation agents in open environments, testing their ability to execute long-horizon, language-conditioned tasks that require interleaved high-level planning and low-level continuous navigation/manipulation. It specifically probes reasoning fidelity, execution success, adaptability to failures, and path efficiency compared to expert trajectories.

Datasets

• EMMOE-100 — total 100; splits: train (90), test (10)

Metrics

• PLWSR (primary) — range: [0, 1]
  • Path Length Weighted Success Rate. Calculated as SR × (length of successful trajectory) / max(length of expert trajectory, length of successful trajectory). It measures the ability gap between the agent and the expert specifically on successful trajectories.
• SR — range: [0, 1] or percent
  • Success Rate. The proportion of tasks completed successfully out of total execution attempts.
• TP — range: [0, 1] or percent
  • Task Progress metric (defined in Section 2.3 of the full paper). Tracks overall task advancement or step efficiency.
• SRR — range: [0, 1] or percent
  • Success Re-planning Rate. Reflects the model's ability to adapt to environments and adjust from failure using feedback information.
• SER — range: [0, 1] or percent
  • Success End Rate. Reflects the model's ability to correctly determine when a task is completed and should be terminated.

Input / output format

Input: Language-conditioned task instructions, object names, background/environmental state information, and execution history/feedback from previous steps.

Output: Sequential action commands (e.g., 'Go to', 'Pick', 'Place', 'Open', 'Close', 'End'), low-level model selection choices, and re-planning outputs upon failure.

Scoring recipe

def compute_emmoe_metrics(predictions, gold, max_steps=20, runs=3):
    success_flags = []
    traj_lengths = []
    for run_preds in predictions:
        completed = check_task_completion(run_preds, gold)
        success_flags.append(completed)
        traj_lengths.append(len(run_preds))
    sr = sum(success_flags) / len(success_flags)
    successful_trajs = [l for l, s in zip(traj_lengths, success_flags) if s]
    expert_len = gold['expert_trajectory_length']
    avg_succ_len = sum(successful_trajs) / len(successful_trajs) if successful_trajs else 0
    plwsr = sr * (avg_succ_len / max(expert_len, avg_succ_len))
    # TP, SRR, SER computed per Sec 2.3 definitions
    return {'SR': sr, 'PLWSR': plwsr, 'TP': ..., 'SRR': ..., 'SER': ...}

Common pitfalls

• DPO alignment significantly improves training split performance but degrades generalization on the test split, particularly causing a sharp drop in SRR.
• High subtask success rates do not guarantee high overall task success; models often fail long-horizon tasks due to context forgetting or hallucination despite completing individual steps.
• Evaluation averages results over 3 runs per task with a strict 20-step limit, which heavily penalizes inefficient planners and can mask sporadic successes.

Evidence (verbatim from paper)

In addition to SR, TP, SER and SRR introduced in Section[2.3], we also choose Path Length Weighted SR (PLWSR)[[27]] as one of our evaluation metrics. PLWSR is defined as SR×(length of successful trajectory) / $max$(length of expert trajectory, length of successful trajectory) and measures the ability gap between the agent and the expert in successful trajectories. All tasks in EMMOE-100 will be used for evaluation, and the remaining ten untrained tasks will serve as our test set. Each task is executed three times with a maximum step limit of 20 each time, the average execution results will be used for the final calculation.

Citation

@misc{li2025emmoe,
  title={EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environments},
  author={Dongping Li et al. (2025)},
  year={2025},
  note={arXiv:2503.08604}
}

• arXiv: 2503.08604