Embodiedbrain Eval

embodiedbrain-eval

EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied Intelligence — Ding Zou et al. (2025) (arXiv:2510.20578, 2025)

What this evaluates

Evaluates an embodied AI model's capabilities in general multimodal reasoning, 3D spatial perception, and long-horizon task planning across multiple public benchmarks and a custom simulation environment.

Datasets

• MM-IFEval — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• BLINK — total ?; splits: test (-1)
• CV-Bench — total ?; splits: test (-1)
• EmbSpatial — total ?; splits: test (-1)
• ERQA — total ?; splits: test (-1)
• EgoPlan — total ?; splits: test (-1)
• EgoPlan2 — total ?; splits: test (-1)
• EgoThink — total ?; splits: test (-1)
• Internal Planning — total ?; splits: test (-1)
• VLM-PlanSim-99 — total 99; splits: test (99)

Metrics

• Action Pair Match F1-Score (primary) — range: [0, 1]
  • Computed from a cross-matching matrix between predicted and ground truth action sequences. Quantity score uses the Hungarian algorithm for max matches; order score uses Longest Common Subsequence (LCS). Precision = Score / Predicted Actions, Recall = Score / Ground Truth Actions, F1-Score = 2PrecisionRecall / (Precision+Recall).
• Task Success Rate — range: percent
  • Percentage of VLM-PlanSim-99 tasks where the parsed structured instructions successfully execute to completion in the AI2-THOR simulator.

Input / output format

Input: Natural language task goal and initial scene image (for VLM-PlanSim-99); standard multimodal prompts per benchmark.

Output: Unstructured text plan, parsed into structured executable instructions, or a list of predicted action pairs.

Scoring recipe

def score_planning(pred_actions, gt_actions):
    M = zeros(len(pred_actions), len(gt_actions))
    for i, pred in enumerate(pred_actions):
        matches = gpt5_mini_evaluator(pred, gt_actions)
        for j in matches: M[i, j] = 1
    qty_score = hungarian_algorithm(M)
    ord_score = lcs_length(pred_actions, gt_actions)
    prec = qty_score / len(pred_actions)
    rec = qty_score / len(gt_actions)
    return 2 * prec * rec / (prec + rec)

Common pitfalls

• Standard text similarity metrics (BLEU/ROUGE) are explicitly noted as insufficient for evaluating plan executability and logical coherence.
• The internal planning benchmark relies on a proprietary GPT-5-Mini model for action matching, making exact reproduction difficult without access to that API.
• VLM-PlanSim-99 requires a specific 4-layer object resolution and parsing pipeline before simulation execution; skipping this step invalidates the Task Success Rate.

Evidence (verbatim from paper)

The final evaluation metrics are composed of Match Quantity Metrics and Match Order Metrics, both presented as Precision, Recall, and F1-Score: Action Pair Match Quantity Score: The maximum number of matches (the match score) is computed from the cross-matching matrix M using the Hungarian Algorithm via the SciPy library. Action Pair Match Order Score: The order score is calculated using the Longest Common Subsequence (LCS) algorithm, where the length of the LCS serves as the match score. The metrics are calculated as follows, with Score representing the derived match score (either quantity or order): Precision = Score / Predicted Actions, Recall = Score / Ground Truth Actions, F1-Score = 2PrecisionRecall / (Precision+Recall)

Citation

@misc{zou2025embodiedbrain,
  title={EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied Intelligence},
  author={Ding Zou et al. (2025)},
  year={2025},
  note={arXiv:2510.20578}
}

• arXiv: 2510.20578