Mimo Embodied Eval

mimo-embodied-eval

MiMo-Embodied: X-Embodied Foundation Model Technical Report — Hao et al. (2025) (arXiv:2511.16518, 2025)

What this evaluates

Evaluates a cross-embodied foundation model's capabilities in affordance prediction, task planning, spatial understanding, and autonomous driving perception/prediction/planning across diverse visual and language inputs.

Datasets

• RoboRefIt — total ?; splits: test (-1)
• Where2Place — total ?; splits: test (-1)
• VABench-Point — total ?; splits: test (-1)
• Part-Afford — total ?; splits: test (-1)
• RoboAfford-Eval — total ?; splits: test (-1)
• EgoPlan2 — total ?; splits: test (-1)
• RoboVQA — total ?; splits: test (-1)
• Cosmos-Reason1 — total ?; splits: test (-1)
• CV-Bench — total ?; splits: test (-1)
• ERQA — total ?; splits: test (-1)
• EmbSpatial — total ?; splits: test (-1)
• SAT — total ?; splits: test (-1)
• RoboSpatial — total ?; splits: test (-1)
• RefSpatial-Bench — total ?; splits: test (-1)
• CRPE-relation — total ?; splits: test (-1)
• MetaVQA — total ?; splits: test (-1)
• VSI-Bench — total ?; splits: test (-1)
• CODA-LM — total ?; splits: test (-1)
• DRAMA — total ?; splits: test (-1)
• MME-RealWorld — total ?; splits: test (-1)
• IDKB — total ?; splits: test (-1)
• OmniDrive — total ?; splits: test (-1)
• NuInstruct — total ?; splits: test (-1)
• DriveLM — total ?; splits: test (-1)
• MAPLM — total ?; splits: test (-1)
• nuScenes-QA — total ?; splits: test (-1)
• LingoQA — total ?; splits: test (-1)
• BDD-X — total ?; splits: test (-1)
• DriveAction — total ?; splits: test (-1)

Metrics

• precision (primary) — range: percent
  • Percentage of correct predictions or spatially valid outputs (e.g., predicted points falling within human-annotated regions) relative to the total number of test instances.

Input / output format

Input: Multimodal inputs consisting of single-view images, multi-view video frames, or panoramic scenes paired with natural language instructions, queries, or action goals.

Output: Textual answers, predicted coordinate points, or generated action sequences/plans corresponding to the input instructions or queries.

Scoring recipe

def compute_precision(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if isinstance(gold_item, dict) and 'region' in gold_item:
            if point_in_region(pred, gold_item['region']):
                correct += 1
        else:
            if pred == gold_item or is_semantically_equivalent(pred, gold_item):
                correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Many benchmarks require custom evaluation scripts (marked with * in tables) rather than official leaderboard scores, so reproducing results requires accessing the authors' evaluation framework.
• Spatial grounding tasks (e.g., VABench-Point) require specific coordinate tolerance thresholds that are not explicitly detailed in the text, making exact replication difficult.
• Some benchmarks evaluate multi-step planning or causal reasoning where outputs are open-ended, requiring LLM-based or heuristic scoring not fully specified.

Evidence (verbatim from paper)

The pointing subset of VABench [[63]] (VABench-Point) evaluates a model’s precision in grounding natural language commands to specific coordinate locations for robotic manipulation, requiring predicted points to fall within human-annotated regions of target objects or free space. Results marked with * are obtained using our evaluation framework.

Citation

@misc{hao2025mimoembodied,
  title={MiMo-Embodied: X-Embodied Foundation Model Technical Report},
  author={Hao et al. (2025)},
  year={2025},
  note={arXiv:2511.16518}
}

• arXiv: 2511.16518