Momagraph Bench Eval

momagraph-bench-eval

MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning — Ju et al. (2025) (arXiv:2512.16909, 2025)

What this evaluates

Evaluates embodied task planning and visual correspondence capabilities of vision-language models. It probes spatial-functional reasoning, multi-step action planning, and cross-view consistency in indoor scenes.

Datasets

• MomaGraph-Bench — total ?; splits: test (-1)
• BLINK — total ?; splits: test (-1)

Metrics

• accuracy (%) (primary) — range: percent
  • Calculated as the number of correctly predicted task plans or correspondence answers divided by the total number of instances, multiplied by 100.
• success rate (%) — range: percent
  • Calculated as the number of successful multi-step task trials divided by the total number of trials, multiplied by 100.

Input / output format

Input: Multi-view RGB-D observations and natural language task instructions.

Output: Task-oriented scene graph G_T (nodes, spatial/functional edges, action types) followed by a structured action sequence, or direct task plan/answer.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

def compute_success_rate(success_flags, total_trials):
    return (sum(success_flags) / total_trials) * 100

Common pitfalls

• Confusing the 'Direct Plan (w/o Graph)' baseline with the 'Graph-then-Plan (w/ Graph)' setting, which significantly impacts reported scores.
• Assuming overall accuracy masks severe degradation on higher-complexity tiers (T3/T4) where open-source models drop sharply.
• Overlooking that real-robot success rates are measured per trial across 10 viewpoint changes, not per individual action step.

Evidence (verbatim from paper)

We report accuracy (%) across four tiers (T1–T4) and the overall score, with and without graph-based reasoning.

Citation

@misc{ju2025momagraph,
  title={MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning},
  author={Ju et al. (2025)},
  year={2025},
  note={arXiv:2512.16909}
}

• arXiv: 2512.16909