Reasonplan3d Eval

reasonplan3d-eval

Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planning — Xueying Jiang et al. (arXiv:2503.12974, 2025)

What this evaluates

Probes a model's ability to infer implicit human intentions from natural language and generate multi-step, route-aware activity plans grounded in 3D scene segmentation. It evaluates both textual planning coherence and spatial reasoning over 3D environments.

Datasets

• ReasonPlan3D — total ?; splits: val (-1)

Metrics

• BLEU-4 (primary) — range: [0, 1]
  • Computes the geometric mean of 1- to 4-gram precisions between predicted and reference text, penalizing brevity via a brevity penalty. Standard convention uses weights (0.25, 0.25, 0.25, 0.25) and sentence-level averaging.
• CIDEr — range: [0, 1]
  • Measures consensus between predicted and reference texts using TF-IDF weighted n-gram similarities. Higher scores indicate better alignment with human reference descriptions.
• METEOR — range: [0, 1]
  • Aligns predicted and reference texts using exact, stem, synonym, and paraphrase matches, then applies a penalty for fragmentation. Designed to correlate better with human judgment than BLEU.
• ROUGE — range: [0, 1]
  • Recall-oriented metric that measures overlap of n-grams, longest common subsequences, or word sequences between predictions and references. Typically reported as ROUGE-L or ROUGE-SU4 in NLG benchmarks.

Input / output format

Input: Point clouds (P), multi-view images (I_mv), and implicit human instructions (X_inst). Internally processed into scene-level embeddings, 3D segmentation masks, and a scene graph (G) before being fed to the MLLM.

Output: A sequence of textual one-step plans terminated by a custom [END] stop token. Each plan includes step-by-step activity instructions and inter-step route details.

Scoring recipe

def evaluate(predictions, references):
    bleu4 = nltk.bleu([references], predictions, weights=(0.25, 0.25, 0.25, 0.25))
    cider = compute_cider(predictions, references)
    meteor = compute_meteor(predictions, references)
    rouge = compute_rouge(predictions, references)
    return {'BLEU-4': bleu4, 'CIDEr': cider, 'METEOR': meteor, 'ROUGE': rouge}

Common pitfalls

• The model generates a variable number of steps ending with a custom [END] token; evaluating with fixed-length truncation or ignoring the stop token will skew metrics.
• Standard n-gram metrics (BLEU, ROUGE) measure lexical overlap but do not verify spatial consistency or 3D route feasibility, potentially rewarding syntactically correct but physically impossible plans.
• CIDEr relies on TF-IDF weighting from a reference corpus; if the ReasonPlan3D reference set is small or domain-specific, scores may not generalize.

Evidence (verbatim from paper)

Benchmarking on the ReasonPlan3D validation set for the 3D Reasoning-Driven Planning task with evaluation metrics BLEU, CIDEr, METEOR, and ROUGE. Best in bold, second underlined.

Citation

@misc{jiang2025exploring3d,
  title={Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planning},
  author={Xueying Jiang et al.},
  year={2025},
  note={arXiv:2503.12974}
}

• arXiv: 2503.12974