Ir3d Bench Eval

ir3d-bench-eval

IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering — Parker Liu et al. (arXiv:2506.23329, 2025)

What this evaluates

Evaluates vision-language models' ability to understand 3D scenes by generating executable scene descriptions from a single 2D image. It shifts evaluation from passive captioning to active reconstruction, probing geometric layout, spatial reasoning, object appearance, and semantic attributes.

Datasets

• IR3D-Bench — total ?; splits: test (-1)

Metrics

• Pixel Distance (primary) — range: [0, 1]
  • Mean L2 distance between predicted and ground-truth object centers. Lower values indicate better localization.
• Bounding Box IoU — range: [0, 1]
  • Intersection over Union between predicted and ground-truth bounding boxes. Higher values indicate better spatial extent estimation.
• Relational Accuracy — range: [0, 1]
  • Accuracy of predicted inter-object spatial relations (e.g., proximity, containment, relative position) compared to ground truth.
• CLIP Score — range: [0, 1]
  • Cosine similarity between CLIP embeddings of predicted and ground-truth object appearance attributes (color, size, material, shape).
• LLM Score — range: [0, 1]
  • Score assigned by an LLM judge evaluating the predicted object appearance and overall scene layout against ground truth.

Input / output format

Input: A single 2D image accompanied by a prompt instructing the VLM to extract geometric information (shape, size, position, material) for each object. Camera intrinsic and extrinsic parameters are fixed and provided to ensure consistent reconstruction.

Output: A structured JSON file containing predicted object attributes, bounding boxes, centers, and scene layout.

Scoring recipe

def compute_metrics(pred_json, gt_json):
    pred_objs = parse_json(pred_json)
    gt_objs = parse_json(gt_json)
    pixel_dist = mean_l2(pred_objs.center, gt_objs.center)
    bbox_iou = compute_iou(pred_objs.bbox, gt_objs.bbox)
    rel_acc = accuracy(pred_objs.relations, gt_objs.relations)
    clip_score = clip_similarity(pred_objs.appearance, gt_objs.appearance)
    llm_score = llm_judge(pred_objs.layout, gt_objs.layout)
    return {
        'pixel_distance': pixel_dist,
        'bbox_iou': bbox_iou,
        'relational_accuracy': rel_acc,
        'clip_score': clip_score,
        'llm_score': llm_score
    }

Common pitfalls

• Several models fail to produce valid JSON outputs, indicating either insufficient 3D understanding or incompatibility with the task format.
• Models often achieve high appearance recognition scores but struggle significantly with spatial reasoning and inter-object relations.
• The benchmark assumes fixed camera parameters; models cannot infer them, which constrains the evaluation to scene attribute prediction rather than full camera recovery.

Evidence (verbatim from paper)

Relational Accuracy, which captures reasoning over inter-object spatial relations, is below 0.3 for most models (GPT-4o: 0.28, Gemini: 0.26), showing persistent errors in understanding relative positions, proximity, and containment.

Citation

@misc{liu2025ir3dbench,
  title={IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering},
  author={Parker Liu et al.},
  year={2025},
  note={arXiv:2506.23329}
}

• arXiv: 2506.23329