Objvariantensemble Eval

objvariantensemble-eval

ObjVariantEnsemble: Advancing Point Cloud LLM Evaluation in Challenging Scenes with Subtly Distinguished Objects — Cao et al. (2024) (arXiv:2412.14837, 2024)

What this evaluates

Evaluates 3D grounding models' spatial reasoning and fine-grained object distinction capabilities by testing their ability to locate a target object among visually similar distractors in ensembled point cloud scenes.

Datasets

• OVE (ObjVariantEnsemble) — total 75000; splits: test (-1)

Metrics

• ACC@0.25 (primary) — range: [0, 1]
  • Accuracy calculated as the fraction of correctly localized objects where the Intersection over Union (IoU) between the predicted and ground-truth 3D bounding boxes is at least 0.25.

Input / output format

Input: A 3D point cloud scene containing a target object and multiple distractors, accompanied by a text query specifying the target's distinguishing attributes or spatial relationship.

Output: A 3D bounding box (coordinates) or spatial location corresponding to the target object in the scene.

Scoring recipe

correct = 0
for pred_box, gt_box in zip(predictions, ground_truth):
    iou = calculate_3d_iou(pred_box, gt_box)
    if iou >= 0.25:
        correct += 1
return correct / len(predictions)

Common pitfalls

• Models may exploit coarse color or shape cues present in training data, failing when only spatial location distinguishes the target.
• Scale mismatches and overlapping bounding boxes in ensembled scenes can artificially inflate or deflate IoU scores if preprocessing is not standardized.
• The benchmark's fine-grained distinction types (location-only vs location+shape/color) require separate evaluation to avoid averaging out specific reasoning failures.

Evidence (verbatim from paper)

Performance(ACC@0.25) of 3D grounding models on OVE across different numbers of distractors, considering only location.

Citation

@misc{cao2024objvariantensemble,
  title={ObjVariantEnsemble: Advancing Point Cloud LLM Evaluation in Challenging Scenes with Subtly Distinguished Objects},
  author={Cao et al. (2024)},
  year={2024},
  note={arXiv:2412.14837}
}

• arXiv: 2412.14837