qhjqhj00 scannet-3d-detection-eval
Anyview: Generalizable Indoor 3D Object Detection with Variable Frames — Zhenyu Wu et al. (arXiv:2310.05346, 2023)
What this evaluates
Evaluates the ability of 3D object detectors to localize and classify indoor objects using variable-frame sparse RGB-D inputs. It probes generalization across different input modalities (reconstructed point clouds, multi-view RGB-D, monocular RGB-D) and varying numbers of input views.
Datasets
- ScanNet — total 1513; splits: train (1201), val (312)
Metrics
mAP@0.25(primary) — range: percent- Mean Average Precision computed at an Intersection-over-Union (IoU) threshold of 0.25 for 3D axis-aligned bounding boxes. AP is calculated by averaging precision over recall thresholds, then averaged across all 18 object categories.
mAP@0.5— range: percent- Same as mAP@0.25 but computed at a stricter IoU threshold of 0.5 for 3D axis-aligned bounding boxes.
Input / output format
Input: Variable-frame sparse RGB-D inputs. For scene-level benchmarks: reconstructed point clouds or multi-view RGB-D images (default 50 views). For monocular: single RGB-D frame. For online: sequential RGB-D frames processed frame-by-frame.
Output: Set of 3D axis-aligned bounding boxes with class labels for 18 selected indoor object categories.
Scoring recipe
def compute_3d_map(preds, golds, iou_thresh=0.25):
ap_scores = []
for cls in classes:
cls_preds = [p for p in preds if p['class'] == cls]
cls_golds = [g for g in golds if g['class'] == cls]
tp, fp = 0, 0
for pred in sorted(cls_preds, key=lambda x: x['score'], reverse=True):
best_iou = max(iou(pred['box'], gt['box']) for gt in cls_golds)
if best_iou >= iou_thresh:
tp += 1
else:
fp += 1
ap = tp / (tp + fp) if (tp + fp) > 0 else 0
ap_scores.append(ap)
return sum(ap_scores) / len(ap_scores) * 100
Common pitfalls
- Confusing ScanNet-Rec (requires pre-reconstructed point clouds) with ScanNet-MV (uses raw multi-view RGB-D), which drastically changes input availability and baseline performance.
- Assuming fixed input scales; the benchmark explicitly tests generalization across 10, 30, and 50 views, as well as streaming online detection, which breaks standard fixed-input detectors.
- Overlooking point-sampling differences in evaluation (e.g., AnyView uses 5000 points/view vs 20000/40000 for baselines), which affects computational cost but not the mAP metric itself.
Evidence (verbatim from paper)
ScanNet-Rec is the mainstream scene-level benchmark used by previous works, whose input data is reconstructed point cloud of a whole scene and ground-truth is generated by computing the axis-aligned bounding boxes for objects in 18 selected categories. We show the performance of different models on scene-level (Rec and MV) and monocular (SV) benchmarks in Table III. 3D object detection results (mAP@0.25 and mAP@0.5) and corresponding training/inference setting of different methods on scene-level and monocular benchmarks.
Citation
@misc{wu2023anyview,
title={Anyview: Generalizable Indoor 3D Object Detection with Variable Frames},
author={Zhenyu Wu et al.},
year={2023},
note={arXiv:2310.05346}
}
- arXiv: 2310.05346