Indoor Lidar Eval

indoor-lidar-eval

INDOOR-LiDAR: Bridging Simulation and Reality for Robot-Centric 360 degree Indoor LiDAR Perception -- A Robot-Centric Hybrid Dataset — Haichuan Li et al. (arXiv:2512.12377, 2025)

What this evaluates

Evaluates 3D object detection and BEV perception capabilities on indoor robotic platforms using LiDAR point clouds. It probes a model's ability to classify indoor objects and localize them with 3D bounding boxes, specifically highlighting the sim-to-real transfer gap in controlled indoor environments.

Datasets

• INDOOR-LiDAR — total ?; splits: simulated test (-1), real-world test (-1)

Metrics

• Precision (P) — range: [0, 1]
  • True positives divided by total predicted positives. Measures the fraction of predicted detections that are correct classifications.
• Mean IoU (primary) — range: [0, 1]
  • Average Intersection over Union between matched predicted and ground-truth bounding boxes across all true positives. Higher values indicate better geometric alignment.
• Acc@IoU0.25 — range: [0, 1]
  • Fraction of predictions with IoU ≥ 0.25 against ground truth. Measures localization tolerance at a low threshold.
• Acc@IoU0.50 — range: [0, 1]
  • Fraction of predictions with IoU ≥ 0.50 against ground truth. Standard detection threshold.
• Acc@IoU0.75 — range: [0, 1]
  • Fraction of predictions with IoU ≥ 0.75 against ground truth. Measures strict localization accuracy.
• L1 distance error — range: meters
  • Manhattan distance between predicted and ground-truth box centers. Lower values indicate better localization.
• L2 distance error — range: meters
  • Euclidean distance between predicted and ground-truth box centers. Lower values indicate better localization.

Input / output format

Input: Dense 3D LiDAR point clouds with intensity maps captured from diverse indoor environments (offices, labs, dining halls, cafes, stairwells).

Output: Predicted 3D bounding boxes with class labels and spatial coordinates.

Scoring recipe

def evaluate(preds, gts):
    tp, fp = 0, 0
    ious, l1s, l2s = [], [], []
    for pred in preds:
        best_iou, best_gt = 0, None
        for gt in gts:
            if pred['class'] == gt['class']:
                iou = calculate_iou(pred['bbox'], gt['bbox'])
                if iou > best_iou:
                    best_iou, best_gt = iou, gt
        if best_iou > 0.5:
            tp += 1
            ious.append(best_iou)
            l1s.append(l1_dist(pred['bbox'], best_gt['bbox']))
            l2s.append(l2_dist(pred['bbox'], best_gt['bbox']))
        else:
            fp += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    mean_iou = sum(ious) / len(ious) if ious else 0
    return precision, mean_iou, l1s, l2s

Common pitfalls

• Models exhibit drastic performance degradation when evaluated on real-world data compared to simulated data, making sim2real transfer a critical evaluation dimension.
• Different architectures dominate different metrics; e.g., GroupFree3D excels at classification precision while PointRCNN leads in geometric localization (Mean IoU/L1/L2).
• Rare or ambiguous object categories (e.g., 'All other') consistently yield 0.00 precision across all evaluated models, requiring careful per-class reporting.

Evidence (verbatim from paper)

To provide a multifarious evaluation of model performance, we report metrics across three aspects of the detection task: • Classification Performance: We use Precision (P) to evaluate a model's ability to correctly classify detection. - Bounding Box Quality: We evaluate the geometric accuracy of the predicted bounding boxes using several metrics: Mean IoU across all true positives, Accuracy at different IoU thresholds, and the L1 and L2 distance errors.

Citation

@misc{li2025indoorlidar,
  title={INDOOR-LiDAR: Bridging Simulation and Reality for Robot-Centric 360 degree Indoor LiDAR Perception -- A Robot-Centric Hybrid Dataset},
  author={Haichuan Li et al.},
  year={2025},
  note={arXiv:2512.12377}
}

• arXiv: 2512.12377