Robosense Eval

robosense-eval

RoboSense: Large-scale Dataset and Benchmark for Egocentric Robot Perception and Navigation in Crowded and Unstructured Environments — Su et al. (2024) (arXiv:2408.15503, 2024)

What this evaluates

Evaluates egocentric robot perception and navigation in crowded, unstructured environments. It probes multi-view 3D detection, 3D multi-object tracking, motion prediction, and 3D/BEV occupancy prediction using synchronized camera, LiDAR, and ultrasonic sensor data.

Datasets

• RoboSense — total 133000; splits: train (-1), test (-1), val (-1); repo https://github.com/suhaisheng/RoboSense

Metrics

• average precision (primary) — range: percent
  • Average Precision computed over recall thresholds. Predictions are matched to ground truth using either Center-Point (CP) distance or the proposed Closest-Collision-Point (CCP) distance, with a relative proportion threshold p (5% for LiDAR, 10% for images).
• sAMOTA — range: percent
  • simplified Average Multi-Object Tracking Accuracy, measuring tracking consistency and identity switches over time.
• minADE — range: meters
  • Minimum Average Displacement Error, the lowest L2 distance between predicted and ground truth future trajectories across multiple sampled hypotheses.
• mIoU-3D — range: percent
  • Mean Intersection over Union calculated in 3D voxel space for occupancy prediction, excluding ground voxels from the calculation.

Input / output format

Input: Synchronized multi-sensor data at 10 FPS: RGB camera frames, fisheye camera frames, LiDAR point clouds, ultrasonic readings, and GPS/IMU localization. Inputs vary by task (e.g., image/point cloud sequences for detection/tracking, history trajectories or sensor data for prediction).

Output: Per instance: predicted 3D bounding boxes with class labels and orientations; track IDs for multi-object tracking; future trajectory waypoints for motion prediction; or 3D/BEV voxel occupancy grids.

Scoring recipe

def compute_3d_ap(pred_boxes, gt_boxes, criterion='CCP', p=0.05):
    matches = []
    for pred in pred_boxes:
        best_dist = float('inf')
        best_gt = None
        for gt in gt_boxes:
            if pred.class != gt.class: continue
            dist = ccp_distance(pred, gt) if criterion=='CCP' else center_distance(pred, gt)
            if dist < best_dist:
                best_dist, best_gt = dist, gt
        if best_dist < p * gt.length:
            matches.append((pred, best_gt))
    return average_precision_over_recall(matches)

Common pitfalls

• Using standard Center Distance or IoU matching instead of the proposed CCP criterion significantly overestimates near-field detection performance.
• The test set is closed (no ground truth provided); models must be submitted to an online benchmark for evaluation.
• Occupancy mIoU scores are artificially lowered because ground voxels are explicitly excluded from the metric calculation.

Evidence (verbatim from paper)

For practical usages, we report performance using our proposed Closest-Collision Distance Proportion (CCDP) as matching criterion. Comparisons of different matching functions on average precision are shown in Fig.[4].

Citation

@misc{su2024robosense,
  title={RoboSense: Large-scale Dataset and Benchmark for Egocentric Robot Perception and Navigation in Crowded and Unstructured Environments},
  author={Su et al. (2024)},
  year={2024},
  note={arXiv:2408.15503}
}

• arXiv: 2408.15503