Calib3d Eval

calib3d-eval

Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding — Kong et al. (2024) (arXiv:2403.17010, 2024)

What this evaluates

Evaluates the calibration quality of 3D scene understanding models by measuring the discrepancy between predicted confidence and actual accuracy. It probes reliability under both in-domain conditions and out-of-domain stressors like adverse weather, sensor failures, and domain shifts.

Datasets

• nuScenes — total ?; splits: train (-1), val (-1)
• SemanticKITTI — total ?; splits: train (-1), val (-1)
• Waymo Open — total ?; splits: train (-1), val (-1)
• SemanticPOSS — total ?; splits: train (-1), val (-1)
• SemanticSTF — total ?; splits: train (-1), val (-1)
• ScribbleKITTI — total ?; splits: train (-1), val (-1)
• Synth4D — total ?; splits: train (-1), val (-1)
• S3DIS — total ?; splits: train (-1), val (-1)

Metrics

• ECE (primary) — range: percent
  • Expected Calibration Error: the weighted average of the absolute difference between predicted confidence and actual accuracy across confidence bins. Lower is better.
• mIoU — range: percent
  • Mean Intersection-over-Union: the average of the IoU scores computed per semantic class across the validation set.
• mRR — range: percent
  • Mean Resilience Rate: measures robustness by comparing IoU under various corruption types to the clean IoU, averaged across corruption categories.

Input / output format

Input: 3D LiDAR point clouds (provided as raw points, voxels, or range images) with ground-truth semantic segmentation labels.

Output: Per-point/voxel predicted semantic class probabilities (or logits) and the corresponding predicted class label.

Scoring recipe

def compute_ece(predictions, labels, num_bins=15):
    confidences = np.max(predictions, axis=-1)
    preds = np.argmax(predictions, axis=-1)
    accuracies = (preds == labels).astype(float)
    bin_boundaries = np.linspace(0, 1, num_bins + 1)
    ece = 0.0
    for i in range(num_bins):
        mask = (confidences >= bin_boundaries[i]) & (confidences < bin_boundaries[i+1])
        if np.sum(mask) == 0: continue
        bin_acc = np.mean(accuracies[mask])
        bin_conf = np.mean(confidences[mask])
        ece += np.sum(mask) * abs(bin_acc - bin_conf)
    return ece / len(labels) * 100

Common pitfalls

• Models are frequently overconfident in mid-to-far depth regions where LiDAR point density drops significantly.
• Training on synthetic data yields deceptively low ECE scores that do not generalize to real-world distributions.
• Weak supervision (e.g., scribble annotations) restricts model capacity, artificially inflating predictive uncertainty compared to dense labels.

Evidence (verbatim from paper)

The expected calibration error (ECE) metric, as depicted in Eq. (2), is the primary benchmark indicator. We also use class-wise Intersection-over-Union (IoU) and mean IoU (mIoU) to measure 3D segmentation accuracy. For robustness probing, we adopt corruption-wise IoU scores and the mean Resilience Rate (mRR) from Robo3D [59] to measure the 3D robustness.

Citation

@misc{kong2024calib3d,
  title={Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding},
  author={Kong et al. (2024)},
  year={2024},
  note={arXiv:2403.17010}
}

• arXiv: 2403.17010