Multimodal Ood Eval

multimodal-ood-eval

Extremely Simple Multimodal Outlier Synthesis for Out-of-Distribution Detection and Segmentation — Liu et al. (2025) (arXiv:2505.16985, 2025)

What this evaluates

Evaluates a model's ability to correctly classify in-distribution (ID) classes while detecting and segmenting out-of-distribution (OOD) objects across multiple modalities (RGB, LiDAR, video, optical flow). It probes robustness to distribution shift and mitigates overconfidence in uncertainty-based methods.

Datasets

• SemanticKITTI — total ?; splits: train (-1), test (-1)
• nuScenes — total ?; splits: train (-1), test (-1)
• CARLA-OOD — total 245; splits: eval (245)
• HMDB51 — total ?; splits: train (-1), test (-1)
• UCF101 — total ?; splits: train (-1), test (-1)
• Kinetics-600 — total ?; splits: train (-1), test (-1)
• HAC — total ?; splits: train (-1), test (-1)
• EPIC-Kitchens — total ?; splits: train (-1), test (-1)

Metrics

• mIoUc — range: percent
  • Mean Intersection over Union computed only over in-distribution (known) classes. Calculated as the average IoU between predicted and ground truth masks for each ID class.
• AUROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve for binary ID vs OOD classification. Measures the trade-off between true positive rate and false positive rate across all thresholds.
• AUPR — range: [0, 1]
  • Area under the Precision-Recall curve for binary ID vs OOD classification. Emphasizes performance on the positive (ID) class, especially under class imbalance.
• FPR@95 — range: percent
  • False Positive Rate when the True Positive Rate is fixed at 95%. Lower values indicate better separation between ID and OOD samples.
• ACC — range: percent
  • Classification accuracy on in-distribution classes. Calculated as the proportion of correctly predicted ID samples out of total ID samples.

Input / output format

Input: Multimodal pairs per instance: RGB images + LiDAR point clouds (segmentation) or video + optical flow (detection). Ground truth labels provided for ID classes; OOD classes are masked as void during training.

Output: Per instance: (1) Segmentation: ID class mask + OOD detection mask/score. (2) Detection: ID class prediction + scalar OOD score (e.g., MaxLogit or Energy).

Scoring recipe

def compute_metrics(predictions, labels, ood_scores):
    # Closed-set metrics
    miouc = mean(iou(pred, gt) for class in known_classes)
    acc = accuracy_score(pred_class, gt_class)
    
    # OOD metrics (binary: 1=ID, 0=OOD)
    auroc = roc_auc_score(labels, ood_scores)
    aupr = average_precision_score(labels, ood_scores)
    
    # FPR@95: find threshold where TPR >= 0.95, then compute FPR
    tpr, fpr, _ = roc_curve(labels, ood_scores)
    idx = np.where(tpr >= 0.95)[0][0]
    fpr95 = fpr[idx]
    
    return {'mIoUc': miouc, 'ACC': acc, 'AUROC': auroc, 'AUPR': aupr, 'FPR@95': fpr95}

Common pitfalls

• Using different OOD scoring functions (MaxLogit vs. Energy) across datasets without explicit standardization can skew FPR@95 and AUROC comparisons.
• Treating OOD classes as 'void' during training but failing to explicitly mask or score them as unknown during inference leads to artificially inflated ID metrics and poor OOD separation.
• Confusing closed-set accuracy/mIoU with OOD detection performance; the protocol strictly separates ID classification quality from OOD detection quality.

Evidence (verbatim from paper)

For OOD segmentation, we evaluate both closed-set and OOD segmentation performance at the point level. For closed-set evaluation, we use the mean Intersection over Union for known classes (mIoUc). For OOD performance, we report the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPR), and the false positive rate at 95% true positive rate (FPR@95). For multimodal OOD detection, we report average accuracy (ACC) instead of mIoUc for closed-set evaluation, as well as AUROC and FPR@95 for OOD performance.

Citation

@misc{liu2025extremelysimplemultimodal,
  title={Extremely Simple Multimodal Outlier Synthesis for Out-of-Distribution Detection and Segmentation},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2505.16985}
}

• arXiv: 2505.16985