Comsamy Eval

comsamy-eval

Open-set Anomaly Segmentation in Complex Scenarios — Xia et al. (2025) (arXiv:2504.19706, 2025)

What this evaluates

Evaluates the robustness of open-set anomaly segmentation models under complex, real-world driving conditions. It probes a model's ability to detect out-of-distribution objects across diverse landforms and adverse weather while correctly ignoring non-driving-area elements and void regions.

Datasets

• ComsAmy — total 468; splits: test (468)

Metrics

• AuPRC (primary) — range: [0, 1]
  • Area under the Precision-Recall Curve. Computed over pixel-level predictions and ground truth, excluding void regions from false positive counts. Higher values indicate better detection on imbalanced data.
• FPR95 — range: [0, 1]
  • False Positive Rate at 95% True Positive Rate. Measures the proportion of non-anomaly pixels incorrectly classified as anomalies when 95% of anomalous pixels are correctly detected. Lower values indicate better discrimination at high recall.

Input / output format

Input: RGB images of driving scenes containing diverse landforms and weather conditions (clear, rain, fog, snow, night, etc.).

Output: Pixel-level segmentation mask with three classes: anomaly, non-anomaly, and void.

Scoring recipe

def compute_metrics(pred_mask, gt_mask, void_mask):
    valid = void_mask == 0
    pred_v = pred_mask[valid]
    gt_v = gt_mask[valid]
    prec, rec, _ = precision_recall_curve(gt_v, pred_v)
    auprc = auc(rec, prec)
    fpr, tpr, _ = roc_curve(gt_v, pred_v)
    fpr95 = interp(0.95, tpr, fpr)
    return auprc, fpr95

Common pitfalls

• Predictions falling within the 'void' class region must be excluded from false positive counts; treating them as errors will artificially inflate FPR95.
• Anomalies are strictly defined as out-of-distribution objects that are either in the driving area or could enter it, and are large/heavy enough to impact driving. Standard Cityscapes classes are considered non-anomalies.
• FPR95 is a lower-is-better metric, unlike accuracy or AuPRC, which can lead to misinterpretation if not carefully tracked.

Evidence (verbatim from paper)

Following previous work*[[2], [3], [4]]*, the Area under the Precision-Recall Curve (AuPRC) and False Positive Rate at 95% True Positive Rate (FPR95) are considered as the metrics to evaluate the model’s ability for anomaly segmentation. AuPRC is a robust metric commonly utilized to evaluate anomaly detection models, particularly in scenarios where the anomalous class constitutes a small proportion of the dataset. It effectively captures the trade-off between precision and recall across all classification thresholds, making it suitable for evaluating performance on imbalanced datasets, typically in anomaly segmentation tasks. Higher AuPRC values correspond to better anomaly detection capabilities. FPR95 quantifies the proportion of non-anomaly pixels incorrectly classified as anomalies when the detection system correctly identifies 95% of anomalous pixels. Lower values of FPR95 indicate superior performance in effectively distinguishing anomalous objects from normal objects at high recall levels. Any prediction that is within the region of void class is not counted as a false positive prediction.

Citation

@misc{xia2025comsamy,
  title={Open-set Anomaly Segmentation in Complex Scenarios},
  author={Xia et al. (2025)},
  year={2025},
  note={arXiv:2504.19706}
}

• arXiv: 2504.19706