Oodis Eval

oodis-eval

OoDIS: Anomaly Instance Segmentation and Detection Benchmark — Nekrasov et al. (2024) (arXiv:2406.11835, 2024)

What this evaluates

Evaluates a model's ability to detect and segment anomalous objects (out-of-distribution instances) in real-world driving scenes. It measures instance-level segmentation and object detection performance on rare, unpredictable obstacles that are not part of the standard in-distribution classes.

Datasets

• OoDIS Benchmark — total 887; splits: test (787), val (100)

Metrics

• AP (primary) — range: [0, 1]
  • Average Precision computed over IoU thresholds T = {50, 55, 60, ..., 95}. AP = (1/|T|) * sum_{t in T} AP_t, where AP_t is the area under the precision-recall curve at IoU threshold t.
• AP50 — range: [0, 1]
  • Average Precision computed at a single IoU threshold of 50%.
• AR@k — range: [0, 1]
  • Average Recall averaged over IoU thresholds T, restricting predictions to the top-k highest confidence scores.
• PPF — range: other
  • Predictions Per Frame, averaged over all images in a dataset to measure over-production of predictions.

Input / output format

Input: RGB images from driving scenes containing in-distribution and anomalous objects.

Output: Per-instance predictions comprising bounding boxes and/or pixel-level segmentation masks, each accompanied by a confidence score.

Scoring recipe

def compute_ap(predictions, ground_truth, iou_thresholds=[50,55,...,95]):
    aps = []
    for t in iou_thresholds:
        tp, fp = 0, 0
        matched_gts = set()
        for pred in sorted(predictions, key=lambda x: x.conf, reverse=True):
            best_gt = max([gt for gt in ground_truth if gt.id not in matched_gts],
                          key=lambda gt: iou(pred, gt), default=None)
            if best_gt and iou(pred, best_gt) >= t/100:
                tp += 1
                matched_gts.add(best_gt.id)
            else:
                fp += 1
        prec = cumsum(tp) / (cumsum(tp) + cumsum(fp))
        rec = cumsum(tp) / len(ground_truth)
        aps.append(apcoco(prec, rec))
    return mean(aps)

Common pitfalls

• Confusing per-pixel AP with instance-level AP; the benchmark explicitly uses instance segmentation/detection AP (COCO/Cityscapes style), not pixel-wise metrics.
• Ignoring 'ignore' regions; predictions overlapping significantly with ambiguous/ignore regions should be discarded, as evaluation only focuses on the outlier class.
• Not weighting dataset scores correctly; the final benchmark score is a weighted average based on the number of images per dataset (RoadAnomaly21, RoadObstacle21, FS L&F).

Evidence (verbatim from paper)

We utilize the Cityscapes instance segmentation evaluation suite for anomaly instance segmentation as well as the COCO evaluation suite for anomalous object detection. Our benchmark primarily uses the AP metric (not the be confused with the per-pixel AP), a standard in instance segmentation and object detection. We consider AP50 (t=50) and the AP, which is an average over different thresholds, namely AP = 1/T sum_{t in T} AP_t for T = {50,55,60,...,95}.

Citation

@misc{nekrasov2024oodis,
  title={OoDIS: Anomaly Instance Segmentation and Detection Benchmark},
  author={Nekrasov et al. (2024)},
  year={2024},
  note={arXiv:2406.11835}
}

• arXiv: 2406.11835