Lod Ood Detection Eval

lod-ood-detection-eval

LoD: Loss-difference OOD Detection by Intentionally Label-Noisifying Unlabeled Wild Data — Geng et al. (2025) (arXiv:2505.12952, 2025)

What this evaluates

Evaluates a model's ability to distinguish in-distribution (ID) samples from out-of-distribution (OOD) samples using a threshold-free loss-difference clustering approach. It measures detection performance across standard benchmarks with diverse natural images and hard benchmarks where OOD classes share the same source dataset as ID.

Datasets

• CIFAR100 — total ?; splits: train (-1), test (-1)
• SVHN — total ?; splits: train (-1), test (-1)
• Places — total ?; splits: val (-1)
• LSUN-Crop — total ?; splits: test (-1)
• LSUN-Resize — total ?; splits: test (-1)
• Textures — total ?; splits: test (-1)
• CIFAR10 — total ?; splits: train (-1), test (-1)
• TinyImageNet — total ?; splits: train (-1), val (-1)

Metrics

• FPR95 (primary) — range: percent
  • False positive rate on OOD examples when the true positive rate on ID examples is fixed at 95%.
• AUROC — range: percent
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all thresholds.
• ACC — range: percent
  • In-distribution classification accuracy, calculated as the percentage of correctly classified ID test samples.

Input / output format

Input: RGB images from ID training sets, unlabeled wild data (mixed ID and OOD images), and OOD test sets. The model processes images through a backbone (WideResNet-40) to extract penultimate features for loss computation and binary OOD classification.

Output: Per image: a loss value (or loss difference) used for OOD scoring, and a binary OOD detection label (ID vs OOD) derived via threshold-free clustering. For ID test data, class predictions are also produced.

Scoring recipe

def compute_metrics(predictions, gold, id_preds, id_labels):
    # predictions: OOD scores (higher = more likely OOD)
    # gold: 1 for ID, 0 for OOD
    # FPR95
    tpr_target = 0.95
    threshold = np.percentile(predictions[gold==1], (1 - tpr_target) * 100)
    fpr95 = np.mean(predictions[gold==0] > threshold) * 100
    # AUROC
    auroc = roc_auc_score(gold, predictions) * 100
    # ACC
    acc = accuracy_score(id_labels, id_preds) * 100
    return fpr95, auroc, acc

Common pitfalls

• The method explicitly uses a threshold-free clustering approach on loss dynamics, so applying a fixed decision threshold during evaluation contradicts the protocol.
• Hard benchmarks construct OOD data by holding out classes from the same dataset as ID (e.g., CIFAR10), which is fundamentally different from standard benchmarks that use entirely different datasets.
• The mixture proportion π controls the ratio of ID to OOD samples in the unlabeled wild training data, not the test distribution, and results vary significantly across π ∈ {0.1, 0.5, 0.9}.

Evidence (verbatim from paper)

we adopt the following evaluation metrics: (1) the false positive rate (FPR95) of OOD examples when true positive rate of ID examples is at 95%, (2) Area Under the Receiver Operating Characteristic curve (AUROC), and (3) ID classification Accuracy (ACC).

Citation

@misc{geng2025lod,
  title={LoD: Loss-difference OOD Detection by Intentionally Label-Noisifying Unlabeled Wild Data},
  author={Geng et al. (2025)},
  year={2025},
  note={arXiv:2505.12952}
}

• arXiv: 2505.12952