Vision Language Ood Eval

vision-language-ood-eval

A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models — Noda et al. (2025) (arXiv:2501.18463, 2025)

What this evaluates

Probes the ability of vision-language models to distinguish in-distribution from out-of-distribution samples under semantic, covariate, and real-world distribution shifts. It evaluates both zero-shot and few-shot prompt learning approaches across multiple benchmarks to assess robustness and ranking consistency.

Datasets

• ImageNet-X — total ?; splits: test (-1)
• ImageNet-FS-X — total ?; splits: test (-1)
• Wilds-FS-X — total ?; splits: test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the probability that a randomly chosen OOD sample receives a higher OOD score than a randomly chosen ID sample. Higher values indicate better separation, with 50% representing random guessing.

Input / output format

Input: Image paired with text prompts (ID class labels and optionally OOD prompts) fed into a vision-language model (e.g., CLIP) to compute similarity scores.

Output: Per-image OOD score (scalar) derived from model similarity outputs, used to rank samples as ID or OOD.

Scoring recipe

def compute_auroc(predictions, gold):
    # predictions: array of OOD scores (higher = more OOD)
    # gold: binary array (1 for ID, 0 for OOD)
    fpr, tpr, _ = roc_curve(gold, predictions)
    return auc(fpr, tpr)

Common pitfalls

• Assuming higher classification accuracy on ID data directly translates to better OOD detection performance, which the paper explicitly shows is not always true.
• Evaluating only on semantic shifts while ignoring covariate shifts, which causes significant and consistent performance drops in CLIP-based methods.
• Treating all OOD datasets as a single homogeneous group without distinguishing between common-OOD and hard-OOD, as method rankings shift significantly between them.

Evidence (verbatim from paper)

For evaluation, we use the area under the receiver operating characteristic curve (AUROC). A higher value indicates better performance, with 50% serving as the baseline.

Citation

@misc{noda2025oodx,
  title={A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models},
  author={Noda et al. (2025)},
  year={2025},
  note={arXiv:2501.18463}
}

• arXiv: 2501.18463