Local Prompt Ood Eval

local-prompt-ood-eval

Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection — Zeng et al. (2024) (arXiv:2409.04796, 2024)

What this evaluates

This evaluation probes a vision-language model's ability to distinguish in-distribution images from out-of-distribution samples using few-shot prompt tuning. It specifically tests fine-grained regional outlier detection by measuring how well the model separates known classes from diverse OOD datasets and semantically similar near-OOD subsets.

Datasets

• ImageNet-1K & OOD combination — total ?; splits: train (-1), test (-1)

Metrics

• FPR95 (primary) — range: percent
  • False positive rate when the true positive rate is fixed at 95%. Lower is better.
• AUROC — range: percent
  • Area under the receiver operating characteristic curve plotting true positive rate against false positive rate. Higher is better.
• ID accuracy — range: percent
  • Top-1 classification accuracy on the in-distribution test set. Higher is better.

Input / output format

Input: Image patches fed into a frozen CLIP-Base/16 vision encoder, combined with learnable local prompts and frozen global prompts.

Output: OOD score (similarity between image and class prompts) for each class, and the predicted class label (argmax of ID scores).

Scoring recipe

def compute_metrics(predictions, gold_labels, id_preds, id_golds):
    # predictions: OOD scores, gold_labels: 0 for ID, 1 for OOD
    tpr, fpr, _ = roc_curve(gold_labels, predictions)
    fpr95 = fpr[tpr >= 0.95][0]
    auroc = auc(fpr, tpr) * 100  # Convert to percent
    id_acc = sum(p == g for p, g in zip(id_preds, id_golds)) / len(id_golds) * 100
    return {'FPR95': fpr95, 'AUROC': auroc, 'ID accuracy': id_acc}

Common pitfalls

• OOD datasets are pooled together for evaluation but reported separately and averaged; mixing them up skews results.
• FPR95 is reported as a percentage (0–100) in tables, not a probability (0–1), which can cause confusion when comparing with other works.
• Few-shot tuning uses only a subset of the training set (4 or 16 shots per class), so results are not directly comparable to zero-shot or full-tuning baselines without noting the shot count.

Evidence (verbatim from paper)

Evaluation metrics. We report the following metrics for evaluation: (1) the area under the receiver operating characteristic curve (AUROC); (2) false positive rate of OOD samples when true positive rate of ID samples is 95% (FPR95); (3) in-distribution data classification accuracy (ID accuracy).

Citation

@misc{zeng2024localprompt,
  title={Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection},
  author={Zeng et al. (2024)},
  year={2024},
  note={arXiv:2409.04796}
}

• arXiv: 2409.04796