Nova Eval

nova-eval

NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRI — Bercea et al. (2025) (arXiv:2505.14064, 2025)

What this evaluates

Evaluates large vision-language models' ability to detect, localize, and reason about rare brain MRI anomalies under extreme clinical and semantic distribution shifts. It probes zero-shot generalization across localization, descriptive captioning, and diagnostic classification without closed-set assumptions.

Datasets

• NOVA — total 906; splits: test (-1)

Metrics

• Top-1 accuracy (primary) — range: [0, 1]
  • Fraction of cases where the model's single highest-confidence diagnostic label matches the ground truth label.
• mAP@50 — range: [0, 1]
  • Mean Average Precision at Intersection over Union (IoU) threshold of 0.50, averaging precision across all recall levels for detected anomalies.
• Clinical Term F1 — range: [0, 1]
  • F1 score computed over the presence of clinically relevant diagnostic terms in generated captions compared to ground truth reports.
• BLEU — range: [0, 1]
  • Bilingual Evaluation Understudy score measuring n-gram overlap between model-generated captions and reference reports.
• Top-5 accuracy — range: [0, 1]
  • Fraction of cases where the ground truth label appears within the model's top 5 highest-confidence diagnostic predictions.

Input / output format

Input: Single brain MRI scan image; for diagnostic reasoning, optionally accompanied by clinical history/metadata.

Output: Bounding box coordinates for localization; free-text clinical caption for description; single or ranked diagnostic label(s) for reasoning.

Scoring recipe

def top1_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return correct / len(gold_labels)

Common pitfalls

• Models frequently generate false-positive bounding boxes on normal anatomical structures (e.g., misinterpreting the orbital cavity as a lesion).
• Diagnostic predictions collapse onto a compressed label space (~30% coverage), failing to recognize rare pathologies present in the ground truth.
• Proprietary models may have encountered similar cases in training data, meaning reported scores represent an upper bound on true zero-shot generalization.

Evidence (verbatim from paper)

Models were evaluated using standard object detection metrics (mAP@30, mAP@50, and mAP@[50:95]), as summarized in Table 1. Performance was assessed via Top-1 and Top-5 classification accuracy (Table 3).

Citation

@misc{bercea2025nova,
  title={NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRI},
  author={Bercea et al. (2025)},
  year={2025},
  note={arXiv:2505.14064}
}

• arXiv: 2505.14064