Medical Radiology Similarity Eval

medical-radiology-similarity-eval

Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis — Shaochen Xu et al. (2024) (arXiv:2402.11398, 2024)

What this evaluates

This evaluation probes the ability of automated metrics to capture deep clinical semantics in radiology reports. It compares LLM-generated similarity scores against traditional lexical overlap metrics, measuring how well each aligns with ground truth annotations derived from clinical NLP tools.

Datasets

• Radiology Report Pairs (CheXpert/NegBio-derived) — total 62500; splits: test (62500)

Metrics

• GPT_sim (primary) — range: [0, 1]
  • Semantic similarity score generated by GPT-4 reasoning over radiology report pairs.
• ROUGE-1 F1 — range: [0, 1]
  • Unigram lexical overlap F1 score between two text pairs.
• ROUGE-2 F1 — range: [0, 1]
  • Bigram lexical overlap F1 score between two text pairs.
• ROUGE-L F1 — range: [0, 1]
  • Longest common subsequence lexical overlap F1 score between two text pairs.
• BLEU — range: [0, 1]
  • Bilingual evaluation understudy score measuring n-gram precision with brevity penalty.

Input / output format

Input: Pairs of radiology report texts.

Output: A single similarity score per text pair, representing the predicted semantic alignment.

Scoring recipe

# For each of the 62,500 report pairs:
pred_score = compute_similarity(pair, method="GPT_sim|ROUGE-1|ROUGE-2|ROUGE-L|BLEU")
gt_score = get_gt_similarity(pair, source="CheXpert|NegBio")
# Aggregate across all pairs
mean_diff = mean(abs(pred_score - gt_score))
# Report mean_diff per method per GT source

Common pitfalls

• Traditional lexical metrics (ROUGE/BLEU) yield higher mean scores but indicate greater deviation from clinical ground truth, contrary to intuition that higher overlap equals better alignment.
• Ground truth is derived from automated clinical annotation tools (CheXpert/NegBio) rather than direct physician ratings, which may not capture all clinical nuances.
• The evaluation emphasizes mean differences, potentially obscuring distributional misalignments; the authors supplement this with hexbin plots covering the 5th–95th percentile range.

Evidence (verbatim from paper)

The comparison leveraged several metrics, including GPT_sim (GPT-4 generated similarities), ROUGE-1 F1, ROUGE-2 F1, ROUGE-L F1, and BLEU scores, to evaluate their performance in aligning with the GT derived from CheXpert and NegBio annotations.

Citation

@misc{xu2024reasoning,
  title={Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis},
  author={Shaochen Xu et al. (2024)},
  year={2024},
  note={arXiv:2402.11398}
}

• arXiv: 2402.11398