Rexrank Eval

rexrank-eval

ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation — Zhang et al. (2024) (arXiv:2411.15122, 2024)

What this evaluates

Evaluates AI models' ability to generate accurate and clinically relevant radiology reports from chest X-ray images. It assesses both linguistic quality and clinical entity extraction/alignment across diverse clinical datasets.

Datasets

• ReXGradient — total 10000; splits: test (-1)
• MIMIC-CXR — total ?; splits: test (-1)
• IU X-ray — total ?; splits: test (-1)
• CheXpert Plus — total ?; splits: test (-1)

Metrics

• 1/RadCliQ-v1 (primary) — range: other
  • Inverse of the RadCliQ-v1 clinical entity alignment score. Higher values indicate better clinical accuracy. Computed as 1 / RadCliQ_v1_score.
• BLEU-2 — range: [0, 1]
  • Standard 2-gram BLEU score measuring n-gram overlap between generated and reference reports.
• BertScore — range: [0, 1]
  • F1 score computed using contextual embeddings from a BERT model to measure semantic similarity.
• SembScore — range: [0, 1]
  • Semantic similarity score based on sentence embeddings, measuring contextual alignment.
• RadGraph — range: [0, 1]
  • F1 score measuring the overlap of extracted radiological entities and relations between prediction and ground truth.
• RaTEScore — range: [0, 1]
  • Radiology Text Embedding score measuring semantic similarity using domain-specific embeddings.
• GREEN — range: [0, 1]
  • Generative Radiology Evaluation metric assessing clinical correctness and completeness of generated reports.
• 1/FineRadScore — range: other
  • Inverse of the FineRadScore, which evaluates fine-grained clinical correctness. Higher values indicate better performance.

Input / output format

Input: Chest X-ray image(s) (and optionally clinical metadata).

Output: Radiology report text, generated either as 'Findings' only or 'Findings + Impression'.

Scoring recipe

def compute_metrics(pred_report, gold_report):
    radcliq = compute_radcliq_v1(pred_report, gold_report)
    bleu2 = compute_bleu2(pred_report, gold_report)
    bertscore = compute_bertscore(pred_report, gold_report)
    sembscore = compute_sembscore(pred_report, gold_report)
    radgraph = compute_radgraph(pred_report, gold_report)
    rtescore = compute_rtescore(pred_report, gold_report)
    green = compute_green(pred_report, gold_report)
    fineradscore = compute_fineradscore(pred_report, gold_report)
    return {
        '1/RadCliQ-v1': 1.0 / radcliq,
        'BLEU-2': bleu2,
        'BertScore': bertscore,
        'SembScore': sembscore,
        'RadGraph': radgraph,
        'RaTEScore': rtescore,
        'GREEN': green,
        '1/FineRadScore': 1.0 / fineradscore
    }

Common pitfalls

• The primary metric is inverted (1/RadCliQ-v1), so higher values indicate better performance, contrary to standard RadCliQ scores.
• Models are evaluated on two distinct tasks: 'Findings' only and 'Findings + Impression', which should not be mixed when comparing results.
• Confidence intervals (95% CI) are reported alongside mean scores, indicating statistical variance that should be considered when ranking models.

Evidence (verbatim from paper)

Table 1, 2, 3 and 4 summarize the performance of various medical report generation models across four different datasets: ReXGradient, MIMIC-CXR, IU X-ray and CheXpert Plus. Among these, MedVersa demonstrates superior performance, achieving the best 1/RadCliQ-v1 scores on ReXGradient (1.01 ± 0.01), MIMIC-CXR (1.10 ± 0.02), and IU X-ray (1.46 ± 0.03) on the findings section.

Citation

@misc{zhang2024rexrank,
  title={ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2411.15122}
}

• arXiv: 2411.15122